os/doc
3
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Fix typos, style and images paths

Browse source 
Global review of styles, typos and images paths
in gjournal-desktop, releng, vinum and vm-design
This commit is contained in:
Sergio Carlavilla Delgado 2021-03-20 18:58:59 +01:00
parent 98b9c01a12
commit e16d21085b
32 changed files with 3690 additions and 1187 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

277
documentation/content/en/articles/contributing/_index.adoc
View file

@ -30,15 +30,24 @@ This article describes the different ways in which an individual or organization
toc::[]
So you want to contribute to FreeBSD? That is great! FreeBSD _relies_ on the contributions of its user base to survive. Your contributions are not only appreciated, they are vital to FreeBSD's continued growth.
So you want to contribute to FreeBSD? That is great! FreeBSD _relies_ on the contributions of its user base to survive.
Your contributions are not only appreciated, they are vital to FreeBSD's continued growth.
A large and growing number of international contributors, of greatly varying ages and areas of technical expertise, develop FreeBSD. There is always more work to be done than there are people available to do it, and more help is always appreciated.
A large and growing number of international contributors, of greatly varying ages and areas of technical expertise, develop FreeBSD.
There is always more work to be done than there are people available to do it, and more help is always appreciated.
As a volunteer, what you do is limited only by what you want to do. However, we do ask that you are aware of what other members of the FreeBSD community will expect of you. You may want to take this into account before deciding to volunteer.
As a volunteer, what you do is limited only by what you want to do.
However, we do ask that you are aware of what other members of the FreeBSD community will expect of you.
You may want to take this into account before deciding to volunteer.
The FreeBSD project is responsible for an entire operating system environment, rather than just a kernel or a few scattered utilities. As such, our [.filename]#TODO# lists span a very wide range of tasks: from documentation, beta testing and presentation, to the system installer and highly specialized types of kernel development. People of any skill level, in almost any area, can almost certainly help the project.
The FreeBSD project is responsible for an entire operating system environment, rather than just a kernel or a few scattered utilities.
As such, our [.filename]#TODO# lists span a very wide range of tasks: from documentation, beta testing and presentation, to the system installer and highly specialized types of kernel development.
People of any skill level, in almost any area, can almost certainly help the project.
Commercial entities engaged in FreeBSD-related enterprises are also encouraged to contact us. Do you need a special extension to make your product work? You will find us receptive to your requests, given that they are not too outlandish. Are you working on a value-added product? Please let us know! We may be able to work cooperatively on some aspect of it. The free software world is challenging many existing assumptions about how software is developed, sold, and maintained, and we urge you to at least give it a second look.
Commercial entities engaged in FreeBSD-related enterprises are also encouraged to contact us.
Do you need a special extension to make your product work? You will find us receptive to your requests, given that they are not too outlandish.
Are you working on a value-added product? Please let us know! We may be able to work cooperatively on some aspect of it.
The free software world is challenging many existing assumptions about how software is developed, sold, and maintained, and we urge you to at least give it a second look.
[[contrib-what]]
== What Is Needed
@ -48,7 +57,9 @@ The following list of tasks and sub-projects represents something of an amalgam
[[non-programmer-tasks]]
=== Ongoing Non-Programmer Tasks
Many people who are involved in FreeBSD are not programmers. The Project includes documentation writers, Web designers, and support people. All that these people need to contribute is an investment of time and a willingness to learn.
Many people who are involved in FreeBSD are not programmers.
The Project includes documentation writers, Web designers, and support people.
All that these people need to contribute is an investment of time and a willingness to learn.
. Read through the FAQ and Handbook periodically. If anything is poorly explained, ambiguous, out of date or incorrect, let us know. Even better, send us a fix (Docbook is not difficult to learn, but there is no objection to ASCII submissions).
. Help translate FreeBSD documentation into your native language. If documentation already exists for your language, you can help translate additional documents or verify that the translations are up-to-date and correct. First take a look at the link:{fdp-primer}#translations[Translations FAQ] in the FreeBSD Documentation Project Primer. You are not committing yourself to translating every single FreeBSD document by doing this - as a volunteer, you can do as much or as little translation as you desire. Once someone begins translating, others almost always join the effort. If you only have the time or energy to translate one part of the documentation, please translate the installation instructions.
@ -57,7 +68,8 @@ Many people who are involved in FreeBSD are not programmers. The Project include
[[ongoing-programmer-tasks]]
=== Ongoing Programmer Tasks
Most of the tasks listed here may require a considerable investment of time, an in-depth knowledge of the FreeBSD kernel, or both. However, there are also many useful tasks which are suitable for "weekend hackers".
Most of the tasks listed here may require a considerable investment of time, an in-depth knowledge of the FreeBSD kernel, or both.
However, there are also many useful tasks which are suitable for "weekend hackers".
. If you run FreeBSD-CURRENT and have a good Internet connection, there is a machine `current.FreeBSD.org` which builds a full release once a day-every now and again, try to install the latest release from it and report any failures in the process.
. Read the {freebsd-bugs}. There may be a problem you can comment constructively on or with patches you can test. Or you could even try to fix one of the problems yourself.
@ -72,15 +84,24 @@ Most of the tasks listed here may require a considerable investment of time, an
=== Work through the PR Database
The https://bugs.FreeBSD.org/search/[FreeBSD PR list] shows all the current active problem reports and requests for enhancement that have been submitted by FreeBSD users. The PR database includes both programmer and non-programmer tasks. Look through the open PRs, and see if anything there takes your interest. Some of these might be very simple tasks that just need an extra pair of eyes to look over them and confirm that the fix in the PR is a good one. Others might be much more complex, or might not even have a fix included at all.
The https://bugs.FreeBSD.org/search/[FreeBSD PR list] shows all the current active problem reports and requests for enhancement that have been submitted by FreeBSD users.
The PR database includes both programmer and non-programmer tasks.
Look through the open PRs, and see if anything there takes your interest.
Some of these might be very simple tasks that just need an extra pair of eyes to look over them and confirm that the fix in the PR is a good one.
Others might be much more complex, or might not even have a fix included at all.
Start with the PRs that have not been assigned to anyone else. If a PR is assigned to someone else, but it looks like something you can handle, email the person it is assigned to and ask if you can work on it-they might already have a patch ready to be tested, or further ideas that you can discuss with them.
Start with the PRs that have not been assigned to anyone else.
If a PR is assigned to someone else, but it looks like something you can handle, email the person it is assigned to and ask if you can work on it-they might already have a patch ready to be tested, or further ideas that you can discuss with them.
=== Ongoing Ports Tasks
The Ports Collection is a perpetual work in progress. We want to provide our users with an easy to use, up to date, high quality repository of third party software. We need people to donate some of their time and effort to help us achieve this goal.
The Ports Collection is a perpetual work in progress.
We want to provide our users with an easy to use, up to date, high quality repository of third party software.
We need people to donate some of their time and effort to help us achieve this goal.
Anyone can get involved, and there are lots of different ways to do so. Contributing to ports is an excellent way to help "give back" something to the project. Whether you are looking for an ongoing role, or a fun challenge for a rainy day, we would love to have your help!
Anyone can get involved, and there are lots of different ways to do so.
Contributing to ports is an excellent way to help "give back" something to the project.
Whether you are looking for an ongoing role, or a fun challenge for a rainy day, we would love to have your help!
There are a number of easy ways you can contribute to keeping the ports tree up to date and in good working order:
@ -91,7 +112,8 @@ There are a number of easy ways you can contribute to keeping the ports tree up
=== Pick one of the items from the Ideas page
The https://wiki.freebsd.org/IdeasPage[FreeBSD list of projects and ideas for volunteers] is also available for people willing to contribute to the FreeBSD project. The list is being regularly updated and contains items for both programmers and non-programmers with information about each project.
The https://wiki.freebsd.org/IdeasPage[FreeBSD list of projects and ideas for volunteers] is also available for people willing to contribute to the FreeBSD project.
The list is being regularly updated and contains items for both programmers and non-programmers with information about each project.
[[contrib-how]]
== How to Contribute
@ -101,25 +123,39 @@ Contributions to the system generally fall into one or more of the following 5 c
[[contrib-general]]
=== Bug Reports and General Commentary
An idea or suggestion of _general_ technical interest should be mailed to the {freebsd-hackers}. Likewise, people with an interest in such things (and a tolerance for a _high_ volume of mail!) may subscribe to the {freebsd-hackers}. See link:{handbook}#eresources-mail[The FreeBSD Handbook] for more information about this and other mailing lists.
An idea or suggestion of _general_ technical interest should be mailed to the {freebsd-hackers}.
Likewise, people with an interest in such things (and a tolerance for a _high_ volume of mail!) may subscribe to the {freebsd-hackers}.
See link:{handbook}#eresources-mail[The FreeBSD Handbook] for more information about this and other mailing lists.
If you find a bug or are submitting a specific change, please report it using the https://bugs.FreeBSD.org/submit/[bug submission form]. Try to fill-in each field of the bug report. Unless they exceed 65KB, include any patches directly in the report. If the patch is suitable to be applied to the source tree put `[PATCH]` in the synopsis of the report. When including patches, _do not_ use cut-and-paste because cut-and-paste turns tabs into spaces and makes them unusable. When patches are a lot larger than 20KB, consider compressing them (eg. with man:gzip[1] or man:bzip2[1]) prior to uploading them.
If you find a bug or are submitting a specific change, please report it using the https://bugs.FreeBSD.org/submit/[bug submission form].
Try to fill-in each field of the bug report.
Unless they exceed 65KB, include any patches directly in the report.
If the patch is suitable to be applied to the source tree put `[PATCH]` in the synopsis of the report.
When including patches, _do not_ use cut-and-paste because cut-and-paste turns tabs into spaces and makes them unusable.
When patches are a lot larger than 20KB, consider compressing them (eg. with man:gzip[1] or man:bzip2[1]) prior to uploading them.
After filing a report, you should receive confirmation along with a tracking number. Keep this tracking number so that you can update us with details about the problem.
After filing a report, you should receive confirmation along with a tracking number.
Keep this tracking number so that you can update us with details about the problem.
See also link:{problem-reports}[this article] on how to write good problem reports.
=== Changes to the Documentation
Changes to the documentation are overseen by the {freebsd-doc}. Please look at the link:{fdp-primer}[FreeBSD Documentation Project Primer] for complete instructions. Send submissions and changes (even small ones are welcome!) using the same method as any other bug report.
Changes to the documentation are overseen by the {freebsd-doc}.
Please look at the link:{fdp-primer}[FreeBSD Documentation Project Primer] for complete instructions.
Send submissions and changes (even small ones are welcome!) using the same method as any other bug report.
=== Changes to Existing Source Code
An addition or change to the existing source code is a somewhat trickier affair and depends a lot on how far out of date you are with the current state of FreeBSD development. There is a special on-going release of FreeBSD known as "FreeBSD-CURRENT" which is made available in a variety of ways for the convenience of developers working actively on the system. See link:{handbook}#current-stable[The FreeBSD Handbook] for more information about getting and using FreeBSD-CURRENT.
An addition or change to the existing source code is a somewhat trickier affair and depends a lot on how far out of date you are with the current state of FreeBSD development.
There is a special on-going release of FreeBSD known as "FreeBSD-CURRENT" which is made available in a variety of ways for the convenience of developers working actively on the system.
See link:{handbook}#current-stable[The FreeBSD Handbook] for more information about getting and using FreeBSD-CURRENT.
Working from older sources unfortunately means that your changes may sometimes be too obsolete or too divergent for easy re-integration into FreeBSD. Chances of this can be minimized somewhat by subscribing to the {freebsd-announce} and the {freebsd-current} lists, where discussions on the current state of the system take place.
Working from older sources unfortunately means that your changes may sometimes be too obsolete or too divergent for easy re-integration into FreeBSD.
Chances of this can be minimized somewhat by subscribing to the {freebsd-announce} and the {freebsd-current} lists, where discussions on the current state of the system take place.
Assuming that you can manage to secure fairly up-to-date sources to base your changes on, the next step is to produce a set of diffs to send to the FreeBSD maintainers. This is done with the man:diff[1] command.
Assuming that you can manage to secure fairly up-to-date sources to base your changes on, the next step is to produce a set of diffs to send to the FreeBSD maintainers.
This is done with the man:diff[1] command.
The preferred man:diff[1] format for submitting patches is the unified output format generated by `diff -u`.
@ -139,19 +175,29 @@ would generate a set of unified diffs for the given source file or directory hie
See man:diff[1] for more information.
Once you have a set of diffs (which you may test with the man:patch[1] command), you should submit them for inclusion with FreeBSD as a bug report. _Do not_ just send the diffs to the {freebsd-hackers} or they will get lost! We greatly appreciate your submission (this is a volunteer project!); because we are busy, we may not be able to address it immediately, but it will remain in the PR database until we do. Indicate your submission by including `[PATCH]` in the synopsis of the report.
Once you have a set of diffs (which you may test with the man:patch[1] command), you should submit them for inclusion with FreeBSD as a bug report.
_Do not_ just send the diffs to the {freebsd-hackers} or they will get lost! We greatly appreciate your submission (this is a volunteer project!); because we are busy, we may not be able to address it immediately, but it will remain in the PR database until we do.
Indicate your submission by including `[PATCH]` in the synopsis of the report.
If you feel it appropriate (e.g. you have added, deleted, or renamed files), bundle your changes into a `tar` file. Archives created with man:shar[1] are also welcome.
If you feel it appropriate (e.g. you have added, deleted, or renamed files), bundle your changes into a `tar` file.
Archives created with man:shar[1] are also welcome.
If your change is of a potentially sensitive nature, such as if you are unsure of copyright issues governing its further distribution then you should send it to {core-email} directly rather than submitting as a bug report. The {core-email} reaches a much smaller group of people who do much of the day-to-day work on FreeBSD. Note that this group is also _very busy_ and so you should only send mail to them where it is truly necessary.
If your change is of a potentially sensitive nature, such as if you are unsure of copyright issues governing its further distribution then you should send it to {core-email} directly rather than submitting as a bug report.
The {core-email} reaches a much smaller group of people who do much of the day-to-day work on FreeBSD.
Note that this group is also _very busy_ and so you should only send mail to them where it is truly necessary.
Please refer to man:intro[9] and man:style[9] for some information on coding style. We would appreciate it if you were at least aware of this information before submitting code.
Please refer to man:intro[9] and man:style[9] for some information on coding style.
We would appreciate it if you were at least aware of this information before submitting code.
=== New Code or Major Value-Added Packages
In the case of a significant contribution of a large body work, or the addition of an important new feature to FreeBSD, it becomes almost always necessary to either send changes as tar files or upload them to a web or FTP site for other people to access. If you do not have access to a web or FTP site, ask on an appropriate FreeBSD mailing list for someone to host the changes for you.
In the case of a significant contribution of a large body work, or the addition of an important new feature to FreeBSD, it becomes almost always necessary to either send changes as tar files or upload them to a web or FTP site for other people to access.
If you do not have access to a web or FTP site, ask on an appropriate FreeBSD mailing list for someone to host the changes for you.
When working with large amounts of code, the touchy subject of copyrights also invariably comes up. FreeBSD prefers free software licenses such as BSD or ISC. Copyleft licenses such as GPLv2 are sometimes permitted. The complete listing can be found on the link:https://www.FreeBSD.org/internal/software-license/[core team licensing policy] page.
When working with large amounts of code, the touchy subject of copyrights also invariably comes up.
FreeBSD prefers free software licenses such as BSD or ISC.
Copyleft licenses such as GPLv2 are sometimes permitted.
The complete listing can be found on the link:https://www.FreeBSD.org/internal/software-license/[core team licensing policy] page.
=== Money or Hardware
@ -160,7 +206,9 @@ We are always very happy to accept donations to further the cause of the FreeBSD
[[donations]]
==== Donating Funds
The https://www.freebsdfoundation.org[FreeBSD Foundation] is a non-profit, tax-exempt foundation established to further the goals of the FreeBSD Project. As a 501(c)3 entity, the Foundation is generally exempt from US federal income tax as well as Colorado State income tax. Donations to a tax-exempt entity are often deductible from taxable federal income.
The https://www.freebsdfoundation.org[FreeBSD Foundation] is a non-profit, tax-exempt foundation established to further the goals of the FreeBSD Project.
As a 501(c)3 entity, the Foundation is generally exempt from US federal income tax as well as Colorado State income tax.
Donations to a tax-exempt entity are often deductible from taxable federal income.
Donations may be sent in check form to:
@ -175,11 +223,13 @@ USA
The FreeBSD Foundation is also able to accept https://www.freebsdfoundation.org/donate/[online donations] through various payment options.
More information about the FreeBSD Foundation can be found in https://people.FreeBSD.org/~jdp/foundation/announcement.html[The FreeBSD Foundation -- an Introduction]. To contact the Foundation by email, write to mailto:info@FreeBSDFoundation.org[info@FreeBSDFoundation.org].
More information about the FreeBSD Foundation can be found in https://people.FreeBSD.org/~jdp/foundation/announcement.html[The FreeBSD Foundation -- an Introduction].
To contact the Foundation by email, write to mailto:info@FreeBSDFoundation.org[info@FreeBSDFoundation.org].
==== Donating Hardware
The FreeBSD Project happily accepts donations of hardware that it can find good use for. If you are interested in donating hardware, please contact the link:https://www.FreeBSD.org/donations/[Donations Liaison Office].
The FreeBSD Project happily accepts donations of hardware that it can find good use for.
If you are interested in donating hardware, please contact the link:https://www.FreeBSD.org/donations/[Donations Liaison Office].
[[ports-contributing]]
== Contributing to ports
@ -189,21 +239,36 @@ The FreeBSD Project happily accepts donations of hardware that it can find good
==== Choosing an unmaintained port
Taking over maintainership of ports that are unmaintained is a great way to get involved. Unmaintained ports are only updated and fixed when somebody volunteers to work on them. There are a large number of unmaintained ports. It is a good idea to start with adopting a port that you use regularly.
Taking over maintainership of ports that are unmaintained is a great way to get involved.
Unmaintained ports are only updated and fixed when somebody volunteers to work on them.
There are a large number of unmaintained ports.
It is a good idea to start with adopting a port that you use regularly.
Unmaintained ports have their `MAINTAINER` set to `ports@FreeBSD.org`. A list of unmaintained ports and their current errors and problem reports can be seen at the http://portsmon.FreeBSD.org/portsconcordanceformaintainer.py?maintainer=ports%40FreeBSD.org[FreeBSD Ports Monitoring System].
Unmaintained ports have their `MAINTAINER` set to `ports@FreeBSD.org`.
A list of unmaintained ports and their current errors and problem reports can be seen at the http://portsmon.FreeBSD.org/portsconcordanceformaintainer.py?maintainer=ports%40FreeBSD.org[FreeBSD Ports Monitoring System].
Some ports affect a large number of others due to dependencies and slave port relationships. Generally, we want people to have some experience before they maintain such ports.
Some ports affect a large number of others due to dependencies and slave port relationships.
Generally, we want people to have some experience before they maintain such ports.
You can find out whether or not a port has dependencies or slave ports by looking at a master index of ports called [.filename]#INDEX#. (The name of the file varies by release of FreeBSD; for instance, [.filename]#INDEX-8#.) Some ports have conditional dependencies that are not included in a default [.filename]#INDEX# build. We expect you to be able to recognize such ports by looking through other ports' [.filename]#Makefile#'s.
You can find out whether or not a port has dependencies or slave ports by looking at a master index of ports called [.filename]#INDEX#.
(The name of the file varies by release of FreeBSD; for instance, [.filename]#INDEX-8#.) Some ports have conditional dependencies that are not included in a default [.filename]#INDEX# build.
We expect you to be able to recognize such ports by looking through other ports' [.filename]#Makefile#'s.
==== How to adopt the port
First make sure you understand your <<maintain-port>>. Also read the link:{porters-handbook}[Porter's Handbook]. _Please do not commit yourself to more than you feel you can comfortably handle._
First make sure you understand your <<maintain-port>>.
Also read the link:{porters-handbook}[Porter's Handbook].
_Please do not commit yourself to more than you feel you can comfortably handle._
You may request maintainership of any unmaintained port as soon as you wish. Simply set `MAINTAINER` to your own email address and send a PR (Problem Report) with the change. If the port has build errors or needs updating, you may wish to include any other changes in the same PR. This will help because many committers are less willing to assign maintainership to someone who does not have a known track record with FreeBSD. Submitting PRs that fix build errors or update ports are the best ways to establish one.
You may request maintainership of any unmaintained port as soon as you wish.
Simply set `MAINTAINER` to your own email address and send a PR (Problem Report) with the change.
If the port has build errors or needs updating, you may wish to include any other changes in the same PR.
This will help because many committers are less willing to assign maintainership to someone who does not have a known track record with FreeBSD.
Submitting PRs that fix build errors or update ports are the best ways to establish one.
File your PR with category `ports` and class `change-request`. A committer will examine your PR, commit the changes, and finally close the PR. Sometimes this process can take a little while (committers are volunteers, too :).
File your PR with category `ports` and class `change-request`.
A committer will examine your PR, commit the changes, and finally close the PR.
Sometimes this process can take a little while (committers are volunteers, too :).
[[maintain-port]]
=== The challenge for port maintainers
@ -213,9 +278,12 @@ This section will give you an idea of why ports need to be maintained and outlin
[[why-maintenance]]
==== Why ports require maintenance
Creating a port is a once-off task. Ensuring that a port is up to date and continues to build and run requires an ongoing maintenance effort. Maintainers are the people who dedicate some of their time to meeting these goals.
Creating a port is a once-off task.
Ensuring that a port is up to date and continues to build and run requires an ongoing maintenance effort.
Maintainers are the people who dedicate some of their time to meeting these goals.
The foremost reason ports need maintenance is to bring the latest and greatest in third party software to the FreeBSD community. An additional challenge is to keep individual ports working within the Ports Collection framework as it evolves.
The foremost reason ports need maintenance is to bring the latest and greatest in third party software to the FreeBSD community.
An additional challenge is to keep individual ports working within the Ports Collection framework as it evolves.
As a maintainer, you will need to manage the following challenges:
@ -239,18 +307,24 @@ As a maintainer, you will need to manage the following challenges:
This section outlines the process to follow to keep your ports up to date.
This is an overview. More information about upgrading a port is available in the link:{porters-handbook}[Porter's Handbook].
This is an overview.
More information about upgrading a port is available in the link:{porters-handbook}[Porter's Handbook].
[.procedure]
====
. Watch for updates
+
Monitor the upstream vendor for new versions, updates and security fixes for the software. Announcement mailing lists or news web pages are useful for doing this. Sometimes users will contact you and ask when your port will be updated. If you are busy with other things or for any reason just cannot update it at the moment, ask if they will help you by submitting an update.
Monitor the upstream vendor for new versions, updates and security fixes for the software.
Announcement mailing lists or news web pages are useful for doing this.
Sometimes users will contact you and ask when your port will be updated.
If you are busy with other things or for any reason just cannot update it at the moment, ask if they will help you by submitting an update.
+
You may also receive automated email from the `FreeBSD Ports Version Check` informing you that a newer version of your port's distfile is available. More information about that system (including how to stop future emails) will be provided in the message.
You may also receive automated email from the `FreeBSD Ports Version Check` informing you that a newer version of your port's distfile is available.
More information about that system (including how to stop future emails) will be provided in the message.
. Incorporate changes
+
When they become available, incorporate the changes into the port. You need to be able to generate a patch between the original port and your updated port.
When they become available, incorporate the changes into the port.
You need to be able to generate a patch between the original port and your updated port.
. Review and test
+
Thoroughly review and test your changes:
@ -263,34 +337,44 @@ Thoroughly review and test your changes:
. Submit changes
+
Send your update by submitting a PR with an explanation of the changes and a patch containing the differences between the original port and the updated one. Please refer to link:{problem-reports}[Writing FreeBSD Problem Reports] for information on how to write a really good PR.
Send your update by submitting a PR with an explanation of the changes and a patch containing the differences between the original port and the updated one.
Please refer to link:{problem-reports}[Writing FreeBSD Problem Reports] for information on how to write a really good PR.
+
[NOTE]
======
Please do not submit a man:shar[1] archive of the entire port; instead, use man:diff[1] `-ruN`. In this way, committers can much more easily see exactly what changes are being made. The Porter's Handbook section on link:{porters-handbook}#port-upgrading[Upgrading] has more information.
Please do not submit a man:shar[1] archive of the entire port; instead, use man:diff[1] `-ruN`.
In this way, committers can much more easily see exactly what changes are being made.
The Porter's Handbook section on link:{porters-handbook}#port-upgrading[Upgrading] has more information.
======
. Wait
+
At some stage a committer will deal with your PR. It may take minutes, or it may take weeks - so please be patient.
At some stage a committer will deal with your PR.
It may take minutes, or it may take weeks - so please be patient.
. Give feedback
+
If a committer finds a problem with your changes, they will most likely refer it back to you. A prompt response will help get your PR committed faster, and is better for maintaining a thread of conversation when trying to resolve any problems.
If a committer finds a problem with your changes, they will most likely refer it back to you.
A prompt response will help get your PR committed faster, and is better for maintaining a thread of conversation when trying to resolve any problems.
. And Finally
+
Your changes will be committed and your port will have been updated. The PR will then be closed by the committer. That's it!
Your changes will be committed and your port will have been updated.
The PR will then be closed by the committer. That's it!
====
===== Ensure your ports continue to build correctly
This section is about discovering and fixing problems that stop your ports from building correctly.
FreeBSD only guarantees that the Ports Collection works on the `-STABLE` branches. In theory, you should be able to get by with running the latest release of each stable branch (since the ABIs are not supposed to change) but if you can run the branch, that is even better.
FreeBSD only guarantees that the Ports Collection works on the `-STABLE` branches.
In theory, you should be able to get by with running the latest release of each stable branch (since the ABIs are not supposed to change) but if you can run the branch, that is even better.
Since the majority of FreeBSD installations run on PC-compatible machines (what is termed the `i386` architecture), we expect you to keep the port working on that architecture. We prefer that ports also work on the `amd64` architecture running native. It is completely fair to ask for help if you do not have one of these machines.
Since the majority of FreeBSD installations run on PC-compatible machines (what is termed the `i386` architecture), we expect you to keep the port working on that architecture.
We prefer that ports also work on the `amd64` architecture running native.
It is completely fair to ask for help if you do not have one of these machines.
[NOTE]
====
The usual failure modes for non-`x86` machines are that the original programmers assumed that, for instance, pointers are `int`-s, or that a relatively lax older gcc compiler was being used. More and more, application authors are reworking their code to remove these assumptions - but if the author is not actively maintaining their code, you may need to do this yourself.
The usual failure modes for non-`x86` machines are that the original programmers assumed that, for instance, pointers are `int`-s, or that a relatively lax older gcc compiler was being used.
More and more, application authors are reworking their code to remove these assumptions - but if the author is not actively maintaining their code, you may need to do this yourself.
====
These are the tasks you need to perform to ensure your port is able to be built:
@ -302,7 +386,9 @@ These are the tasks you need to perform to ensure your port is able to be built:
Check your mail for mail from `pkg-fallout@FreeBSD.org` and the http://portscout.FreeBSD.org[distfiles scanner] to see if any of the port which are failing to build are out of date.
. Collect information
+
Once you are aware of a problem, collect information to help you fix it. Build errors reported by `pkg-fallout` are accompanied by logs which will show you where the build failed. If the failure was reported to you by a user, ask them to send you information which may help in diagnosing the problem, such as:
Once you are aware of a problem, collect information to help you fix it.
Build errors reported by `pkg-fallout` are accompanied by logs which will show you where the build failed.
If the failure was reported to you by a user, ask them to send you information which may help in diagnosing the problem, such as:
** Build logs
** The commands and options used to build the port (including options set in [.filename]#/etc/make.conf#)
@ -313,20 +399,25 @@ Once you are aware of a problem, collect information to help you fix it. Build e
. Investigate and find a solution
+
Unfortunately there is no straightforward process to follow to do this. Remember, though: if you are stuck, ask for help! The {freebsd-ports} is a good place to start, and the upstream developers are often very helpful.
Unfortunately there is no straightforward process to follow to do this.
Remember, though: if you are stuck, ask for help! The {freebsd-ports} is a good place to start, and the upstream developers are often very helpful.
. Submit changes
+
Just as with updating a port, you should now incorporate changes, review and test, submit your changes in a PR, and provide feedback if required.
. Send patches to upstream authors
+
In some cases, you will have to make patches to the port to make it run on FreeBSD. Some (but not all) upstream authors will accept such patches back into their code for the next release. If so, this may even help their users on other BSD-based systems as well and perhaps save duplicated effort. Please consider sending any applicable patches to the authors as a courtesy.
In some cases, you will have to make patches to the port to make it run on FreeBSD.
Some (but not all) upstream authors will accept such patches back into their code for the next release.
If so, this may even help their users on other BSD-based systems as well and perhaps save duplicated effort.
Please consider sending any applicable patches to the authors as a courtesy.
====
===== Investigate bug reports and PRs related to your port
This section is about discovering and fixing bugs.
FreeBSD-specific bugs are generally caused by assumptions about the build and runtime environments that do not apply to FreeBSD. You are less likely to encounter a problem of this type, but it can be more subtle and difficult to diagnose.
FreeBSD-specific bugs are generally caused by assumptions about the build and runtime environments that do not apply to FreeBSD.
You are less likely to encounter a problem of this type, but it can be more subtle and difficult to diagnose.
These are the tasks you need to perform to ensure your port continues to work as intended:
@ -334,16 +425,19 @@ These are the tasks you need to perform to ensure your port continues to work as
====
. Respond to bug reports
+
Bugs may be reported to you through email via the https://bugs.FreeBSD.org/search/[Problem Report database]. Bugs may also be reported directly to you by users.
Bugs may be reported to you through email via the https://bugs.FreeBSD.org/search/[Problem Report database].
Bugs may also be reported directly to you by users.
+
You should respond to PRs and other reports within 14 days, but please try not to take that long. Try to respond as soon as possible, even if it is just to say you need some more time before you can work on the PR.
You should respond to PRs and other reports within 14 days, but please try not to take that long.
Try to respond as soon as possible, even if it is just to say you need some more time before you can work on the PR.
+
If you have not responded after 14 days, any committer may commit from a PR that you have not responded to via a `maintainer-timeout`.
. Collect information
+
If the person reporting the bug has not also provided a fix, you need to collect the information that will allow you to generate one.
+
If the bug is reproducible, you can collect most of the required information yourself. If not, ask the person who reported the bug to collect the information for you, such as:
If the bug is reproducible, you can collect most of the required information yourself.
If not, ask the person who reported the bug to collect the information for you, such as:
** A detailed description of their actions, expected program behavior and actual behavior
** Copies of input data used to trigger the bug
@ -353,28 +447,43 @@ If the bug is reproducible, you can collect most of the required information you
. Eliminate incorrect reports
+
Some bug reports may be incorrect. For example, the user may have simply misused the program; or their installed packages may be out of date and require updating. Sometimes a reported bug is not specific to FreeBSD. In this case report the bug to the upstream developers. If the bug is within your capabilities to fix, you can also patch the port so that the fix is applied before the next upstream release.
Some bug reports may be incorrect.
For example, the user may have simply misused the program; or their installed packages may be out of date and require updating.
Sometimes a reported bug is not specific to FreeBSD.
In this case report the bug to the upstream developers.
If the bug is within your capabilities to fix, you can also patch the port so that the fix is applied before the next upstream release.
. Find a solution
+
As with build errors, you will need to sort out a fix to the problem. Again, remember to ask if you are stuck!
As with build errors, you will need to sort out a fix to the problem.
Again, remember to ask if you are stuck!
. Submit or approve changes
+
Just as with updating a port, you should now incorporate changes, review and test, and submit your changes in a PR (or send a follow-up if a PR already exists for the problem). If another user has submitted changes in the PR, you can also send a follow-up saying whether or not you approve the changes.
Just as with updating a port, you should now incorporate changes, review and test, and submit your changes in a PR (or send a follow-up if a PR already exists for the problem).
If another user has submitted changes in the PR, you can also send a follow-up saying whether or not you approve the changes.
====
===== Providing support
Part of being a maintainer is providing support - not for the software in general - but for the port and any FreeBSD-specific quirks and problems. Users may contact you with questions, suggestions, problems and patches. Most of the time their correspondence will be specific to FreeBSD.
Part of being a maintainer is providing support - not for the software in general - but for the port and any FreeBSD-specific quirks and problems.
Users may contact you with questions, suggestions, problems and patches.
Most of the time their correspondence will be specific to FreeBSD.
Occasionally you may have to invoke your skills in diplomacy, and kindly point users seeking general support to the appropriate resources. Less frequently you will encounter a person asking why the `RPMS` are not up to date or how can they get the software to run under Foo Linux. Take the opportunity to tell them that your port is up to date (if it is, of course!), and suggest that they try FreeBSD.
Occasionally you may have to invoke your skills in diplomacy, and kindly point users seeking general support to the appropriate resources.
Less frequently you will encounter a person asking why the `RPMS` are not up to date or how can they get the software to run under Foo Linux.
Take the opportunity to tell them that your port is up to date (if it is, of course!), and suggest that they try FreeBSD.
Sometimes users and developers will decide that you are a busy person whose time is valuable and do some of the work for you. For example, they might:
Sometimes users and developers will decide that you are a busy person whose time is valuable and do some of the work for you.
For example, they might:
* submit a PR or send you patches to update your port,
* investigate and perhaps provide a fix to a PR, or
* otherwise submit changes to your port.
In these cases your main obligation is to respond in a timely manner. Again, the timeout for non-responsive maintainers is 14 days. After this period changes may be committed unapproved. They have taken the trouble to do this for you; so please try to at least respond promptly. Then review, approve, modify or discuss their changes with them as soon as possible.
In these cases your main obligation is to respond in a timely manner.
Again, the timeout for non-responsive maintainers is 14 days.
After this period changes may be committed unapproved.
They have taken the trouble to do this for you; so please try to at least respond promptly.
Then review, approve, modify or discuss their changes with them as soon as possible.
If you can make them feel that their contribution is appreciated (and it should be) you will have a better chance persuading them to do more things for you in the future :-).
@ -383,37 +492,57 @@ If you can make them feel that their contribution is appreciated (and it should
There are two really good places to find a port that needs some attention.
You can use the https://bugs.freebsd.org/search[web interface] to the Problem Report database to search through and view unresolved PRs. The majority of ports PRs are updates, but with a little searching and skimming over synopses you should be able to find something interesting to work on (the `sw-bug` class is a good place to start).
You can use the https://bugs.freebsd.org/search[web interface] to the Problem Report database to search through and view unresolved PRs.
The majority of ports PRs are updates, but with a little searching and skimming over synopses you should be able to find something interesting to work on (the `sw-bug` class is a good place to start).
The other place is the http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System]. In particular look for unmaintained ports with build errors and ports that are marked `BROKEN`. It is OK to send changes for a maintained port as well, but remember to ask the maintainer in case they are already working on the problem.
The other place is the http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System].
In particular look for unmaintained ports with build errors and ports that are marked `BROKEN`.
It is OK to send changes for a maintained port as well, but remember to ask the maintainer in case they are already working on the problem.
Once you have found a bug or problem, collect information, investigate and fix! If there is an existing PR, follow up to that. Otherwise create a new PR. Your changes will be reviewed and, if everything checks out, committed.
Once you have found a bug or problem, collect information, investigate and fix! If there is an existing PR, follow up to that.
Otherwise create a new PR.
Your changes will be reviewed and, if everything checks out, committed.
[[mortal-coil]]
=== When to call it quits
As your interests and commitments change, you may find that you no longer have time to continue some (or all) of your ports contributions. That is fine! Please let us know if you are no longer using a port or have otherwise lost time or interest in being a maintainer. In this way we can go ahead and allow other people to try to work on existing problems with the port without waiting for your response. Remember, FreeBSD is a volunteer project, so if maintaining a port is no fun any more, it is probably time to let someone else do it!
As your interests and commitments change, you may find that you no longer have time to continue some (or all) of your ports contributions.
That is fine! Please let us know if you are no longer using a port or have otherwise lost time or interest in being a maintainer.
In this way we can go ahead and allow other people to try to work on existing problems with the port without waiting for your response.
Remember, FreeBSD is a volunteer project, so if maintaining a port is no fun any more, it is probably time to let someone else do it!
In any case, the Ports Management Team (`portmgr`) reserves the right to reset your maintainership if you have not actively maintained your port in some time. (Currently, this is set to 3 months.) By this, we mean that there are unresolved problems or pending updates that have not been worked on during that time.
In any case, the Ports Management Team (`portmgr`) reserves the right to reset your maintainership if you have not actively maintained your port in some time.
(Currently, this is set to 3 months.)
By this, we mean that there are unresolved problems or pending updates that have not been worked on during that time.
[[resources]]
=== Resources for ports maintainers and contributors
The link:{porters-handbook}[Porter's Handbook] is your hitchhiker's guide to the ports system. Keep it handy!
link:{problem-reports}[Writing FreeBSD Problem Reports] describes how to best formulate and submit a PR. In 2005 more than eleven thousand ports PRs were submitted! Following this article will greatly assist us in reducing the time needed to handle your PRs.
link:{problem-reports}[Writing FreeBSD Problem Reports] describes how to best formulate and submit a PR.
In 2005 more than eleven thousand ports PRs were submitted! Following this article will greatly assist us in reducing the time needed to handle your PRs.
The https://bugs.freebsd.org/bugzilla/query.cgi[Problem Report database].
The http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System] can show you cross-referenced information about ports such as build errors and problem reports. If you are a maintainer you can use it to check on the build status of your ports. As a contributor you can use it to find broken and unmaintained ports that need to be fixed.
The http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System] can show you cross-referenced information about ports such as build errors and problem reports.
If you are a maintainer you can use it to check on the build status of your ports.
As a contributor you can use it to find broken and unmaintained ports that need to be fixed.
The http://portscout.FreeBSD.org[FreeBSD Ports distfile scanner] can show you ports for which the distfiles are not fetchable. You can check on your own ports or use it to find ports that need their `MASTER_SITES` updated.
The http://portscout.FreeBSD.org[FreeBSD Ports distfile scanner] can show you ports for which the distfiles are not fetchable.
You can check on your own ports or use it to find ports that need their `MASTER_SITES` updated.
package:ports-mgmt/poudriere[] is the most thorough way to test a port through the entire cycle of installation, packaging, and deinstallation. Documentation is located at the https://github.com/freebsd/poudriere[poudriere github repository]
package:ports-mgmt/poudriere[] is the most thorough way to test a port through the entire cycle of installation, packaging, and deinstallation.
Documentation is located at the https://github.com/freebsd/poudriere[poudriere github repository]
man:portlint[1] is an application which can be used to verify that your port conforms to many important stylistic and functional guidelines. portlint is a simple heuristic application, so you should use it __only as a guide__. If portlint suggests changes which seem unreasonable, consult the link:{porters-handbook}[Porter's Handbook] or ask for advice.
man:portlint[1] is an application which can be used to verify that your port conforms to many important stylistic and functional guidelines.
portlint is a simple heuristic application, so you should use it __only as a guide__.
If portlint suggests changes which seem unreasonable, consult the link:{porters-handbook}[Porter's Handbook] or ask for advice.
The {freebsd-ports} is for general ports-related discussion. It is a good place to ask for help. You can https://lists.freebsd.org/mailman/listinfo[subscribe, or read and search the list archives]. Reading the archives of the {freebsd-ports-bugs} and the {svn-ports-head} may also be of interest.
The {freebsd-ports} is for general ports-related discussion.
It is a good place to ask for help.
You can https://lists.freebsd.org/mailman/listinfo[subscribe, or read and search the list archives].
Reading the archives of the {freebsd-ports-bugs} and the {svn-ports-head} may also be of interest.
[[ideas-contributing]]
== Getting Started in Other Areas

18
documentation/content/en/articles/contributors/_index.adoc
View file

@ -96,7 +96,8 @@ The following individuals and businesses have generously contributed hardware fo
[[staff-committers]]
== The FreeBSD Developers
These are the people who have commit privileges and do the engineering work on the FreeBSD source tree. All core team members are also developers.
These are the people who have commit privileges and do the engineering work on the FreeBSD source tree.
All core team members are also developers.
(in alphabetical order by last name):
@ -105,7 +106,8 @@ include::content/en/articles/contributors/contrib-committers.adoc[]
[[contrib-corealumni]]
== Core Team Alumni
The following people were members of the FreeBSD core team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project.
The following people were members of the FreeBSD core team during the periods indicated.
We thank them for their past efforts in the service of the FreeBSD project.
_In rough reverse chronological order:_
@ -114,7 +116,8 @@ include::content/en/articles/contributors/contrib-corealumni.adoc[]
[[contrib-develalumni]]
== Development Team Alumni
The following people were members of the FreeBSD development team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project.
The following people were members of the FreeBSD development team during the periods indicated.
We thank them for their past efforts in the service of the FreeBSD project.
_In rough reverse chronological order:_
@ -123,7 +126,8 @@ include::content/en/articles/contributors/contrib-develalumni.adoc[]
[[contrib-portmgralumni]]
== Ports Management Team Alumni
The following people were members of the FreeBSD portmgr team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project.
The following people were members of the FreeBSD portmgr team during the periods indicated.
We thank them for their past efforts in the service of the FreeBSD project.
_In rough reverse chronological order:_
@ -132,7 +136,8 @@ include::content/en/articles/contributors/contrib-portmgralumni.adoc[]
[[contrib-develinmemoriam]]
== Development Team: In Memoriam
During the many years that the FreeBSD Project has been in existence, sadly, some of our developers have passed away. Here are some remembrances.
During the many years that the FreeBSD Project has been in existence, sadly, some of our developers have passed away.
Here are some remembrances.
_In rough reverse chronological order of their passing:_
@ -141,7 +146,8 @@ include::content/en/articles/contributors/contrib-develinmemoriam.adoc[]
[[contrib-derived]]
== Derived Software Contributors
This software was originally derived from William F. Jolitz's 386BSD release 0.1, though almost none of the original 386BSD specific code remains. This software has been essentially re-implemented from the 4.4BSD-Lite release provided by the Computer Science Research Group (CSRG) at the University of California, Berkeley and associated academic contributors.
This software was originally derived from William F. Jolitz's 386BSD release 0.1, though almost none of the original 386BSD specific code remains.
This software has been essentially re-implemented from the 4.4BSD-Lite release provided by the Computer Science Research Group (CSRG) at the University of California, Berkeley and associated academic contributors.
There are also portions of NetBSD and OpenBSD that have been integrated into FreeBSD as well, and we would therefore like to thank all the contributors to NetBSD and OpenBSD for their work.

50
documentation/content/en/articles/cups/_index.adoc
View file

@ -29,9 +29,13 @@ toc::[]
[[printing-cups]]
== An Introduction to the Common Unix Printing System (CUPS)
CUPS, the Common UNIX Printing System, provides a portable printing layer for UNIX(R)-based operating systems. It has been developed by Easy Software Products to promote a standard printing solution for all UNIX(R) vendors and users.
CUPS, the Common UNIX Printing System, provides a portable printing layer for UNIX(R)-based operating systems.
It has been developed by Easy Software Products to promote a standard printing solution for all UNIX(R) vendors and users.
CUPS uses the Internet Printing Protocol (IPP) as the basis for managing print jobs and queues. The Line Printer Daemon (LPD), Server Message Block (SMB), and AppSocket (aka JetDirect) protocols are also supported with reduced functionality. CUPS adds network printer browsing and PostScript Printer Description (PPD) based printing options to support real-world printing under UNIX(R). As a result, CUPS is ideally-suited for sharing and accessing printers in mixed environments of FreeBSD, Linux(R), Mac OS(R) X, or Windows(R).
CUPS uses the Internet Printing Protocol (IPP) as the basis for managing print jobs and queues.
The Line Printer Daemon (LPD), Server Message Block (SMB), and AppSocket (aka JetDirect) protocols are also supported with reduced functionality.
CUPS adds network printer browsing and PostScript Printer Description (PPD) based printing options to support real-world printing under UNIX(R).
As a result, CUPS is ideally-suited for sharing and accessing printers in mixed environments of FreeBSD, Linux(R), Mac OS(R) X, or Windows(R).
The main site for CUPS is http://www.cups.org/[http://www.cups.org/].
@ -45,12 +49,14 @@ To install CUPS using a precompiled binary, issue the following command from a r
# pkg install cups
....
Other optional, but recommended, packages are package:print/gutenprint[] and package:print/hplip[], both of which add drivers and utilities for a variety of printers. Once installed, the CUPS configuration files can be found in the directory [.filename]#/usr/local/etc/cups#.
Other optional, but recommended, packages are package:print/gutenprint[] and package:print/hplip[], both of which add drivers and utilities for a variety of printers.
Once installed, the CUPS configuration files can be found in the directory [.filename]#/usr/local/etc/cups#.
[[printing-cups-configuring-server]]
== Configuring the CUPS Print Server
After installation, a few files must be edited in order to configure the CUPS server. First, create or modify, as the case may be, the file [.filename]#/etc/devfs.rules# and add the following information to set the proper permissions on all potential printer devices and to associate printers with the `cups` user group:
After installation, a few files must be edited in order to configure the CUPS server.
First, create or modify, as the case may be, the file [.filename]#/etc/devfs.rules# and add the following information to set the proper permissions on all potential printer devices and to associate printers with the `cups` user group:
[.programlisting]
....
@ -63,7 +69,8 @@ add path 'usb/X.Y.Z' mode 0660 group cups
[NOTE]
====
Note that _X_, _Y_, and _Z_ should be replaced with the target USB device listed in the [.filename]#/dev/usb# directory that corresponds to the printer. To find the correct device, examine the output of man:dmesg[8], where [.filename]#ugenX.Y# lists the printer device, which is a symbolic link to a USB device in [.filename]#/dev/usb#.
Note that _X_, _Y_, and _Z_ should be replaced with the target USB device listed in the [.filename]#/dev/usb# directory that corresponds to the printer.
To find the correct device, examine the output of man:dmesg[8], where [.filename]#ugenX.Y# lists the printer device, which is a symbolic link to a USB device in [.filename]#/dev/usb#.
====
Next, add two lines to [.filename]#/etc/rc.conf# as follows:
@ -94,21 +101,31 @@ Once these changes have been made, the man:devfs[8] and CUPS systems must both b
[[printing-cups-configuring-printers]]
== Configuring Printers on the CUPS Print Server
After the CUPS system has been installed and configured, the administrator can begin configuring the local printers attached to the CUPS print server. This part of the process is very similar, if not identical, to configuring CUPS printers on other UNIX(R)-based operating systems, such as a Linux(R) distribution.
After the CUPS system has been installed and configured, the administrator can begin configuring the local printers attached to the CUPS print server.
This part of the process is very similar, if not identical, to configuring CUPS printers on other UNIX(R)-based operating systems, such as a Linux(R) distribution.
The primary means for managing and administering the CUPS server is through the web-based interface, which can be found by launching a web browser and entering http://localhost:631[http://localhost:631] in the browser's URL bar. If the CUPS server is on another machine on the network, substitute the server's local IP address for `localhost`. The CUPS web interface is fairly self-explanatory, as there are sections for managing printers and print jobs, authorizing users, and more. Additionally, on the right-hand side of the Administration screen are several check-boxes allowing easy access to commonly-changed settings, such as whether to share published printers connected to the system, whether to allow remote administration of the CUPS server, and whether to allow users additional access and privileges to the printers and print jobs.
The primary means for managing and administering the CUPS server is through the web-based interface, which can be found by launching a web browser and entering http://localhost:631[http://localhost:631] in the browser's URL bar.
If the CUPS server is on another machine on the network, substitute the server's local IP address for `localhost`.
The CUPS web interface is fairly self-explanatory, as there are sections for managing printers and print jobs, authorizing users, and more.
Additionally, on the right-hand side of the Administration screen are several check-boxes allowing easy access to commonly-changed settings, such as whether to share published printers connected to the system, whether to allow remote administration of the CUPS server, and whether to allow users additional access and privileges to the printers and print jobs.
Adding a printer is generally as easy as clicking "Add Printer" at the Administration screen of the CUPS web interface, or clicking one of the "New Printers Found" buttons also at the Administration screen. When presented with the "Device" drop-down box, simply select the desired locally-attached printer, and then continue through the process. If one has added the package:print/gutenprint-cups[] or package:print/hplip[] ports or packages as referenced above, then additional print drivers will be available in the subsequent screens that might provide more stability or features.
Adding a printer is generally as easy as clicking "Add Printer" at the Administration screen of the CUPS web interface, or clicking one of the "New Printers Found" buttons also at the Administration screen.
When presented with the "Device" drop-down box, simply select the desired locally-attached printer, and then continue through the process.
If one has added the package:print/gutenprint-cups[] or package:print/hplip[] ports or packages as referenced above, then additional print drivers will be available in the subsequent screens that might provide more stability or features.
[[printing-cups-clients]]
== Configuring CUPS Clients
Once the CUPS server has been configured and printers have been added and published to the network, the next step is to configure the clients, or the machines that are going to access the CUPS server. If one has a single desktop machine that is acting as both server and client, then much of this information may not be needed.
Once the CUPS server has been configured and printers have been added and published to the network, the next step is to configure the clients, or the machines that are going to access the CUPS server.
If one has a single desktop machine that is acting as both server and client, then much of this information may not be needed.
[[printing-cups-clients-unix]]
=== UNIX(R) Clients
CUPS will also need to be installed on your UNIX(R) clients. Once CUPS is installed on the clients, then CUPS printers that are shared across the network are often automatically discovered by the printer managers for various desktop environments such as GNOME or KDE. Alternatively, one can access the local CUPS interface on the client machine at http://localhost:631[http://localhost:631] and click on "Add Printer" in the Administration section. When presented with the "Device" drop-down box, simply select the networked CUPS printer, if it was automatically discovered, or select `ipp` or `http` and enter the IPP or HTTP URI of the networked CUPS printer, usually in one of the two following syntaxes:
CUPS will also need to be installed on your UNIX(R) clients.
Once CUPS is installed on the clients, then CUPS printers that are shared across the network are often automatically discovered by the printer managers for various desktop environments such as GNOME or KDE.
Alternatively, one can access the local CUPS interface on the client machine at http://localhost:631[http://localhost:631] and click on "Add Printer" in the Administration section.
When presented with the "Device" drop-down box, simply select the networked CUPS printer, if it was automatically discovered, or select `ipp` or `http` and enter the IPP or HTTP URI of the networked CUPS printer, usually in one of the two following syntaxes:
[.programlisting]
....
@ -132,7 +149,10 @@ In this case, _server-ip_ would be replaced by the local IP address of the CUPS
[[printing-cups-clients-windows]]
=== Windows(R) Clients
Versions of Windows(R) prior to XP did not have the capability to natively network with IPP-based printers. However, Windows(R) XP and later versions do have this capability. Therefore, to add a CUPS printer in these versions of Windows(R) is quite easy. Generally, the Windows(R) administrator will run the Windows(R) `Add Printer` wizard, select `Network Printer` and then enter the URI in the following syntax:
Versions of Windows(R) prior to XP did not have the capability to natively network with IPP-based printers.
However, Windows(R) XP and later versions do have this capability.
Therefore, to add a CUPS printer in these versions of Windows(R) is quite easy.
Generally, the Windows(R) administrator will run the Windows(R) `Add Printer` wizard, select `Network Printer` and then enter the URI in the following syntax:
[.programlisting]
....
@ -144,7 +164,13 @@ If one has an older version of Windows(R) without native IPP printing support, t
[[printing-cups-troubleshooting]]
== CUPS Troubleshooting
Difficulties with CUPS often lies in permissions. First, double check the man:devfs[8] permissions as outlined above. Next, check the actual permissions of the devices created in the file system. It is also helpful to make sure your user is a member of the `cups` group. If the permissions check boxes in the Administration section of the CUPS web interface do not seem to be working, another fix might be to manually backup the main CUPS configuration file located at [.filename]#/usr/local/etc/cups/cupsd.conf# and edit the various configuration options and try different combinations of configuration options. One sample [.filename]#/usr/local/etc/cups/cupsd.conf# to test is listed below. Please note that this sample [.filename]#cupsd.conf# sacrifices security for easier configuration; once the administrator successfully connects to the CUPS server and configures the clients, it is advisable to revisit this configuration file and begin locking down access.
Difficulties with CUPS often lies in permissions.
First, double check the man:devfs[8] permissions as outlined above.
Next, check the actual permissions of the devices created in the file system.
It is also helpful to make sure your user is a member of the `cups` group.
If the permissions check boxes in the Administration section of the CUPS web interface do not seem to be working, another fix might be to manually backup the main CUPS configuration file located at [.filename]#/usr/local/etc/cups/cupsd.conf# and edit the various configuration options and try different combinations of configuration options.
One sample [.filename]#/usr/local/etc/cups/cupsd.conf# to test is listed below.
Please note that this sample [.filename]#cupsd.conf# sacrifices security for easier configuration; once the administrator successfully connects to the CUPS server and configures the clients, it is advisable to revisit this configuration file and begin locking down access.
[.programlisting]
....

92
documentation/content/en/articles/explaining-bsd/_index.adoc
View file

@ -33,7 +33,11 @@ toc::[]
[[what-is-bsd]]
== What is BSD?
BSD stands for "Berkeley Software Distribution". It is the name of distributions of source code from the University of California, Berkeley, which were originally extensions to AT&T's Research UNIX(R) operating system. Several open source operating system projects are based on a release of this source code known as 4.4BSD-Lite. In addition, they comprise a number of packages from other Open Source projects, including notably the GNU project. The overall operating system comprises:
BSD stands for "Berkeley Software Distribution".
It is the name of distributions of source code from the University of California, Berkeley, which were originally extensions to AT&T's Research UNIX(R) operating system.
Several open source operating system projects are based on a release of this source code known as 4.4BSD-Lite.
In addition, they comprise a number of packages from other Open Source projects, including notably the GNU project.
The overall operating system comprises:
* The BSD kernel, which handles process scheduling, memory management, symmetric multi-processing (SMP), device drivers, etc.
* The C library, the base API for the system.
@ -44,23 +48,44 @@ __The BSD C library is based on code from Berkeley, not the GNU project.__
__Some of the utilities are derived from the GNU project, others are not.__
* The X Window system, which handles graphical display.
+
The X Window system used in most versions of BSD is maintained by the http://www.X.org/[X.Org project]. FreeBSD allows the user to choose from a variety of desktop environments, such as Gnome, KDE, or Xfce; and lightweight window managers like Openbox, Fluxbox, or Awesome.
The X Window system used in most versions of BSD is maintained by the http://www.X.org/[X.Org project].
FreeBSD allows the user to choose from a variety of desktop environments, such as Gnome, KDE, or Xfce; and lightweight window managers like Openbox, Fluxbox, or Awesome.
* Many other programs and utilities.
[[what-a-real-unix]]
== What, a real UNIX(R)?
The BSD operating systems are not clones, but open source derivatives of AT&T's Research UNIX(R) operating system, which is also the ancestor of the modern UNIX(R) System V. This may surprise you. How could that happen when AT&T has never released its code as open source?
The BSD operating systems are not clones, but open source derivatives of AT&T's Research UNIX(R) operating system, which is also the ancestor of the modern UNIX(R) System V.
This may surprise you.
How could that happen when AT&T has never released its code as open source?
It is true that AT&T UNIX(R) is not open source, and in a copyright sense BSD is very definitely _not_ UNIX(R), but on the other hand, AT&T has imported sources from other projects, noticeably the Computer Sciences Research Group (CSRG) of the University of California in Berkeley, CA. Starting in 1976, the CSRG started releasing tapes of their software, calling them _Berkeley Software Distribution_ or __BSD__.
Initial BSD releases consisted mainly of user programs, but that changed dramatically when the CSRG landed a contract with the Defense Advanced Research Projects Agency (DARPA) to upgrade the communications protocols on their network, ARPANET. The new protocols were known as the __Internet Protocols__, later _TCP/IP_ after the most important protocols. The first widely distributed implementation was part of 4.2BSD, in 1982.
Initial BSD releases consisted mainly of user programs, but that changed dramatically when the CSRG landed a contract with the Defense Advanced Research Projects Agency (DARPA) to upgrade the communications protocols on their network, ARPANET.
The new protocols were known as the __Internet Protocols__, later _TCP/IP_ after the most important protocols.
The first widely distributed implementation was part of 4.2BSD, in 1982.
In the course of the 1980s, a number of new workstation companies sprang up. Many preferred to license UNIX(R) rather than developing operating systems for themselves. In particular, Sun Microsystems licensed UNIX(R) and implemented a version of 4.2BSD, which they called SunOS(TM). When AT&T themselves were allowed to sell UNIX(R) commercially, they started with a somewhat bare-bones implementation called System III, to be quickly followed by System V. The System V code base did not include networking, so all implementations included additional software from the BSD, including the TCP/IP software, but also utilities such as the _csh_ shell and the _vi_ editor. Collectively, these enhancements were known as the __Berkeley Extensions__.
In the course of the 1980s, a number of new workstation companies sprang up.
Many preferred to license UNIX(R) rather than developing operating systems for themselves.
In particular, Sun Microsystems licensed UNIX(R) and implemented a version of 4.2BSD, which they called SunOS(TM).
When AT&T themselves were allowed to sell UNIX(R) commercially, they started with a somewhat bare-bones implementation called System III, to be quickly followed by System V.
The System V code base did not include networking, so all implementations included additional software from the BSD, including the TCP/IP software, but also utilities such as the _csh_ shell and the _vi_ editor.
Collectively, these enhancements were known as the __Berkeley Extensions__.
The BSD tapes contained AT&T source code and thus required a UNIX(R) source license. By 1990, the CSRG's funding was running out, and it faced closure. Some members of the group decided to release the BSD code, which was Open Source, without the AT&T proprietary code. This finally happened with the __Networking Tape 2__, usually known as __Net/2__. Net/2 was not a complete operating system: about 20% of the kernel code was missing. One of the CSRG members, William F. Jolitz, wrote the remaining code and released it in early 1992 as __386BSD__. At the same time, another group of ex-CSRG members formed a commercial company called http://www.bsdi.com/[Berkeley Software Design Inc.] and released a beta version of an operating system called http://www.bsdi.com/[BSD/386], which was based on the same sources. The name of the operating system was later changed to BSD/OS.
The BSD tapes contained AT&T source code and thus required a UNIX(R) source license.
By 1990, the CSRG's funding was running out, and it faced closure.
Some members of the group decided to release the BSD code, which was Open Source, without the AT&T proprietary code.
This finally happened with the __Networking Tape 2__, usually known as __Net/2__.
Net/2 was not a complete operating system: about 20% of the kernel code was missing.
One of the CSRG members, William F. Jolitz, wrote the remaining code and released it in early 1992 as __386BSD__.
At the same time, another group of ex-CSRG members formed a commercial company called http://www.bsdi.com/[Berkeley Software Design Inc.] and released a beta version of an operating system called http://www.bsdi.com/[BSD/386], which was based on the same sources.
The name of the operating system was later changed to BSD/OS.
386BSD never became a stable operating system. Instead, two other projects split off from it in 1993: http://www.NetBSD.org/[NetBSD] and link:https://www.FreeBSD.org/[FreeBSD]. The two projects originally diverged due to differences in patience waiting for improvements to 386BSD: the NetBSD people started early in the year, and the first version of FreeBSD was not ready until the end of the year. In the meantime, the code base had diverged sufficiently to make it difficult to merge. In addition, the projects had different aims, as we will see below. In 1996, http://www.OpenBSD.org/[OpenBSD] split off from NetBSD, and in 2003, http://www.dragonflybsd.org/[DragonFlyBSD] split off from FreeBSD.
386BSD never became a stable operating system.
Instead, two other projects split off from it in 1993: http://www.NetBSD.org/[NetBSD] and link:https://www.FreeBSD.org/[FreeBSD].
The two projects originally diverged due to differences in patience waiting for improvements to 386BSD: the NetBSD people started early in the year, and the first version of FreeBSD was not ready until the end of the year.
In the meantime, the code base had diverged sufficiently to make it difficult to merge.
In addition, the projects had different aims, as we will see below. In 1996, http://www.OpenBSD.org/[OpenBSD] split off from NetBSD, and in 2003, http://www.dragonflybsd.org/[DragonFlyBSD] split off from FreeBSD.
[[why-is-bsd-not-better-known]]
== Why is BSD not better known?
@ -71,27 +96,42 @@ For a number of reasons, BSD is relatively unknown:
. Much of Linux's popularity is due to factors external to the Linux projects, such as the press, and to companies formed to provide Linux services. Until recently, the open source BSDs had no such proponents.
. In 1992, AT&T sued http://www.bsdi.com/[BSDI], the vendor of BSD/386, alleging that the product contained AT&T-copyrighted code. The case was settled out of court in 1994, but the spectre of the litigation continues to haunt people. In March 2000 an article published on the web claimed that the court case had been "recently settled".
+
One detail that the lawsuit did clarify is the naming: in the 1980s, BSD was known as "BSD UNIX(R)". With the elimination of the last vestige of AT&T code from BSD, it also lost the right to the name UNIX(R). Thus you will see references in book titles to "the 4.3BSD UNIX(R) operating system" and "the 4.4BSD operating system".
One detail that the lawsuit did clarify is the naming: in the 1980s, BSD was known as "BSD UNIX(R)".
With the elimination of the last vestige of AT&T code from BSD, it also lost the right to the name UNIX(R).
Thus you will see references in book titles to "the 4.3BSD UNIX(R) operating system" and "the 4.4BSD operating system".
[[comparing-bsd-and-linux]]
== Comparing BSD and Linux
So what is really the difference between, say, Debian Linux and FreeBSD? For the average user, the difference is surprisingly small: Both are UNIX(R) like operating systems. Both are developed by non-commercial projects (this does not apply to many other Linux distributions, of course). In the following section, we will look at BSD and compare it to Linux. The description applies most closely to FreeBSD, which accounts for an estimated 80% of the BSD installations, but the differences from NetBSD, OpenBSD and DragonFlyBSD are small.
So what is really the difference between, say, Debian Linux and FreeBSD? For the average user, the difference is surprisingly small: Both are UNIX(R) like operating systems.
Both are developed by non-commercial projects (this does not apply to many other Linux distributions, of course).
In the following section, we will look at BSD and compare it to Linux.
The description applies most closely to FreeBSD, which accounts for an estimated 80% of the BSD installations, but the differences from NetBSD, OpenBSD and DragonFlyBSD are small.
=== Who owns BSD?
No one person or corporation owns BSD. It is created and distributed by a community of highly technical and committed contributors all over the world. Some of the components of BSD are Open Source projects in their own right and managed by different project maintainers.
No one person or corporation owns BSD.
It is created and distributed by a community of highly technical and committed contributors all over the world.
Some of the components of BSD are Open Source projects in their own right and managed by different project maintainers.
=== How is BSD developed and updated?
The BSD kernels are developed and updated following the Open Source development model. Each project maintains a publicly accessible _source tree_ which contains all source files for the project, including documentation and other incidental files. Users can obtain a complete copy of any version.
The BSD kernels are developed and updated following the Open Source development model.
Each project maintains a publicly accessible _source tree_ which contains all source files for the project, including documentation and other incidental files.
Users can obtain a complete copy of any version.
A large number of developers worldwide contribute to improvements to BSD. They are divided into three kinds:
A large number of developers worldwide contribute to improvements to BSD.
They are divided into three kinds:
* _Contributors_ write code or documentation. They are not permitted to commit (add code) directly to the source tree. In order for their code to be included in the system, it must be reviewed and checked in by a registered developer, known as a __committer__.
* _Committers_ are developers with write access to the source tree. In order to become a committer, an individual must show ability in the area in which they are active.
+
It is at the individual committer's discretion whether they should obtain authority before committing changes to the source tree. In general, an experienced committer may make changes which are obviously correct without obtaining consensus. For example, a documentation project committer may correct typographical or grammatical errors without review. On the other hand, developers making far-reaching or complicated changes are expected to submit their changes for review before committing them. In extreme cases, a core team member with a function such as Principal Architect may order that changes be removed from the tree, a process known as _backing out_. All committers receive mail describing each individual commit, so it is not possible to commit secretly.
It is at the individual committer's discretion whether they should obtain authority before committing changes to the source tree.
In general, an experienced committer may make changes which are obviously correct without obtaining consensus.
For example, a documentation project committer may correct typographical or grammatical errors without review.
On the other hand, developers making far-reaching or complicated changes are expected to submit their changes for review before committing them
In extreme cases, a core team member with a function such as Principal Architect may order that changes be removed from the tree, a process known as _backing out_.
All committers receive mail describing each individual commit, so it is not possible to commit secretly.
* The _Core team_. FreeBSD and NetBSD each have a core team which manages the project. The core teams developed in the course of the projects, and their role is not always well-defined. It is not necessary to be a developer in order to be a core team member, though it is normal. The rules for the core team vary from one project to the other, but in general they have more say in the direction of the project than non-core team members have.
This arrangement differs from Linux in a number of ways:
@ -103,13 +143,20 @@ This arrangement differs from Linux in a number of ways:
=== BSD releases
FreeBSD, NetBSD and OpenBSD provide the system in three different "releases". As with Linux, releases are assigned a number such as 1.4.1 or 3.5. In addition, the version number has a suffix indicating its purpose:
FreeBSD, NetBSD and OpenBSD provide the system in three different "releases".
As with Linux, releases are assigned a number such as 1.4.1 or 3.5.
In addition, the version number has a suffix indicating its purpose:
. The development version of the system is called _CURRENT_. FreeBSD assigns a number to CURRENT, for example FreeBSD 5.0-CURRENT. NetBSD uses a slightly different naming scheme and appends a single-letter suffix which indicates changes in the internal interfaces, for example NetBSD 1.4.3G. OpenBSD does not assign a number ("OpenBSD-current"). All new development on the system goes into this branch.
. At regular intervals, between two and four times a year, the projects bring out a _RELEASE_ version of the system, which is available on CD-ROM and for free download from FTP sites, for example OpenBSD 2.6-RELEASE or NetBSD 1.4-RELEASE. The RELEASE version is intended for end users and is the normal version of the system. NetBSD also provides _patch releases_ with a third digit, for example NetBSD 1.4.2.
. As bugs are found in a RELEASE version, they are fixed, and the fixes are added to the SVN tree. In FreeBSD, the resultant version is called the _STABLE_ version, while in NetBSD and OpenBSD it continues to be called the RELEASE version. Smaller new features can also be added to this branch after a period of test in the CURRENT branch. Security and other important bug fixes are also applied to all supported RELEASE versions.
_By contrast, Linux maintains two separate code trees: the stable version and the development version. Stable versions have an even minor version number, such as 2.0, 2.2 or 2.4. Development versions have an odd minor version number, such as 2.1, 2.3 or 2.5. In each case, the number is followed by a further number designating the exact release. In addition, each vendor adds their own userland programs and utilities, so the name of the distribution is also important. Each distribution vendor also assigns version numbers to the distribution, so a complete description might be something like "TurboLinux 6.0 with kernel 2.2.14"_
_By contrast, Linux maintains two separate code trees: the stable version and the development version.
Stable versions have an even minor version number, such as 2.0, 2.2 or 2.4.
Development versions have an odd minor version number, such as 2.1, 2.3 or 2.5.
In each case, the number is followed by a further number designating the exact release.
In addition, each vendor adds their own userland programs and utilities, so the name of the distribution is also important.
Each distribution vendor also assigns version numbers to the distribution, so a complete description might be something like "TurboLinux 6.0 with kernel 2.2.14"_
=== What versions of BSD are available?
@ -129,19 +176,26 @@ There are also two additional BSD UNIX(R) operating systems which are not open s
=== How does the BSD license differ from the GNU Public license?
Linux is available under the http://www.fsf.org/copyleft/gpl.html[GNU General Public License] (GPL), which is designed to eliminate closed source software. In particular, any derivative work of a product released under the GPL must also be supplied with source code if requested. By contrast, the http://www.opensource.org/licenses/bsd-license.html[BSD license] is less restrictive: binary-only distributions are allowed. This is particularly attractive for embedded applications.
Linux is available under the http://www.fsf.org/copyleft/gpl.html[GNU General Public License] (GPL), which is designed to eliminate closed source software.
In particular, any derivative work of a product released under the GPL must also be supplied with source code if requested.
By contrast, the http://www.opensource.org/licenses/bsd-license.html[BSD license] is less restrictive: binary-only distributions are allowed.
This is particularly attractive for embedded applications.
=== What else should I know?
Since fewer applications are available for BSD than Linux, the BSD developers created a Linux compatibility package, which allows Linux programs to run under BSD. The package includes both kernel modifications, in order to correctly perform Linux system calls, and Linux compatibility files such as the C library. There is no noticeable difference in execution speed between a Linux application running on a Linux machine and a Linux application running on a BSD machine of the same speed.
Since fewer applications are available for BSD than Linux, the BSD developers created a Linux compatibility package, which allows Linux programs to run under BSD.
The package includes both kernel modifications, in order to correctly perform Linux system calls, and Linux compatibility files such as the C library.
There is no noticeable difference in execution speed between a Linux application running on a Linux machine and a Linux application running on a BSD machine of the same speed.
The "all from one supplier" nature of BSD means that upgrades are much easier to handle than is frequently the case with Linux. BSD handles library version upgrades by providing compatibility modules for earlier library versions, so it is possible to run binaries which are several years old with no problems.
The "all from one supplier" nature of BSD means that upgrades are much easier to handle than is frequently the case with Linux.
BSD handles library version upgrades by providing compatibility modules for earlier library versions, so it is possible to run binaries which are several years old with no problems.
=== Which should I use, BSD or Linux?
What does this all mean in practice? Who should use BSD, who should use Linux?
This is a very difficult question to answer. Here are some guidelines:
This is a very difficult question to answer.
Here are some guidelines:
* "If it ain't broke, don't fix it": If you already use an open source operating system, and you are happy with it, there is probably no good reason to change.
* BSD systems, in particular FreeBSD, can have notably higher performance than Linux. But this is not across the board. In many cases, there is little or no difference in performance. In some cases, Linux may perform better than FreeBSD.

106
documentation/content/en/articles/filtering-bridges/_index.adoc
View file

@ -22,9 +22,12 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
Often it is useful to divide one physical network (like an Ethernet) into two separate segments without having to create subnets, and use a router to link them together. The device that connects the two networks in this way is called a bridge. A FreeBSD system with two network interfaces is enough in order to act as a bridge.
Often it is useful to divide one physical network (like an Ethernet) into two separate segments without having to create subnets, and use a router to link them together.
The device that connects the two networks in this way is called a bridge.
A FreeBSD system with two network interfaces is enough in order to act as a bridge.
A bridge works by scanning the addresses of MAC level (Ethernet addresses) of the devices connected to each of its network interfaces and then forwarding the traffic between the two networks only if the source and the destination are on different segments. Under many points of view a bridge is similar to an Ethernet switch with only two ports.
A bridge works by scanning the addresses of MAC level (Ethernet addresses) of the devices connected to each of its network interfaces and then forwarding the traffic between the two networks only if the source and the destination are on different segments.
Under many points of view a bridge is similar to an Ethernet switch with only two ports.
'''
@ -33,26 +36,35 @@ toc::[]
[[filtering-bridges-why]]
== Why use a filtering bridge?
More and more frequently, thanks to the lowering costs of broad band Internet connections (xDSL) and also because of the reduction of available IPv4 addresses, many companies are connected to the Internet 24 hours on 24 and with few (sometimes not even a power of 2) IP addresses. In these situations it is often desirable to have a firewall that filters incoming and outgoing traffic from and towards Internet, but a packet filtering solution based on router may not be applicable, either due to subnetting issues, the router is owned by the connectivity supplier (ISP), or because it does not support such functionalities. In these scenarios the use of a filtering bridge is highly advised.
More and more frequently, thanks to the lowering costs of broad band Internet connections (xDSL) and also because of the reduction of available IPv4 addresses, many companies are connected to the Internet 24 hours on 24 and with few (sometimes not even a power of 2) IP addresses.
In these situations it is often desirable to have a firewall that filters incoming and outgoing traffic from and towards Internet, but a packet filtering solution based on router may not be applicable, either due to subnetting issues, the router is owned by the connectivity supplier (ISP), or because it does not support such functionalities.
In these scenarios the use of a filtering bridge is highly advised.
A bridge-based firewall can be configured and inserted between the xDSL router and your Ethernet hub/switch without any IP numbering issues.
[[filtering-bridges-how]]
== How to Install
Adding bridge functionalities to a FreeBSD system is not difficult. Since 4.5 release it is possible to load such functionalities as modules instead of having to rebuild the kernel, simplifying the procedure a great deal. In the following subsections I will explain both installation ways.
Adding bridge functionalities to a FreeBSD system is not difficult.
Since 4.5 release it is possible to load such functionalities as modules instead of having to rebuild the kernel, simplifying the procedure a great deal.
In the following subsections I will explain both installation ways.
[IMPORTANT]
====
_Do not_ follow both instructions: a procedure _excludes_ the other one. Select the best choice according to your needs and abilities.
_Do not_ follow both instructions: a procedure _excludes_ the other one.
Select the best choice according to your needs and abilities.
====
Before going on, be sure to have at least two Ethernet cards that support the promiscuous mode for both reception and transmission, since they must be able to send Ethernet packets with any address, not just their own. Moreover, to have a good throughput, the cards should be PCI bus mastering cards. The best choices are still the Intel EtherExpress(TM) Pro, followed by the 3Com(R) 3c9xx series. To simplify the firewall configuration it may be useful to have two cards of different manufacturers (using different drivers) in order to distinguish clearly which interface is connected to the router and which to the inner network.
Before going on, be sure to have at least two Ethernet cards that support the promiscuous mode for both reception and transmission, since they must be able to send Ethernet packets with any address, not just their own.
Moreover, to have a good throughput, the cards should be PCI bus mastering cards.
The best choices are still the Intel EtherExpress(TM) Pro, followed by the 3Com(R) 3c9xx series.
To simplify the firewall configuration it may be useful to have two cards of different manufacturers (using different drivers) in order to distinguish clearly which interface is connected to the router and which to the inner network.
[[filtering-bridges-kernel]]
=== Kernel Configuration
So you have decided to use the older but well tested installation method. To begin, you have to add the following rows to your kernel configuration file:
So you have decided to use the older but well tested installation method.
To begin, you have to add the following rows to your kernel configuration file:
[.programlisting]
....
@ -63,7 +75,8 @@ options IPFIREWALL_VERBOSE
The first line is to compile the bridge support, the second one is the firewall and the third one is the logging functions of the firewall.
Now it is necessary to build and install the new kernel. You may find detailed instructions in the link:{handbook}#kernelconfig-building[Building and Installing a Custom Kernel] section of the FreeBSD Handbook.
Now it is necessary to build and install the new kernel.
You may find detailed instructions in the link:{handbook}#kernelconfig-building[Building and Installing a Custom Kernel] section of the FreeBSD Handbook.
[[filtering-bridges-modules]]
=== Modules Loading
@ -75,12 +88,16 @@ If you have chosen to use the new and simpler installation method, the only thin
bridge_load="YES"
....
In this way, during the system startup, the [.filename]#bridge.ko# module will be loaded together with the kernel. It is not required to add a similar row for the [.filename]#ipfw.ko# module, since it will be loaded automatically after the execution of the steps in the following section.
In this way, during the system startup, the [.filename]#bridge.ko# module will be loaded together with the kernel.
It is not required to add a similar row for the [.filename]#ipfw.ko# module, since it will be loaded automatically after the execution of the steps in the following section.
[[filtering-bridges-finalprep]]
== Final Preparation
Before rebooting in order to load the new kernel or the required modules (according to the previously chosen installation method), you have to make some changes to the [.filename]#/etc/rc.conf# configuration file. The default rule of the firewall is to reject all IP packets. Initially we will set up an `open` firewall, in order to verify its operation without any issue related to packet filtering (in case you are going to execute this procedure remotely, such configuration will avoid you to remain isolated from the network). Put these lines in [.filename]#/etc/rc.conf#:
Before rebooting in order to load the new kernel or the required modules (according to the previously chosen installation method), you have to make some changes to the [.filename]#/etc/rc.conf# configuration file.
The default rule of the firewall is to reject all IP packets.
Initially we will set up an `open` firewall, in order to verify its operation without any issue related to packet filtering (in case you are going to execute this procedure remotely, such configuration will avoid you to remain isolated from the network).
Put these lines in [.filename]#/etc/rc.conf#:
[.programlisting]
....
@ -92,9 +109,18 @@ firewall_logging="YES"
The first row will enable the firewall (and will load the module [.filename]#ipfw.ko# if it is not compiled in the kernel), the second one to set up it in `open` mode (as explained in [.filename]#/etc/rc.firewall#), the third one to not show rules loading and the fourth one to enable logging support.
About the configuration of the network interfaces, the most used way is to assign an IP to only one of the network cards, but the bridge will work equally even if both interfaces or none has a configured IP. In the last case (IP-less) the bridge machine will be still more hidden, as inaccessible from the network: to configure it, you have to login from console or through a third network interface separated from the bridge. Sometimes, during the system startup, some programs require network access, say for domain resolution: in this case it is necessary to assign an IP to the external interface (the one connected to Internet, where DNS server resides), since the bridge will be activated at the end of the startup procedure. It means that the [.filename]#fxp0# interface (in our case) must be mentioned in the ifconfig section of the [.filename]#/etc/rc.conf# file, while the [.filename]#xl0# is not. Assigning an IP to both the network cards does not make much sense, unless, during the start procedure, applications should access to services on both Ethernet segments.
About the configuration of the network interfaces, the most used way is to assign an IP to only one of the network cards, but the bridge will work equally even if both interfaces or none has a configured IP.
In the last case (IP-less) the bridge machine will be still more hidden, as inaccessible from the network: to configure it, you have to login from console or through a third network interface separated from the bridge.
Sometimes, during the system startup, some programs require network access, say for domain resolution: in this case it is necessary to assign an IP to the external interface (the one connected to Internet, where DNS server resides), since the bridge will be activated at the end of the startup procedure.
It means that the [.filename]#fxp0# interface (in our case) must be mentioned in the ifconfig section of the [.filename]#/etc/rc.conf# file, while the [.filename]#xl0# is not.
Assigning an IP to both the network cards does not make much sense, unless, during the start procedure, applications should access to services on both Ethernet segments.
There is another important thing to know. When running IP over Ethernet, there are actually two Ethernet protocols in use: one is IP, the other is ARP. ARP does the conversion of the IP address of a host into its Ethernet address (MAC layer). In order to allow the communication between two hosts separated by the bridge, it is necessary that the bridge will forward ARP packets. Such protocol is not included in the IP layer, since it exists only with IP over Ethernet. The FreeBSD firewall filters exclusively on the IP layer and therefore all non-IP packets (ARP included) will be forwarded without being filtered, even if the firewall is configured to not permit anything.
There is another important thing to know.
When running IP over Ethernet, there are actually two Ethernet protocols in use: one is IP, the other is ARP.
ARP does the conversion of the IP address of a host into its Ethernet address (MAC layer).
In order to allow the communication between two hosts separated by the bridge, it is necessary that the bridge will forward ARP packets.
Such protocol is not included in the IP layer, since it exists only with IP over Ethernet.
The FreeBSD firewall filters exclusively on the IP layer and therefore all non-IP packets (ARP included) will be forwarded without being filtered, even if the firewall is configured to not permit anything.
Now it is time to reboot the system and use it as before: there will be some new messages about the bridge and the firewall, but the bridge will not be activated and the firewall, being in `open` mode, will not avoid any operations.
@ -114,16 +140,32 @@ At this point, to enable the bridge, you have to execute the following commands
The first row specifies which interfaces should be activated by the bridge, the second one will enable the firewall on the bridge and finally the third one will enable the bridge.
At this point you should be able to insert the machine between two sets of hosts without compromising any communication abilities between them. If so, the next step is to add the `net.link.ether.bridge._[blah]_=_[blah]_` portions of these rows to the [.filename]#/etc/sysctl.conf# file, in order to have them execute at startup.
At this point you should be able to insert the machine between two sets of hosts without compromising any communication abilities between them.
If so, the next step is to add the `net.link.ether.bridge._[blah]_=_[blah]_` portions of these rows to the [.filename]#/etc/sysctl.conf# file, in order to have them execute at startup.
[[filtering-bridges-ipfirewall]]
== Configuring The Firewall
Now it is time to create your own file with custom firewall rules, in order to secure the inside network. There will be some complication in doing this because not all of the firewall functionalities are available on bridged packets. Furthermore, there is a difference between the packets that are in the process of being forwarded and packets that are being received by the local machine. In general, incoming packets are run through the firewall only once, not twice as is normally the case; in fact they are filtered only upon receipt, so rules that use `out` or `xmit` will never match. Personally, I use `in via` which is an older syntax, but one that has a sense when you read it. Another limitation is that you are restricted to use only `pass` or `drop` commands for packets filtered by a bridge. Sophisticated things like `divert`, `forward` or `reject` are not available. Such options can still be used, but only on traffic to or from the bridge machine itself (if it has an IP address).
Now it is time to create your own file with custom firewall rules, in order to secure the inside network.
There will be some complication in doing this because not all of the firewall functionalities are available on bridged packets.
Furthermore, there is a difference between the packets that are in the process of being forwarded and packets that are being received by the local machine.
In general, incoming packets are run through the firewall only once, not twice as is normally the case; in fact they are filtered only upon receipt, so rules that use `out` or `xmit` will never match.
Personally, I use `in via` which is an older syntax, but one that has a sense when you read it.
Another limitation is that you are restricted to use only `pass` or `drop` commands for packets filtered by a bridge.
Sophisticated things like `divert`, `forward` or `reject` are not available.
Such options can still be used, but only on traffic to or from the bridge machine itself (if it has an IP address).
New in FreeBSD 4.0, is the concept of stateful filtering. This is a big improvement for UDP traffic, which typically is a request going out, followed shortly thereafter by a response with the exact same set of IP addresses and port numbers (but with source and destination reversed, of course). For firewalls that have no statekeeping, there is almost no way to deal with this sort of traffic as a single session. But with a firewall that can "remember" an outgoing UDP packet and, for the next few minutes, allow a response, handling UDP services is trivial. The following example shows how to do it. It is possible to do the same thing with TCP packets. This allows you to avoid some denial of service attacks and other nasty tricks, but it also typically makes your state table grow quickly in size.
New in FreeBSD 4.0, is the concept of stateful filtering.
This is a big improvement for UDP traffic, which typically is a request going out, followed shortly thereafter by a response with the exact same set of IP addresses and port numbers (but with source and destination reversed, of course).
For firewalls that have no statekeeping, there is almost no way to deal with this sort of traffic as a single session.
But with a firewall that can "remember" an outgoing UDP packet and, for the next few minutes, allow a response, handling UDP services is trivial.
The following example shows how to do it.
It is possible to do the same thing with TCP packets.
This allows you to avoid some denial of service attacks and other nasty tricks, but it also typically makes your state table grow quickly in size.
Let's look at an example setup. Note first that at the top of [.filename]#/etc/rc.firewall# there are already standard rules for the loopback interface [.filename]#lo0#, so we should not have to care for them anymore. Custom rules should be put in a separate file (say [.filename]#/etc/rc.firewall.local#) and loaded at system startup, by modifying the row of [.filename]#/etc/rc.conf# where we defined the `open` firewall:
Let's look at an example setup.
Note first that at the top of [.filename]#/etc/rc.firewall# there are already standard rules for the loopback interface [.filename]#lo0#, so we should not have to care for them anymore.
Custom rules should be put in a separate file (say [.filename]#/etc/rc.firewall.local#) and loaded at system startup, by modifying the row of [.filename]#/etc/rc.conf# where we defined the `open` firewall:
[.programlisting]
....
@ -187,27 +229,45 @@ add pass icmp from any to any icmptypes 11
add drop log all from any to any
....
Those of you who have set up firewalls before may notice some things missing. In particular, there are no anti-spoofing rules, in fact we did _not_ add:
Those of you who have set up firewalls before may notice some things missing.
In particular, there are no anti-spoofing rules, in fact we did _not_ add:
[.programlisting]
....
add deny all from 1.2.3.4/8 to any in via fxp0
....
That is, drop packets that are coming in from the outside claiming to be from our network. This is something that you would commonly do to be sure that someone does not try to evade the packet filter, by generating nefarious packets that look like they are from the inside. The problem with that is that there is _at least_ one host on the outside interface that you do not want to ignore: the router. But usually, the ISP anti-spoofs at their router, so we do not need to bother that much.
That is, drop packets that are coming in from the outside claiming to be from our network.
This is something that you would commonly do to be sure that someone does not try to evade the packet filter, by generating nefarious packets that look like they are from the inside.
The problem with that is that there is _at least_ one host on the outside interface that you do not want to ignore: the router.
But usually, the ISP anti-spoofs at their router, so we do not need to bother that much.
The last rule seems to be an exact duplicate of the default rule, that is, do not let anything pass that is not specifically allowed. But there is a difference: all suspected traffic will be logged.
The last rule seems to be an exact duplicate of the default rule, that is, do not let anything pass that is not specifically allowed.
But there is a difference: all suspected traffic will be logged.
There are two rules for passing SMTP and DNS traffic towards the mail server and the name server, if you have them. Obviously the whole rule set should be flavored to personal taste, this is only a specific example (rule format is described accurately in the man:ipfw[8] man page). Note that in order for "relay" and "ns" to work, name service lookups must work _before_ the bridge is enabled. This is an example of making sure that you set the IP on the correct network card. Alternatively it is possible to specify the IP address instead of the host name (required if the machine is IP-less).
There are two rules for passing SMTP and DNS traffic towards the mail server and the name server, if you have them.
Obviously the whole rule set should be flavored to personal taste, this is only a specific example (rule format is described accurately in the man:ipfw[8] man page).
Note that in order for "relay" and "ns" to work, name service lookups must work _before_ the bridge is enabled.
This is an example of making sure that you set the IP on the correct network card.
Alternatively it is possible to specify the IP address instead of the host name (required if the machine is IP-less).
People that are used to setting up firewalls are probably also used to either having a `reset` or a `forward` rule for ident packets (TCP port 113). Unfortunately, this is not an applicable option with the bridge, so the best thing is to simply pass them to their destination. As long as that destination machine is not running an ident daemon, this is relatively harmless. The alternative is dropping connections on port 113, which creates some problems with services like IRC (the ident probe must timeout).
People that are used to setting up firewalls are probably also used to either having a `reset` or a `forward` rule for ident packets (TCP port 113).
Unfortunately, this is not an applicable option with the bridge, so the best thing is to simply pass them to their destination.
As long as that destination machine is not running an ident daemon, this is relatively harmless.
The alternative is dropping connections on port 113, which creates some problems with services like IRC (the ident probe must timeout).
The only other thing that is a little weird that you may have noticed is that there is a rule to let the bridge machine speak, and another for internal hosts. Remember that this is because the two sets of traffic will take different paths through the kernel and into the packet filter. The inside net will go through the bridge, while the local machine will use the normal IP stack to speak. Thus the two rules to handle the different cases. The `in via fxp0` rules work for both paths. In general, if you use `in via` rules throughout the filter, you will need to make an exception for locally generated packets, because they did not come in via any of our interfaces.
The only other thing that is a little weird that you may have noticed is that there is a rule to let the bridge machine speak, and another for internal hosts.
Remember that this is because the two sets of traffic will take different paths through the kernel and into the packet filter.
The inside net will go through the bridge, while the local machine will use the normal IP stack to speak.
Thus the two rules to handle the different cases.
The `in via fxp0` rules work for both paths.
In general, if you use `in via` rules throughout the filter, you will need to make an exception for locally generated packets, because they did not come in via any of our interfaces.
[[filtering-bridges-contributors]]
== Contributors
Many parts of this article have been taken, updated and adapted from an old text about bridging, edited by Nick Sayer. A pair of inspirations are due to an introduction on bridging by Steve Peterson.
Many parts of this article have been taken, updated and adapted from an old text about bridging, edited by Nick Sayer.
A pair of inspirations are due to an introduction on bridging by Steve Peterson.
A big thanks to Luigi Rizzo for the implementation of the bridge code in FreeBSD and for the time he has dedicated to me answering all of my related questions.

143
documentation/content/en/articles/fonts/_index.adoc
View file

@ -21,7 +21,8 @@ trademarks: ["freebsd", "adobe", "apple", "linux", "microsoft", "opengroup", "ge
[.abstract-title]
Abstract
This document contains a description of the various font files that may be used with FreeBSD and the syscons driver, X11, Ghostscript and Groff. Cookbook examples are provided for switching the syscons display to 80x60 mode, and for using type 1 fonts with the above application programs.
This document contains a description of the various font files that may be used with FreeBSD and the syscons driver, X11, Ghostscript and Groff.
Cookbook examples are provided for switching the syscons display to 80x60 mode, and for using type 1 fonts with the above application programs.
'''
@ -30,12 +31,15 @@ toc::[]
[[intro]]
== Introduction
There are many sources of fonts available, and one might ask how they might be used with FreeBSD. The answer can be found by carefully searching the documentation for the component that one would like to use. This is very time consuming, so this tutorial is an attempt to provide a shortcut for others who might be interested.
There are many sources of fonts available, and one might ask how they might be used with FreeBSD.
The answer can be found by carefully searching the documentation for the component that one would like to use.
This is very time consuming, so this tutorial is an attempt to provide a shortcut for others who might be interested.
[[terminology]]
== Basic Terminology
There are many different font formats and associated font file suffixes. A few that will be addressed here are:
There are many different font formats and associated font file suffixes.
A few that will be addressed here are:
[.filename]#.pfa#, [.filename]#.pfb#::
PostScript(R) type 1 fonts. The [.filename]#.pfa# is the __A__scii form and [.filename]#.pfb# the __B__inary form.
@ -55,12 +59,16 @@ An indirect reference to a TrueType font (not an actual font)
[.filename]#.fon#, [.filename]#.fnt#::
Bitmapped screen fonts
The [.filename]#.fot# is used by Windows(R) as sort of a symbolic link to the actual TrueType(R) font ([.filename]#.ttf#) file. The [.filename]#.fon# font files are also used by Windows. I know of no way to use this font format with FreeBSD.
The [.filename]#.fot# is used by Windows(R) as sort of a symbolic link to the actual TrueType(R) font ([.filename]#.ttf#) file. The [.filename]#.fon# font files are also used by Windows.
I know of no way to use this font format with FreeBSD.
[[font-formats]]
== What Font Formats Can I Use?
Which font file format is useful depends on the application being used. FreeBSD by itself uses no fonts. Application programs and/or drivers may make use of the font files. Here is a small cross reference of application/driver to the font type suffixes:
Which font file format is useful depends on the application being used.
FreeBSD by itself uses no fonts.
Application programs and/or drivers may make use of the font files.
Here is a small cross reference of application/driver to the font type suffixes:
Driver::
@ -84,12 +92,16 @@ Groff:::
Povray:::
[.filename]#.ttf#
The [.filename]#.fnt# suffix is used quite frequently. I suspect that whenever someone wanted to create a specialized font file for their application, more often than not they chose this suffix. Therefore, it is likely that files with this suffix are not all the same format; specifically, the [.filename]#.fnt# files used by syscons under FreeBSD may not be the same format as a [.filename]#.fnt# one encounters in the MS-DOS(R)/Windows(R) environment. I have not made any attempt at using other [.filename]#.fnt# files other than those provided with FreeBSD.
The [.filename]#.fnt# suffix is used quite frequently.
I suspect that whenever someone wanted to create a specialized font file for their application, more often than not they chose this suffix.
Therefore, it is likely that files with this suffix are not all the same format; specifically, the [.filename]#.fnt# files used by syscons under FreeBSD may not be the same format as a [.filename]#.fnt# one encounters in the MS-DOS(R)/Windows(R) environment.
I have not made any attempt at using other [.filename]#.fnt# files other than those provided with FreeBSD.
[[virtual-console]]
== Setting a Virtual Console to 80x60 Line Mode
First, an 8x8 font must be loaded. To do this, [.filename]#/etc/rc.conf# should contain the line (change the font name to an appropriate one for your locale):
First, an 8x8 font must be loaded.
To do this, [.filename]#/etc/rc.conf# should contain the line (change the font name to an appropriate one for your locale):
[.programlisting]
....
@ -103,9 +115,11 @@ The command to actually switch the mode is man:vidcontrol[1]:
% vidcontrol VGA_80x60
....
Various screen-oriented programs, such as man:vi[1], must be able to determine the current screen dimensions. As this is achieved this through `ioctl` calls to the console driver (such as man:syscons[4]) they will correctly determine the new screen dimensions.
Various screen-oriented programs, such as man:vi[1], must be able to determine the current screen dimensions.
As this is achieved this through `ioctl` calls to the console driver (such as man:syscons[4]) they will correctly determine the new screen dimensions.
To make this more seamless, one can embed these commands in the startup scripts so it takes place when the system boots. To do this is add this line to [.filename]#/etc/rc.conf#.
To make this more seamless, one can embed these commands in the startup scripts so it takes place when the system boots.
To do this is add this line to [.filename]#/etc/rc.conf#.
[.programlisting]
....
@ -117,9 +131,15 @@ References: man:rc.conf[5], man:vidcontrol[1].
[[type1-fonts-x11]]
== Using Type 1 Fonts with X11
X11 can use either the [.filename]#.pfa# or the [.filename]#.pfb# format fonts. The X11 fonts are located in various subdirectories under [.filename]#/usr/X11R6/lib/X11/fonts#. Each font file is cross referenced to its X11 name by the contents of [.filename]#fonts.dir# in each directory.
X11 can use either the [.filename]#.pfa# or the [.filename]#.pfb# format fonts.
The X11 fonts are located in various subdirectories under [.filename]#/usr/X11R6/lib/X11/fonts#.
Each font file is cross referenced to its X11 name by the contents of [.filename]#fonts.dir# in each directory.
There is already a directory named [.filename]#Type1#. The most straight forward way to add a new font is to put it into this directory. A better way is to keep all new fonts in a separate directory and use a symbolic link to the additional font. This allows one to more easily keep track of ones fonts without confusing them with the fonts that were originally provided. For example:
There is already a directory named [.filename]#Type1#.
The most straight forward way to add a new font is to put it into this directory.
A better way is to keep all new fonts in a separate directory and use a symbolic link to the additional font.
This allows one to more easily keep track of ones fonts without confusing them with the fonts that were originally provided.
For example:
[source,shell]
....
@ -139,7 +159,8 @@ Maintain an index to cross reference the fonts
% echo showboat - InfoMagic CICA, Dec 1994, /fonts/atm/showboat >>INDEX
....
Now, to use a new font with X11, one must make the font file available and update the font name files. The X11 font names look like:
Now, to use a new font with X11, one must make the font file available and update the font name files.
The X11 font names look like:
[.programlisting]
....
@ -156,7 +177,10 @@ Now, to use a new font with X11, one must make the font file available and updat
foundry family weight slant width additional style
....
A new name needs to be created for each new font. If you have some information from the documentation that accompanied the font, then it could serve as the basis for creating the name. If there is no information, then you can get some idea by using man:strings[1] on the font file. For example:
A new name needs to be created for each new font.
If you have some information from the documentation that accompanied the font, then it could serve as the basis for creating the name.
If there is no information, then you can get some idea by using man:strings[1] on the font file.
For example:
[source,shell]
....
@ -203,26 +227,33 @@ Family::
The name of the font.
Weight::
Normal, bold, medium, semibold, etc. From the man:strings[1] output above, it appears that this font has a weight of __medium__.
Normal, bold, medium, semibold, etc.
From the man:strings[1] output above, it appears that this font has a weight of __medium__.
Slant::
__r__oman, __i__talic, __o__blique, etc. Since the _ItalicAngle_ is zero, _roman_ will be used.
__r__oman, __i__talic, __o__blique, etc.
Since the _ItalicAngle_ is zero, _roman_ will be used.
Width::
Normal, wide, condensed, extended, etc. Until it can be examined, the assumption will be __normal__.
Normal, wide, condensed, extended, etc.
Until it can be examined, the assumption will be __normal__.
Additional style::
Usually omitted, but this will indicate that the font contains decorative capital letters.
Spacing::
proportional or monospaced. _Proportional_ is used since _isFixedPitch_ is false.
proportional or monospaced.
_Proportional_ is used since _isFixedPitch_ is false.
All of these names are arbitrary, but one should strive to be compatible with the existing conventions. A font is referenced by name with possible wild cards by an X11 program, so the name chosen should make some sense. One might begin by simply using
All of these names are arbitrary, but one should strive to be compatible with the existing conventions.
A font is referenced by name with possible wild cards by an X11 program, so the name chosen should make some sense.
One might begin by simply using
[source,shell]
....
...-normal-r-normal-...-p-...
....
as the name, and then use man:xfontsel[1] to examine it and adjust the name based on the appearance of the font.
So, to complete our example:
@ -261,7 +292,10 @@ References: man:xfontsel[1], man:xset[1], The X Windows System in a Nutshell, ht
[[type1-fonts-ghostscript]]
== Using Type 1 Fonts with Ghostscript
Ghostscript references a font via its [.filename]#Fontmap#. This must be modified in a similar way to the X11 [.filename]#fonts.dir#. Ghostscript can use either the [.filename]#.pfa# or the [.filename]#.pfb# format fonts. Using the font from the previous example, here is how to use it with Ghostscript:
Ghostscript references a font via its [.filename]#Fontmap#.
This must be modified in a similar way to the X11 [.filename]#fonts.dir#.
Ghostscript can use either the [.filename]#.pfa# or the [.filename]#.pfb# format fonts.
Using the font from the previous example, here is how to use it with Ghostscript:
[source,shell]
....
@ -299,9 +333,15 @@ References: [.filename]#fonts.txt# in the Ghostscript 4.01 distribution
[[type1-fonts-groff]]
== Using Type 1 Fonts with Groff
Now that the new font can be used by both X11 and Ghostscript, how can one use the new font with groff? First of all, since we are dealing with type 1 PostScript(R) fonts, the groff device that is applicable is the _ps_ device. A font file must be created for each font that groff can use. A groff font name is just a file in [.filename]#/usr/share/groff_font/devps#. With our example, the font file could be [.filename]#/usr/share/groff_font/devps/SHOWBOAT#. The file must be created using tools provided by groff.
Now that the new font can be used by both X11 and Ghostscript, how can one use the new font with groff? First of all, since we are dealing with type 1 PostScript(R) fonts, the groff device that is applicable is the _ps_ device.
A font file must be created for each font that groff can use.
A groff font name is just a file in [.filename]#/usr/share/groff_font/devps#.
With our example, the font file could be [.filename]#/usr/share/groff_font/devps/SHOWBOAT#.
The file must be created using tools provided by groff.
The first tool is `afmtodit`. This is not normally installed, so it must be retrieved from the source distribution. I found I had to change the first line of the file, so I did:
The first tool is `afmtodit`.
This is not normally installed, so it must be retrieved from the source distribution.
I found I had to change the first line of the file, so I did:
[source,shell]
....
@ -313,7 +353,8 @@ The first tool is `afmtodit`. This is not normally installed, so it must be retr
:wq
....
This tool will create the groff font file from the metrics file ([.filename]#.afm# suffix.) Continuing with our example:
This tool will create the groff font file from the metrics file ([.filename]#.afm# suffix.)
Continuing with our example:
[source,shell]
....
@ -330,7 +371,12 @@ Now create the groff font file
The font can now be referenced with the name SHOWBOAT.
If Ghostscript is used to drive the printers on the system, then nothing more needs to be done. However, if true PostScript(R) printers are used, then the font must be downloaded to the printer in order for the font to be used (unless the printer happens to have the showboat font built in or on an accessible font disk.) The final step is to create a downloadable font. The `pfbtops` tool is used to create the [.filename]#.pfa# format of the font, and [.filename]#download# is modified to reference the new font. The [.filename]#download# must reference the internal name of the font. This can easily be determined from the groff font file as illustrated:
If Ghostscript is used to drive the printers on the system, then nothing more needs to be done.
However, if true PostScript(R) printers are used, then the font must be downloaded to the printer in order for the font to be used (unless the printer happens to have the showboat font built in or on an accessible font disk.)
The final step is to create a downloadable font.
The `pfbtops` tool is used to create the [.filename]#.pfa# format of the font, and [.filename]#download# is modified to reference the new font.
The [.filename]#download# must reference the internal name of the font.
This can easily be determined from the groff font file as illustrated:
[source,shell]
....
@ -398,33 +444,41 @@ References: [.filename]#/usr/src/gnu/usr.bin/groff/afmtodit/afmtodit.man#, man:g
[[convert-truetype]]
== Converting TrueType Fonts to a groff/PostScript Format For groff
This potentially requires a bit of work, simply because it depends on some utilities that are not installed as part of the base system. They are:
This potentially requires a bit of work, simply because it depends on some utilities that are not installed as part of the base system.
They are:
`ttf2pf`::
TrueType to PostScript conversion utilities. This allows conversion of a TrueType font to an ascii font metric ([.filename]#.afm#) file.
TrueType to PostScript conversion utilities.
This allows conversion of a TrueType font to an ascii font metric ([.filename]#.afm#) file.
+
Currently available at http://sunsite.icm.edu.pl/pub/GUST/contrib/BachoTeX98/ttf2pf/[http://sunsite.icm.edu.pl/pub/GUST/contrib/BachoTeX98/ttf2pf/]. Note: These files are PostScript programs and must be downloaded to disk by holding down kbd:[Shift] when clicking on the link. Otherwise, your browser may try to launch ghostview to view them.
Currently available at http://sunsite.icm.edu.pl/pub/GUST/contrib/BachoTeX98/ttf2pf/[http://sunsite.icm.edu.pl/pub/GUST/contrib/BachoTeX98/ttf2pf/].
Note: These files are PostScript programs and must be downloaded to disk by holding down kbd:[Shift] when clicking on the link.
Otherwise, your browser may try to launch ghostview to view them.
+
The files of interest are:
** [.filename]#GS_TTF.PS#
** [.filename]#PF2AFM.PS#
** [.filename]#ttf2pf.ps#
+
The funny upper/lower case is due to their being intended also for DOS shells. [.filename]#ttf2pf.ps# makes use of the others as upper case, so any renaming must be consistent with this. (Actually, [.filename]#GS_TTF.PS# and [.filename]#PFS2AFM.PS# are supposedly part of the Ghostscript distribution, but it is just as easy to use these as an isolated utility. FreeBSD does not seem to include the latter.) You also may want to have these installed to [.filename]#/usr/local/shared/groff_font/devps#(?).
The funny upper/lower case is due to their being intended also for DOS shells.
[.filename]#ttf2pf.ps# makes use of the others as upper case, so any renaming must be consistent with this.
(Actually, [.filename]#GS_TTF.PS# and [.filename]#PFS2AFM.PS# are supposedly part of the Ghostscript distribution, but it is just as easy to use these as an isolated utility.
FreeBSD does not seem to include the latter.)
You also may want to have these installed to [.filename]#/usr/local/shared/groff_font/devps#(?).
`afmtodit`::
Creates font files for use with groff from ascii font metrics file. This usually resides in the directory, [.filename]#/usr/src/contrib/groff/afmtodit#, and requires some work to get going.
Creates font files for use with groff from ascii font metrics file.
This usually resides in the directory, [.filename]#/usr/src/contrib/groff/afmtodit#, and requires some work to get going.
+
[NOTE]
====
If you are paranoid about working in the [.filename]#/usr/src# tree, simply copy the contents of the above directory to a work location.
====
+
In the work area, you will need to make the utility. Just type:
In the work area, you will need to make the utility.
Just type:
+
[source,shell]
....
# make -f Makefile.sub afmtodit
@ -443,7 +497,8 @@ Once all these utilities are in place, you are ready to commence:
+
Where, _TTF_name_ is your TrueType font file, _PS_font_name_ is the file name for [.filename]#.pfa#, _AFM_name_ is the name you wish for [.filename]#.afm#. If you do not specify output file names for the [.filename]#.pfa# or [.filename]#.afm# files, then default names will be generated from the TrueType font file name.
+
This also produces a [.filename]#.pfa#, the ascii PostScript font metrics file ([.filename]#.pfb# is for the binary form). This will not be needed, but could (I think) be useful for a fontserver.
This also produces a [.filename]#.pfa#, the ascii PostScript font metrics file ([.filename]#.pfb# is for the binary form).
This will not be needed, but could (I think) be useful for a fontserver.
+
For example, to convert the 30f9 Barcode font using the default file names, use the following command:
+
@ -469,14 +524,17 @@ Converting 3of9.ttf to A.pfa and B.afm.
. Create the groff PostScript file:
+
Change directories to [.filename]#/usr/share/groff_font/devps# so as to make the following command easier to execute. You will probably need root privileges for this. (Or, if you are paranoid about working there, make sure you reference the files [.filename]#DESC#, [.filename]#text.enc# and [.filename]#generate/textmap# as being in this directory.)
Change directories to [.filename]#/usr/share/groff_font/devps# so as to make the following command easier to execute.
You will probably need root privileges for this.
(Or, if you are paranoid about working there, make sure you reference the files [.filename]#DESC#, [.filename]#text.enc# and [.filename]#generate/textmap# as being in this directory.)
+
[source,shell]
....
% afmtodit -d DESC -e text.enc file.afm generate/textmap PS_font_name
....
+
Where, [.filename]#file.afm# is the _AFM_name_ created by `ttf2pf.ps` above, and _PS_font_name_ is the font name used from that command, as well as the name that man:groff[1] will use for references to this font. For example, assuming you used the first `tiff2pf.ps` above, then the 3of9 Barcode font can be created using the command:
Where, [.filename]#file.afm# is the _AFM_name_ created by `ttf2pf.ps` above, and _PS_font_name_ is the font name used from that command, as well as the name that man:groff[1] will use for references to this font.
For example, assuming you used the first `tiff2pf.ps` above, then the 3of9 Barcode font can be created using the command:
+
[source,shell]
....
@ -485,16 +543,21 @@ Where, [.filename]#file.afm# is the _AFM_name_ created by `ttf2pf.ps` above, and
+
Ensure that the resulting _PS_font_name_ file (e.g., [.filename]#3of9# in the example above) is located in the directory [.filename]#/usr/share/groff_font/devps# by copying or moving it there.
+
Note that if [.filename]#ttf2pf.ps# assigns a font name using the one it finds in the TrueType font file and you want to use a different name, you must edit the [.filename]#.afm# prior to running `afmtodit`. This name must also match the one used in the Fontmap file if you wish to pipe man:groff[1] into man:gs[1].
Note that if [.filename]#ttf2pf.ps# assigns a font name using the one it finds in the TrueType font file and you want to use a different name, you must edit the [.filename]#.afm# prior to running `afmtodit`.
This name must also match the one used in the Fontmap file if you wish to pipe man:groff[1] into man:gs[1].
[[truetype-for-other-programs]]
== Can TrueType Fonts be Used with Other Programs?
The TrueType font format is used by Windows, Windows 95, and Mac's. It is quite popular and there are a great number of fonts available in this format.
The TrueType font format is used by Windows, Windows 95, and Mac's.
It is quite popular and there are a great number of fonts available in this format.
Unfortunately, there are few applications that I am aware of that can use this format: Ghostscript and Povray come to mind. Ghostscript's support, according to the documentation, is rudimentary and the results are likely to be inferior to type 1 fonts. Povray version 3 also has the ability to use TrueType fonts, but I rather doubt many people will be creating documents as a series of raytraced pages :-).
Unfortunately, there are few applications that I am aware of that can use this format: Ghostscript and Povray come to mind.
Ghostscript's support, according to the documentation, is rudimentary and the results are likely to be inferior to type 1 fonts.
Povray version 3 also has the ability to use TrueType fonts, but I rather doubt many people will be creating documents as a series of raytraced pages :-).
This rather dismal situation may soon change. The http://www.freetype.org/[FreeType Project] is currently developing a useful set of FreeType tools:
This rather dismal situation may soon change.
The http://www.freetype.org/[FreeType Project] is currently developing a useful set of FreeType tools:
* The `xfsft` font server for X11 can serve TrueType fonts in addition to regular fonts. Though currently in beta, it is said to be quite usable. See http://www.dcs.ed.ac.uk/home/jec/programs/xfsft/[Juliusz Chroboczek's page] for further information. Porting instructions for FreeBSD can be found at http://math.missouri.edu/~stephen/software/[Stephen Montgomery's software page].
* xfstt is another font server for X11, available under link:ftp://sunsite.unc.edu/pub/Linux/X11/fonts/[ftp://sunsite.unc.edu/pub/Linux/X11/fonts/].
@ -504,7 +567,9 @@ This rather dismal situation may soon change. The http://www.freetype.org/[FreeT
[[obtaining-additional-fonts]]
== Where Can Additional Fonts be Obtained?
Many fonts are available on the Internet. They are either entirely free, or are share-ware. In addition many fonts are available in the [.filename]#x11-fonts/# category in the ports collection
Many fonts are available on the Internet.
They are either entirely free, or are share-ware.
In addition many fonts are available in the [.filename]#x11-fonts/# category in the ports collection
[[additional-questions]]
== Additional Questions

66
documentation/content/en/articles/freebsd-questions/_index.adoc
View file

@ -23,7 +23,8 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
This document provides useful information for people looking to prepare an e-mail to the FreeBSD-questions mailing list. Advice and hints are given that will maximize the chance that the reader will receive useful replies.
This document provides useful information for people looking to prepare an e-mail to the FreeBSD-questions mailing list.
Advice and hints are given that will maximize the chance that the reader will receive useful replies.
This document is regularly posted to the FreeBSD-questions mailing list.
@ -33,35 +34,49 @@ toc::[]
== Introduction
`FreeBSD-questions` is a mailing list maintained by the FreeBSD project to help people who have questions about the normal use of FreeBSD. Another group, `FreeBSD-hackers`, discusses more advanced questions such as future development work.
`FreeBSD-questions` is a mailing list maintained by the FreeBSD project to help people who have questions about the normal use of FreeBSD.
Another group, `FreeBSD-hackers`, discusses more advanced questions such as future development work.
[NOTE]
====
The term "hacker" has nothing to do with breaking into other people's computers. The correct term for the latter activity is "cracker", but the popular press has not found out yet. The FreeBSD hackers disapprove strongly of cracking security, and have nothing to do with it. For a longer description of hackers, see Eric Raymond's http://www.catb.org/~esr/faqs/hacker-howto.html[How To Become A Hacker]
The term "hacker" has nothing to do with breaking into other people's computers.
The correct term for the latter activity is "cracker", but the popular press has not found out yet.
The FreeBSD hackers disapprove strongly of cracking security, and have nothing to do with it.
For a longer description of hackers, see Eric Raymond's http://www.catb.org/~esr/faqs/hacker-howto.html[How To Become A Hacker]
====
This is a regular posting aimed to help both those seeking advice from FreeBSD-questions (the "newcomers"), and also those who answer the questions (the "hackers").
Inevitably there is some friction, which stems from the different viewpoints of the two groups. The newcomers accuse the hackers of being arrogant, stuck-up, and unhelpful, while the hackers accuse the newcomers of being stupid, unable to read plain English, and expecting everything to be handed to them on a silver platter. Of course, there is an element of truth in both these claims, but for the most part these viewpoints come from a sense of frustration.
Inevitably there is some friction, which stems from the different viewpoints of the two groups.
The newcomers accuse the hackers of being arrogant, stuck-up, and unhelpful, while the hackers accuse the newcomers of being stupid, unable to read plain English, and expecting everything to be handed to them on a silver platter.
Of course, there is an element of truth in both these claims, but for the most part these viewpoints come from a sense of frustration.
In this document, I would like to do something to relieve this frustration and help everybody get better results from FreeBSD-questions. In the following section, I recommend how to submit a question; after that, we will look at how to answer one.
In this document, I would like to do something to relieve this frustration and help everybody get better results from FreeBSD-questions.
In the following section, I recommend how to submit a question; after that, we will look at how to answer one.
== How to Subscribe to FreeBSD-questions
FreeBSD-questions is a mailing list, so you need mail access. Point your WWW browser to the {freebsd-questions}. In the section titled "Subscribing to freebsd-questions" fill in the "Your email address" field; the other fields are optional.
FreeBSD-questions is a mailing list, so you need mail access.
Point your WWW browser to the {freebsd-questions}.
In the section titled "Subscribing to freebsd-questions" fill in the "Your email address" field; the other fields are optional.
[NOTE]
====
The password fields in the subscription form provide only mild security, but should prevent others from messing with your subscription. _Do not use a valuable password_ as it will occasionally be emailed back to you in cleartext.
The password fields in the subscription form provide only mild security, but should prevent others from messing with your subscription.
_Do not use a valuable password_ as it will occasionally be emailed back to you in cleartext.
====
You will receive a confirmation message from mailman; follow the included instructions to complete your subscription.
Finally, when you get the "Welcome" message from mailman telling you the details of the list and subscription area password, __please save it__. If you ever should want to leave the list, you will need the information there. See the next section for more details.
Finally, when you get the "Welcome" message from mailman telling you the details of the list and subscription area password, __please save it__.
If you ever should want to leave the list, you will need the information there.
See the next section for more details.
== How to Unsubscribe from FreeBSD-questions
When you subscribed to FreeBSD-questions, you got a welcome message from mailman. In this message, amongst other things, it told you how to unsubscribe. Here is a typical message:
When you subscribed to FreeBSD-questions, you got a welcome message from mailman.
In this message, amongst other things, it told you how to unsubscribe.
Here is a typical message:
....
Welcome to the
@ -104,18 +119,22 @@ From the URL specified in your "Welcome" message you may visit the "Account mana
A confirmation message will be sent to you from mailman; follow the included instructions to finish unsubscribing.
If you have done this, and you still cannot figure out what is going on, send a message to mailto:freebsd-questions-request@FreeBSD.org[freebsd-questions-request@FreeBSD.org], and they will sort things out for you. _Do not_ send a message to FreeBSD-questions: they cannot help you.
If you have done this, and you still cannot figure out what is going on, send a message to mailto:freebsd-questions-request@FreeBSD.org[freebsd-questions-request@FreeBSD.org], and they will sort things out for you.
_Do not_ send a message to FreeBSD-questions: they cannot help you.
== Should I ask `-questions` or `-hackers`?
Two mailing lists handle general questions about FreeBSD, `FreeBSD-questions` and `FreeBSD-hackers`. In some cases, it is not really clear which group you should ask. The following criteria should help for 99% of all questions, however:
Two mailing lists handle general questions about FreeBSD, `FreeBSD-questions` and `FreeBSD-hackers`.
In some cases, it is not really clear which group you should ask.
The following criteria should help for 99% of all questions, however:
. If the question is of a general nature, ask `FreeBSD-questions`. Examples might be questions about installing FreeBSD or the use of a particular UNIX(R) utility.
. If you think the question relates to a bug, but you are not sure, or you do not know how to look for it, send the message to `FreeBSD-questions`.
. If the question relates to a bug, and you are _sure_ that it is a bug (for example, you can pinpoint the place in the code where it happens, and you maybe have a fix), then send the message to `FreeBSD-hackers`.
. If the question relates to enhancements to FreeBSD, and you can make suggestions about how to implement them, then send the message to `FreeBSD-hackers`.
There are also a number of other link:{handbook}#eresources-mail[specialized mailing lists], which caters to more specific interests. The criteria above still apply, and it is in your interest to stick to them, since you are more likely to get good results that way.
There are also a number of other link:{handbook}#eresources-mail[specialized mailing lists], which caters to more specific interests.
The criteria above still apply, and it is in your interest to stick to them, since you are more likely to get good results that way.
== Before Submitting a Question
@ -134,15 +153,16 @@ When submitting a question to FreeBSD-questions, consider the following points:
* Not everybody who answers FreeBSD questions reads every message: they look at the subject line and decide whether it interests them. Clearly, it is in your interest to specify a subject. "FreeBSD problem" or "Help" are not enough. If you provide no subject at all, many people will not bother reading it. If your subject is not specific enough, the people who can answer it may not read it.
* Format your message so that it is legible, and PLEASE DO NOT SHOUT!!!!!. We appreciate that a lot of people do not speak English as their first language, and we try to make allowances for that, but it is really painful to try to read a message written full of typos or without any line breaks.
+
Do not underestimate the effect that a poorly formatted mail message has, not just on the FreeBSD-questions mailing list. Your mail message is all people see of you, and if it is poorly formatted, one line per paragraph, badly spelt, or full of errors, it will give people a poor impression of you.
Do not underestimate the effect that a poorly formatted mail message has, not just on the FreeBSD-questions mailing list.
Your mail message is all people see of you, and if it is poorly formatted, one line per paragraph, badly spelt, or full of errors, it will give people a poor impression of you.
+
A lot of badly formatted messages come from http://www.lemis.com/email.html[bad mailers or badly configured mailers]. The following mailers are known to send out badly formatted messages without you finding out about them:
A lot of badly formatted messages come from http://www.lemis.com/email.html[bad mailers or badly configured mailers].
The following mailers are known to send out badly formatted messages without you finding out about them:
** Eudora(R)
** exmh
** Microsoft(R) Exchange
** Microsoft(R) Outlook(R)
+
Try not to use MIME: a lot of people use mailers which do not get on very well with MIME.
* Make sure your time and time zone are set correctly. This may seem a little silly, since your message still gets there, but many of the people you are trying to reach get several hundred messages a day. They frequently sort the incoming messages by subject and by date, and if your message does not come before the first answer, they may assume they missed it and not bother to look.
@ -168,7 +188,8 @@ There is a judgement call here, of course, but the output of the man:dmesg[8] co
This redirects the information to the file [.filename]#/tmp/dmesg.out#.
* If you do all this, and you still do not get an answer, there could be other reasons. For example, the problem is so complicated that nobody knows the answer, or the person who does know the answer was offline. If you do not get an answer after, say, a week, it might help to re-send the message. If you do not get an answer to your second message, though, you are probably not going to get one from this forum. Resending the same message again and again will only make you unpopular.
To summarize, let's assume you know the answer to the following question (yes, it is the same one in each case). You choose which of these two questions you would be more prepared to answer:
To summarize, let's assume you know the answer to the following question (yes, it is the same one in each case).
You choose which of these two questions you would be more prepared to answer:
.Message 1
[example]
@ -201,7 +222,9 @@ Missing Operating System.
== How to Follow up to a Question
Often you will want to send in additional information to a question you have already sent. The best way to do this is to reply to your original message. This has three advantages:
Often you will want to send in additional information to a question you have already sent.
The best way to do this is to reply to your original message.
This has three advantages:
. You include the original message text, so people will know what you are talking about. Do not forget to trim unnecessary text out, though.
. The text in the subject line stays the same (you did remember to put one in, did you not?). Many mailers will sort messages by subject. This helps group messages together.
@ -214,7 +237,8 @@ Before you answer a question to FreeBSD-questions, consider:
. A lot of the points on submitting questions also apply to answering questions. Read them.
. Has somebody already answered the question? The easiest way to check this is to sort your incoming mail by subject: then (hopefully) you will see the question followed by any answers, all together.
+
If somebody has already answered it, it does not automatically mean that you should not send another answer. But it makes sense to read all the other answers first.
If somebody has already answered it, it does not automatically mean that you should not send another answer.
But it makes sense to read all the other answers first.
. Do you have something to contribute beyond what has already been said? In general, "Yeah, me too" answers do not help much, although there are exceptions, like when somebody is describing a problem they are having, and they do not know whether it is their fault or whether there is something wrong with the hardware or software. If you do send a "me too" answer, you should also include any further relevant information.
. Are you sure you understand the question? Very frequently, the person who asks the question is confused or does not express themselves very well. Even with the best understanding of the system, it is easy to send a reply which does not answer the question. This does not help: you will leave the person who submitted the question more frustrated or confused than ever. If nobody else answers, and you are not too sure either, you can always ask for more information.
. Are you sure your answer is correct? If not, wait a day or so. If nobody else comes up with a better answer, you can still reply and say, for example, "I do not know if this is correct, but since nobody else has replied, why don't you try replacing your ATAPI CDROM with a frog?".
@ -225,6 +249,8 @@ If somebody has already answered it, it does not automatically mean that you sho
. Most mailers change the subject line on a reply by prepending a text such as "Re: ". If your mailer does not do it automatically, you should do it manually.
. If the submitter did not abide by format conventions (lines too long, inappropriate subject line) _please_ fix it. In the case of an incorrect subject line (such as "HELP!!??"), change the subject line to (say) "Re: Difficulties with sync PPP (was: HELP!!??)". That way other people trying to follow the thread will have less difficulty following it.
+
In such cases, it is appropriate to say what you did and why you did it, but try not to be rude. If you find you can not answer without being rude, do not answer.
In such cases, it is appropriate to say what you did and why you did it, but try not to be rude.
If you find you can not answer without being rude, do not answer.
+
If you just want to reply to a message because of its bad format, just reply to the submitter, not to the list. You can just send him this message in reply, if you like.
If you just want to reply to a message because of its bad format, just reply to the submitter, not to the list.
You can just send him this message in reply, if you like.

123
documentation/content/en/articles/freebsd-releng/_index.adoc
View file

@ -46,15 +46,19 @@ toc::[]
[[introduction]]
== Introduction to the FreeBSD Release Engineering Process
Development of FreeBSD has a very specific workflow. In general, all changes to the FreeBSD base system are committed to the {branchHead} branch, which reflects the top of the source tree.
Development of FreeBSD has a very specific workflow.
In general, all changes to the FreeBSD base system are committed to the {branchHead} branch, which reflects the top of the source tree.
After a reasonable testing period, changes can then be merged to the {branchStable} branches. The default minimum timeframe before merging to {branchStable} branches is three (3) days.
After a reasonable testing period, changes can then be merged to the {branchStable} branches.
The default minimum timeframe before merging to {branchStable} branches is three (3) days.
Although a general rule to wait a minimum of three days before merging from {branchHead}, there are a few special circumstances where an immediate merge may be necessary, such as a critical security fix, or a bug fix that directly inhibits the release build process.
After several months, and the number of changes in the {branchStable} branch have grown significantly, it is time to release the next version of FreeBSD. These releases have been historically referred to as "point" releases.
After several months, and the number of changes in the {branchStable} branch have grown significantly, it is time to release the next version of FreeBSD.
These releases have been historically referred to as "point" releases.
In between releases from the {branchStable} branches, approximately every two (2) years, a release will be cut directly from {branchHead}. These releases have been historically referred to as "dot-zero" releases.
In between releases from the {branchStable} branches, approximately every two (2) years, a release will be cut directly from {branchHead}.
These releases have been historically referred to as "dot-zero" releases.
This article will highlight the workflow and responsibilities of the {teamRe} for both "dot-zero" and "point"' releases.
@ -87,7 +91,9 @@ Wrapping up the release cycle.
[[releng-prep]]
== General Information and Preparation
Approximately two months before the start of the release cycle, the {teamRe} decides on a schedule for the release. The schedule includes the various milestone points of the release cycle, such as freeze dates, branch dates, and build dates. For example:
Approximately two months before the start of the release cycle, the {teamRe} decides on a schedule for the release.
The schedule includes the various milestone points of the release cycle, such as freeze dates, branch dates, and build dates.
For example:
[.informaltable]
[cols="1,1", frame="none", options="header"]
@ -171,17 +177,23 @@ Items marked with "[*]" are "as needed".
If the release is being created from an existing {branchStable} branch, the KBI freeze date can be excluded, since the KBI is already considered frozen on established {branchStable} branches.
====
When writing the release cycle schedule, a number of things need to be taken into consideration, in particular milestones where the target date depends on predefined milestones upon which there is a dependency. For example, the Ports Collection release tag originates from the active quarterly branch at the time of the last `RC`. This in part defines which quarterly branch is used, when the release tag can happen, and what revision of the ports tree is used for the final `RELEASE` build.
When writing the release cycle schedule, a number of things need to be taken into consideration, in particular milestones where the target date depends on predefined milestones upon which there is a dependency.
For example, the Ports Collection release tag originates from the active quarterly branch at the time of the last `RC`.
This in part defines which quarterly branch is used, when the release tag can happen, and what revision of the ports tree is used for the final `RELEASE` build.
After general agreement on the schedule, the {teamRe} emails the schedule to the FreeBSD Developers.
It is somewhat typical that many developers will inform the {teamRe} about various works-in-progress. In some cases, an extension for the in-progress work will be requested, and in other cases, a request for "blanket approval" to a particular subset of the tree will be made.
It is somewhat typical that many developers will inform the {teamRe} about various works-in-progress.
In some cases, an extension for the in-progress work will be requested, and in other cases, a request for "blanket approval" to a particular subset of the tree will be made.
When such requests are made, it is important to make sure timelines (even if estimated) are discussed. For blanket approvals, the length of time for the blanket approval should be made clear. For example, a FreeBSD developer may request blanket approvals from the start of the code slush until the start of the `RC` builds.
When such requests are made, it is important to make sure timelines (even if estimated) are discussed.
For blanket approvals, the length of time for the blanket approval should be made clear.
For example, a FreeBSD developer may request blanket approvals from the start of the code slush until the start of the `RC` builds.
[NOTE]
====
In order to keep track of blanket approvals, the {teamRe} uses an internal repository to keep a running log of such requests, which defines the area upon which a blanket approval was granted, the author(s), when the blanket approval expires, and the reason the approval was granted. One example of this is granting blanket approval to [.filename]#release/doc/# to all {teamRe} members until the final `RC` to update the release notes and other release-related documentation.
In order to keep track of blanket approvals, the {teamRe} uses an internal repository to keep a running log of such requests, which defines the area upon which a blanket approval was granted, the author(s), when the blanket approval expires, and the reason the approval was granted.
One example of this is granting blanket approval to [.filename]#release/doc/# to all {teamRe} members until the final `RC` to update the release notes and other release-related documentation.
====
[NOTE]
@ -191,7 +203,9 @@ The {teamRe} also uses this repository to track pending approval requests that a
Depending on the underlying set of code in question, and the overall impact the set of code has on FreeBSD as a whole, such requests may be approved or denied by the {teamRe}.
The same applies to work-in-progress extensions. For example, in-progress work for a new device driver that is otherwise isolated from the rest of the tree may be granted an extension. A new scheduler, however, may not be feasible, especially if such dramatic changes do not exist in another branch.
The same applies to work-in-progress extensions.
For example, in-progress work for a new device driver that is otherwise isolated from the rest of the tree may be granted an extension.
A new scheduler, however, may not be feasible, especially if such dramatic changes do not exist in another branch.
The schedule is also added to the Project website, in the `doc/` repository, in [.filename]#head/en_US.ISO8859-1/htdocs/releases/{branchRevision}R/schedule.xml#. This file is continuously updated as the release cycle progresses.
@ -230,9 +244,11 @@ The code slush does not enforce commit approvals to the branch.
The code freeze marks the point in time where all commits to the branch require explicit approval from the {teamRe}.
The FreeBSD Subversion repository contains several hooks to perform sanity checks before any commit is actually committed to the tree. One of these hooks will evaluate if committing to a particular branch requires specific approval.
The FreeBSD Subversion repository contains several hooks to perform sanity checks before any commit is actually committed to the tree.
One of these hooks will evaluate if committing to a particular branch requires specific approval.
To enforce commit approvals by the {teamRe}, the Release Engineer updates [.filename]#base/svnadmin/conf/approvers#, and commits the change back to the repository. Once this is done, any change to the branch must include an "Approved by:" line in the commit message.
To enforce commit approvals by the {teamRe}, the Release Engineer updates [.filename]#base/svnadmin/conf/approvers#, and commits the change back to the repository.
Once this is done, any change to the branch must include an "Approved by:" line in the commit message.
The "Approved by:" line must match the second column in [.filename]#base/svnadmin/conf/approvers#, otherwise the commit will be rejected by the repository hooks.
@ -244,7 +260,8 @@ During the code freeze, FreeBSD committers are urged to follow the link:https://
[[releng-terms-kbi-freeze]]
=== The KBI/KPI Freeze
KBI/KPI stability implies that the caller of a function across two different releases of software that implement the function results in the same end state. The caller, whether it is a process, thread, or function, expects the function to operate in a certain way, otherwise the KBI/KPI stability on the branch is broken.
KBI/KPI stability implies that the caller of a function across two different releases of software that implement the function results in the same end state.
The caller, whether it is a process, thread, or function, expects the function to operate in a certain way, otherwise the KBI/KPI stability on the branch is broken.
[[releng-website]]
== Website Changes During the Release Cycle
@ -321,7 +338,8 @@ The relevant release-related documents exist in the [.filename]#doc# repository
[[releng-ports-beta-rc]]
=== Ports Changes During `BETA`, `RC`, and the Final `RELEASE`
For each build during the release cycle, the `MANIFEST` files containing the `SHA256` of the various distribution sets, such as `base.txz`, `kernel.txz`, and so on, are added to the package:misc/freebsd-release-manifests[] port. This allows utilities other than , such as package:ports-mgmt/poudriere[], to safely use these distribution sets by providing a mechanism through which the checksums can be verified.
For each build during the release cycle, the `MANIFEST` files containing the `SHA256` of the various distribution sets, such as `base.txz`, `kernel.txz`, and so on, are added to the package:misc/freebsd-release-manifests[] port.
This allows utilities other than , such as package:ports-mgmt/poudriere[], to safely use these distribution sets by providing a mechanism through which the checksums can be verified.
[[releng-head]]
== Release from {branchHead}
@ -331,20 +349,24 @@ This section describes the general procedures of the FreeBSD release cycle from
[[releng-head-builds-alpha]]
=== FreeBSD "`ALPHA`" Builds
Starting with the FreeBSD 10.0-RELEASE cycle, the notion of "`ALPHA`" builds was introduced. Unlike the `BETA` and `RC` builds, `ALPHA` builds are not included in the FreeBSD Release schedule.
Starting with the FreeBSD 10.0-RELEASE cycle, the notion of "`ALPHA`" builds was introduced.
Unlike the `BETA` and `RC` builds, `ALPHA` builds are not included in the FreeBSD Release schedule.
The idea behind `ALPHA` builds is to provide regular FreeBSD-provided builds before the creation of the {branchStable} branch.
FreeBSD `ALPHA` snapshots should be built approximately once a week.
For the first `ALPHA` build, the `BRANCH` value in [.filename]#sys/conf/newvers.sh# needs to be changed from `CURRENT` to `ALPHA1`. For subsequent `ALPHA` builds, increment each `ALPHA__N__` value by one.
For the first `ALPHA` build, the `BRANCH` value in [.filename]#sys/conf/newvers.sh# needs to be changed from `CURRENT` to `ALPHA1`.
For subsequent `ALPHA` builds, increment each `ALPHA__N__` value by one.
See <<releng-building>> for information on building the `ALPHA` images.
[[releng-head-branching]]
=== Creating the {branchStablex} Branch
When creating the {branchStable} branch, several changes are required in both the new {branchStable} branch and the {branchHead} branch. The files listed are relative to the repository root. To create the new {branchStablex} branch in Subversion:
When creating the {branchStable} branch, several changes are required in both the new {branchStable} branch and the {branchHead} branch.
The files listed are relative to the repository root.
To create the new {branchStablex} branch in Subversion:
[source,shell,subs="attributes"]
....
@ -453,7 +475,8 @@ This section describes the general procedures of the FreeBSD release cycle from
[[releng-stable-slush]]
=== FreeBSD `stable` Branch Code Slush
In preparation for the code freeze on a `stable` branch, several files need to be updated to reflect the release cycle is officially in progress. These files are all relative to the top-most level of the stable branch:
In preparation for the code freeze on a `stable` branch, several files need to be updated to reflect the release cycle is officially in progress.
These files are all relative to the top-most level of the stable branch:
[.informaltable]
[cols="1,1", frame="none", options="header"]
@ -482,7 +505,9 @@ In the `doc` repository, also update [.filename]#head/en_US.ISO8859-1/htdocs/rel
[[releng-stable-builds-beta]]
=== FreeBSD `BETA` Builds
Following the code slush, the next phase of the release cycle is the code freeze. This is the point at which all commits to the stable branch require explicit approval from the {teamRe}. This is enforced by pre-commit hooks in the Subversion repository by editing [.filename]#base/svnadmin/conf/approvers# to include a regular expression matching the {branchStablex} branch for the release:
Following the code slush, the next phase of the release cycle is the code freeze.
This is the point at which all commits to the stable branch require explicit approval from the {teamRe}.
This is enforced by pre-commit hooks in the Subversion repository by editing [.filename]#base/svnadmin/conf/approvers# to include a regular expression matching the {branchStablex} branch for the release:
[.programlisting,subs="attributes"]
....
@ -492,17 +517,23 @@ Following the code slush, the next phase of the release cycle is the code freeze
[NOTE]
====
There are two general exceptions to requiring commit approval during the release cycle. The first is any change that needs to be committed by the Release Engineer in order to proceed with the day-to-day workflow of the release cycle, the other is security fixes that may occur during the release cycle.
There are two general exceptions to requiring commit approval during the release cycle.
The first is any change that needs to be committed by the Release Engineer in order to proceed with the day-to-day workflow of the release cycle, the other is security fixes that may occur during the release cycle.
====
Once the code freeze is in effect, the next build from the branch is labeled `BETA1`. This is done by updating the `BRANCH` value in [.filename]#sys/conf/newvers.sh# from `PRERELEASE` to `BETA1`.
Once the code freeze is in effect, the next build from the branch is labeled `BETA1`.
This is done by updating the `BRANCH` value in [.filename]#sys/conf/newvers.sh# from `PRERELEASE` to `BETA1`.
Once this is done, the first set of `BETA` builds are started. Subsequent `BETA` builds do not require updates to any files other than [.filename]#sys/conf/newvers.sh#, incrementing the `BETA` build number.
Once this is done, the first set of `BETA` builds are started.
Subsequent `BETA` builds do not require updates to any files other than [.filename]#sys/conf/newvers.sh#, incrementing the `BETA` build number.
[[releng-stable-branching]]
=== Creating the {branchRelengx} Branch
When the first `RC` (Release Candidate) build is ready to begin, the {branchReleng} branch is created. This is a multi-step process that must be done in a specific order, in order to avoid anomalies such as overlaps with `__FreeBSD_version` values, for example. The paths listed below are relative to the repository root. The order of commits and what to change are:
When the first `RC` (Release Candidate) build is ready to begin, the {branchReleng} branch is created.
This is a multi-step process that must be done in a specific order, in order to avoid anomalies such as overlaps with `__FreeBSD_version` values, for example.
The paths listed below are relative to the repository root.
The order of commits and what to change are:
[source,shell,subs="attributes"]
....
@ -565,7 +596,8 @@ This section describes the build scripts used by {teamRe} to produce development
Prior to FreeBSD 9.0-RELEASE, [.filename]#src/release/Makefile# was updated to support , and the [.filename]#src/release/generate-release.sh# script was introduced as a wrapper to automate invoking the targets.
Prior to FreeBSD 9.2-RELEASE, [.filename]#src/release/release.sh# was introduced, which heavily based on [.filename]#src/release/generate-release.sh# included support to specify configuration files to override various options and environment variables. Support for configuration files provided support for cross building each architecture for a release by specifying a separate configuration file for each invocation.
Prior to FreeBSD 9.2-RELEASE, [.filename]#src/release/release.sh# was introduced, which heavily based on [.filename]#src/release/generate-release.sh# included support to specify configuration files to override various options and environment variables.
Support for configuration files provided support for cross building each architecture for a release by specifying a separate configuration file for each invocation.
As a brief example of using [.filename]#src/release/release.sh# to build a single release in [.filename]#/scratch#:
@ -603,11 +635,13 @@ The wrapper script is called [.filename]#thermite.sh#, which is available in the
Using [.filename]#thermite.sh# is covered in <<releng-build-snapshot>> and <<releng-build-release>>.
Each architecture and individual kernel have their own configuration file used by [.filename]#release.sh#. Each branch has its own [.filename]#defaults-X.conf# configuration which contains entries common throughout each architecture, where overrides or special variables are set and/or overridden in the per-build files.
Each architecture and individual kernel have their own configuration file used by [.filename]#release.sh#.
Each branch has its own [.filename]#defaults-X.conf# configuration which contains entries common throughout each architecture, where overrides or special variables are set and/or overridden in the per-build files.
The per-build configuration file naming scheme is in the form of [.filename]#${revision}-${TARGET_ARCH}-${KERNCONF}-${type}.conf#, where the uppercase variables are equivalent to what uses in the build system, and lowercase variables are set within the configuration files, mapping to the major version of the respective branch.
Each branch also has its own [.filename]#builds-X.conf# configuration, which is used by [.filename]#thermite.sh#. The [.filename]#thermite.sh# script iterates through each ${revision}, ${TARGET_ARCH}, ${KERNCONF}, and ${type} value, creating a master list of what to build. However, a given combination from the list will only be built if the respective configuration file exists, which is where the naming scheme above is relevant.
Each branch also has its own [.filename]#builds-X.conf# configuration, which is used by [.filename]#thermite.sh#. The [.filename]#thermite.sh# script iterates through each ${revision}, ${TARGET_ARCH}, ${KERNCONF}, and ${type} value, creating a master list of what to build.
However, a given combination from the list will only be built if the respective configuration file exists, which is where the naming scheme above is relevant.
There are two paths of file sourcing:
@ -630,14 +664,17 @@ The official release build machines have a specific filesystem layout, which usi
The build scripts reside in [.filename]#/releng/scripts-snapshot/scripts# or [.filename]#/releng/scripts-release/scripts# respectively, to avoid collisions between an `RC` build from a releng branch versus a `STABLE` snapshot from the respective stable branch.
A separate dataset exists for the final build images, [.filename]#/snap/ftp#. This directory contains both snapshots and releases directories. They are only used if the `EVERYTHINGISFINE` variable is defined in [.filename]#main.conf#.
A separate dataset exists for the final build images, [.filename]#/snap/ftp#. This directory contains both snapshots and releases directories.
They are only used if the `EVERYTHINGISFINE` variable is defined in [.filename]#main.conf#.
[NOTE]
====
The `EVERYTHINGISFINE` variable name was chosen to avoid colliding with a variable that might be possibly set in the user environment, accidentally enabling the behavior that depends on it being defined.
====
As [.filename]#thermite.sh# iterates through the master list of combinations and locates the per-build configuration file, a ZFS dataset is created under [.filename]#/releng#, such as [.filename]#/releng/12-amd64-GENERIC-snap#. The `src/`, `ports/`, and `doc/` trees are checked out to separate ZFS datasets, such as [.filename]#/releng/12-src-snap#, which are then cloned and mounted into the respective build datasets. This is done to avoid checking out a given tree more than once.
As [.filename]#thermite.sh# iterates through the master list of combinations and locates the per-build configuration file, a ZFS dataset is created under [.filename]#/releng#, such as [.filename]#/releng/12-amd64-GENERIC-snap#.
The `src/`, `ports/`, and `doc/` trees are checked out to separate ZFS datasets, such as [.filename]#/releng/12-src-snap#, which are then cloned and mounted into the respective build datasets.
This is done to avoid checking out a given tree more than once.
Assuming these filesystem paths, [.filename]#thermite.sh# would be invoked as:
@ -664,19 +701,25 @@ The generated output should be double-checked for correctness, and the email its
[NOTE]
====
These helper scripts only apply to development snapshot builds. Announcements during the release cycle (excluding the final release announcement) are created from an email template. A sample of the email template currently used can be found link:here[here].
These helper scripts only apply to development snapshot builds.
Announcements during the release cycle (excluding the final release announcement) are created from an email template.
A sample of the email template currently used can be found link:here[here].
====
[[releng-build-release]]
=== Building FreeBSD Releases
Similar to building FreeBSD development snapshots, [.filename]#thermite.sh# would be invoked the same way. The difference between development snapshots and release builds, `BETA` and `RC` included, is that the configuration files must be named with `release` instead of `snap` as the type, as mentioned above.
Similar to building FreeBSD development snapshots, [.filename]#thermite.sh# would be invoked the same way.
The difference between development snapshots and release builds, `BETA` and `RC` included, is that the configuration files must be named with `release` instead of `snap` as the type, as mentioned above.
In addition, the `BUILDTYPE` and `types` must be changed from `snap` to `release` in [.filename]#defaults-12.conf# and [.filename]#builds-12.conf#, respectively.
When building `BETA`, `RC`, and the final `RELEASE`, also statically set `BUILDSVNREV` to the revision on the branch reflecting the name change, `BUILDDATE` to the date the builds are started in `YYYYMMDD` format. If the `doc/` and `ports/` trees have been tagged, also set `PORTBRANCH` and `DOCBRANCH` to the relevant tag path in the Subversion repository, replacing `HEAD` with the last changed revision. Also set `releasesrc` in [.filename]#builds-12.conf# to the relevant branch, such as {branchStablex} or {branchRelengx}.
When building `BETA`, `RC`, and the final `RELEASE`, also statically set `BUILDSVNREV` to the revision on the branch reflecting the name change, `BUILDDATE` to the date the builds are started in `YYYYMMDD` format.
If the `doc/` and `ports/` trees have been tagged, also set `PORTBRANCH` and `DOCBRANCH` to the relevant tag path in the Subversion repository, replacing `HEAD` with the last changed revision.
Also set `releasesrc` in [.filename]#builds-12.conf# to the relevant branch, such as {branchStablex} or {branchRelengx}.
During the release cycle, a copy of [.filename]#CHECKSUM.SHA512# and [.filename]#CHECKSUM.SHA256# for each architecture are stored in the {teamRe} internal repository in addition to being included in the various announcement emails. Each [.filename]#MANIFEST# containing the hashes of [.filename]#base.txz#, [.filename]#kernel.txz#, etc. are added to package:misc/freebsd-release-manifests[] in the Ports Collection, as well.
During the release cycle, a copy of [.filename]#CHECKSUM.SHA512# and [.filename]#CHECKSUM.SHA256# for each architecture are stored in the {teamRe} internal repository in addition to being included in the various announcement emails.
Each [.filename]#MANIFEST# containing the hashes of [.filename]#base.txz#, [.filename]#kernel.txz#, etc. are added to package:misc/freebsd-release-manifests[] in the Ports Collection, as well.
In preparation for the release build, several files need to be updated:
@ -696,7 +739,9 @@ In preparation for the release build, several files need to be updated:
|Replace `__FreeBSD_version` with the value in [.filename]#sys/sys/param.h#
|===
After building the final `RELEASE`, the {branchRelengx} branch is tagged as {branchReleasex} using the revision from which the `RELEASE` was built. Similar to creating the {branchStablex} and {branchRelengx} branches, this is done with `svn cp`. From the repository root:
After building the final `RELEASE`, the {branchRelengx} branch is tagged as {branchReleasex} using the revision from which the `RELEASE` was built.
Similar to creating the {branchStablex} and {branchRelengx} branches, this is done with `svn cp`.
From the repository root:
[source,shell,subs="attributes"]
....
@ -716,7 +761,8 @@ Staging FreeBSD snapshots and releases is a two part process:
* Creating the directory structure to match the hierarchy on `ftp-master`
+
If `EVERYTHINGISFINE` is defined in the build configuration files, [.filename]#main.conf# in the case of the build scripts referenced above, this happens automatically in the after the build is complete, creating the directory structure in [.filename]#${DESTDIR}/R/ftp-stage# with a path structure matching what is expected on `ftp-master`. This is equivalent to running the following in the directly:
If `EVERYTHINGISFINE` is defined in the build configuration files, [.filename]#main.conf# in the case of the build scripts referenced above, this happens automatically in the after the build is complete, creating the directory structure in [.filename]#${DESTDIR}/R/ftp-stage# with a path structure matching what is expected on `ftp-master`.
This is equivalent to running the following in the directly:
+
[source,shell,subs="attributes"]
....
@ -736,7 +782,8 @@ On `ftp-master` in the FreeBSD Project infrastructure, this step requires `root`
[[releng-mirrors-publishing]]
=== Publishing FreeBSD Installation Media
Once the images are staged in [.filename]#/archive/tmp/#, they are ready to be made public by putting them in [.filename]#/archive/pub/FreeBSD#. In order to reduce propagation time, is used to create hard links from [.filename]#/archive/tmp# to [.filename]#/archive/pub/FreeBSD#.
Once the images are staged in [.filename]#/archive/tmp/#, they are ready to be made public by putting them in [.filename]#/archive/pub/FreeBSD#.
In order to reduce propagation time, is used to create hard links from [.filename]#/archive/tmp# to [.filename]#/archive/pub/FreeBSD#.
[NOTE]
====
@ -769,7 +816,8 @@ This section describes general post-release tasks.
[[releng-wrapup-en]]
=== Post-Release Errata Notices
As the release cycle approaches conclusion, it is common to have several EN (Errata Notice) candidates to address issues that were discovered late in the cycle. Following the release, the {teamRe} and the {teamSecteam} revisit changes that were not approved prior to the final release, and depending on the scope of the change in question, may issue an EN.
As the release cycle approaches conclusion, it is common to have several EN (Errata Notice) candidates to address issues that were discovered late in the cycle.
Following the release, the {teamRe} and the {teamSecteam} revisit changes that were not approved prior to the final release, and depending on the scope of the change in question, may issue an EN.
[NOTE]
====
@ -780,7 +828,8 @@ To request an Errata Notice after a release cycle has completed, a developer sho
The completed Errata Notice template should be emailed together with either a patch against the {branchReleng} branch or a list of revisions from the {branchStable} branch.
For Errata Notice requests immediately following the release, the request should be emailed to both the {teamRe} and the {teamSecteam}. Once the {branchReleng} branch has been handed over to the {teamSecteam} as described in <<releng-wrapup-handoff>>, Errata Notice requests should be sent to the {teamSecteam}.
For Errata Notice requests immediately following the release, the request should be emailed to both the {teamRe} and the {teamSecteam}.
Once the {branchReleng} branch has been handed over to the {teamSecteam} as described in <<releng-wrapup-handoff>>, Errata Notice requests should be sent to the {teamSecteam}.
[[releng-wrapup-handoff]]
=== Handoff to the {teamSecteam}

55
documentation/content/en/articles/freebsd-update-server/_index.adoc
View file

@ -24,7 +24,9 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
This article describes building an internal FreeBSD Update Server. The https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] is written by `{cperciva}`, Security Officer Emeritus of FreeBSD. For users that think it is convenient to update their systems against an official update server, building their own FreeBSD Update Server may help to extend its functionality by supporting manually-tweaked FreeBSD releases or by providing a local mirror that will allow faster updates for a number of machines.
This article describes building an internal FreeBSD Update Server.
The https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] is written by `{cperciva}`, Security Officer Emeritus of FreeBSD.
For users that think it is convenient to update their systems against an official update server, building their own FreeBSD Update Server may help to extend its functionality by supporting manually-tweaked FreeBSD releases or by providing a local mirror that will allow faster updates for a number of machines.
'''
@ -38,7 +40,11 @@ This article was subsequently printed at https://people.freebsd.org/~jgh/files/f
[[introduction]]
== Introduction
Experienced users or administrators are often responsible for several machines or environments. They understand the difficult demands and challenges of maintaining such an infrastructure. Running a FreeBSD Update Server makes it easier to deploy security and software patches to selected test machines before rolling them out to production. It also means a number of systems can be updated from the local network rather than a potentially slower Internet connection. This article outlines the steps involved in creating an internal FreeBSD Update Server.
Experienced users or administrators are often responsible for several machines or environments.
They understand the difficult demands and challenges of maintaining such an infrastructure.
Running a FreeBSD Update Server makes it easier to deploy security and software patches to selected test machines before rolling them out to production.
It also means a number of systems can be updated from the local network rather than a potentially slower Internet connection.
This article outlines the steps involved in creating an internal FreeBSD Update Server.
[[prerequisites]]
== Prerequisites
@ -66,7 +72,8 @@ Download the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/
% svn co https://svn.freebsd.org/base/user/cperciva/freebsd-update-build freebsd-update-server
....
Update [.filename]#scripts/build.conf# appropriately. It is sourced during all build operations.
Update [.filename]#scripts/build.conf# appropriately.
It is sourced during all build operations.
Here is the default [.filename]#build.conf#, which should be modified to suit your environment.
@ -154,7 +161,9 @@ export EOL=1275289200 <.>
[[build]]
== Building Update Code
The first step is to run [.filename]#scripts/make.sh#. This will build some binaries, create directories, and generate an RSA signing key used for approving builds. In this step, a passphrase will have to be supplied for the final creation of the signing key.
The first step is to run [.filename]#scripts/make.sh#.
This will build some binaries, create directories, and generate an RSA signing key used for approving builds.
In this step, a passphrase will have to be supplied for the final creation of the signing key.
[source,shell]
....
@ -181,7 +190,8 @@ Verifying - enter aes-256-cbc encryption password:
[NOTE]
====
Keep a note of the generated key fingerprint. This value is required in [.filename]#/etc/freebsd-update.conf# for binary updates.
Keep a note of the generated key fingerprint.
This value is required in [.filename]#/etc/freebsd-update.conf# for binary updates.
====
At this point, we are ready to stage a build.
@ -236,12 +246,14 @@ world|base|/usr/lib/libalias_ftp.a
...
....
Then the build of the world is performed again, with world patches. A more detailed explanation may be found in [.filename]#scripts/build.subr#.
Then the build of the world is performed again, with world patches.
A more detailed explanation may be found in [.filename]#scripts/build.subr#.
[WARNING]
====
During this second build cycle, the network time protocol daemon, man:ntpd[8], is turned off. Per `{cperciva}`, Security Officer Emeritus of FreeBSD, "the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] build code needs to identify timestamps which are stored in files so that they can be ignored when comparing builds to determine which files need to be updated. This timestamp-finding works by doing two builds 400 days apart and comparing the results."
During this second build cycle, the network time protocol daemon, man:ntpd[8], is turned off.
Per `{cperciva}`, Security Officer Emeritus of FreeBSD, "the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] build code needs to identify timestamps which are stored in files so that they can be ignored when comparing builds to determine which files need to be updated.
This timestamp-finding works by doing two builds 400 days apart and comparing the results."
====
[source,shell]
@ -319,7 +331,9 @@ they look sensible, then run
to sign the release.
....
Approve the build if everything is correct. More information on determining this can be found in the distributed source file named [.filename]#USAGE#. Execute [.filename]#scripts/approve.sh#, as directed. This will sign the release, and move components into a staging area suitable for uploading.
Approve the build if everything is correct.
More information on determining this can be found in the distributed source file named [.filename]#USAGE#. Execute [.filename]#scripts/approve.sh#, as directed.
This will sign the release, and move components into a staging area suitable for uploading.
[source,shell]
....
@ -357,13 +371,17 @@ In the event update code needs to be re-uploaded, this may be done by changing t
====
The uploaded files will need to be in the document root of the webserver in order for updates to be distributed. The exact configuration will vary depending on the web server used. For the Apache web server, please refer to the link:{handbook}#network-apache[Configuration of Apache servers] section in the Handbook.
The uploaded files will need to be in the document root of the webserver in order for updates to be distributed.
The exact configuration will vary depending on the web server used.
For the Apache web server, please refer to the link:{handbook}#network-apache[Configuration of Apache servers] section in the Handbook.
Update client's `KeyPrint` and `ServerName` in [.filename]#/etc/freebsd-update.conf#, and perform updates as instructed in the link:{handbook}#updating-upgrading-freebsdupdate[FreeBSD Update] section of the Handbook.
[IMPORTANT]
====
In order for FreeBSD Update Server to work properly, updates for both the _current_ release and the release _one wants to upgrade to_ need to be built. This is necessary for determining the differences of files between releases. For example, when upgrading a FreeBSD system from 7.1-RELEASE to 7.2-RELEASE, updates will need to be built and uploaded to your distribution server for both versions.
In order for FreeBSD Update Server to work properly, updates for both the _current_ release and the release _one wants to upgrade to_ need to be built.
This is necessary for determining the differences of files between releases.
For example, when upgrading a FreeBSD system from 7.1-RELEASE to 7.2-RELEASE, updates will need to be built and uploaded to your distribution server for both versions.
====
For reference, the entire run of link:../../source/articles/freebsd-update-server/init.txt[init.sh] is attached.
@ -388,9 +406,14 @@ Create the patch directory of the respective release under [.filename]#/usr/loca
% cd /usr/local/freebsd-update-server/patches/7.1-RELEASE
....
As an example, take the patch for man:named[8]. Read the advisory, and grab the necessary file from link:https://www.FreeBSD.org/security/advisories/[FreeBSD Security Advisories]. More information on interpreting the advisory, can be found in the link:{handbook}#security-advisories[FreeBSD Handbook].
As an example, take the patch for man:named[8].
Read the advisory, and grab the necessary file from link:https://www.FreeBSD.org/security/advisories/[FreeBSD Security Advisories].
More information on interpreting the advisory, can be found in the link:{handbook}#security-advisories[FreeBSD Handbook].
In the https://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc[security brief], this advisory is called `SA-09:12.bind`. After downloading the file, it is required to rename the file to an appropriate patch level. It is suggested to keep this consistent with official FreeBSD patch levels, but its name may be freely chosen. For this build, let us follow the currently established practice of FreeBSD and call this `p7`. Rename the file:
In the https://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc[security brief], this advisory is called `SA-09:12.bind`.
After downloading the file, it is required to rename the file to an appropriate patch level.
It is suggested to keep this consistent with official FreeBSD patch levels, but its name may be freely chosen.
For this build, let us follow the currently established practice of FreeBSD and call this `p7`. Rename the file:
[source,shell]
....
@ -399,7 +422,8 @@ In the https://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc[securit
[NOTE]
====
When running a patch level build, it is assumed that previous patches are in place. When a patch build is run, it will run all patches contained in the patch directory.
When running a patch level build, it is assumed that previous patches are in place.
When a patch build is run, it will run all patches contained in the patch directory.
There can be custom patches added to any build. Use the number zero, or any other number.
====
@ -410,7 +434,8 @@ There can be custom patches added to any build. Use the number zero, or any othe
It is up to the administrator of the FreeBSD Update Server to take appropriate measures to verify the authenticity of every patch.
====
At this point, a _diff_ is ready to be built. The software checks first to see if a [.filename]#scripts/init.sh# has been run on the respective release prior to running the diff build.
At this point, a _diff_ is ready to be built.
The software checks first to see if a [.filename]#scripts/init.sh# has been run on the respective release prior to running the diff build.
[source,shell]
....

125
documentation/content/en/articles/geom-class/_index.adoc
View file

@ -22,7 +22,8 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
This text documents some starting points in developing GEOM classes, and kernel modules in general. It is assumed that the reader is familiar with C userland programming.
This text documents some starting points in developing GEOM classes, and kernel modules in general.
It is assumed that the reader is familiar with C userland programming.
'''
@ -34,7 +35,8 @@ toc::[]
[[intro-docs]]
=== Documentation
Documentation on kernel programming is scarce - it is one of few areas where there is nearly nothing in the way of friendly tutorials, and the phrase "use the source!" really holds true. However, there are some bits and pieces (some of them seriously outdated) floating around that should be studied before beginning to code:
Documentation on kernel programming is scarce - it is one of few areas where there is nearly nothing in the way of friendly tutorials, and the phrase "use the source!" really holds true.
However, there are some bits and pieces (some of them seriously outdated) floating around that should be studied before beginning to code:
* The link:{developers-handbook}[FreeBSD Developer's Handbook] - part of the documentation project, it does not contain anything specific to kernel programming, but rather some general useful information.
* The link:{arch-handbook}[FreeBSD Architecture Handbook] - also from the documentation project, contains descriptions of several low-level facilities and procedures. The most important chapter is 13, link:{arch-handbook}#driverbasics[Writing FreeBSD device drivers].
@ -47,9 +49,14 @@ Documentation on kernel programming is scarce - it is one of few areas where the
[[prelim]]
== Preliminaries
The best way to do kernel development is to have (at least) two separate computers. One of these would contain the development environment and sources, and the other would be used to test the newly written code by network-booting and network-mounting filesystems from the first one. This way if the new code contains bugs and crashes the machine, it will not mess up the sources (and other "live" data). The second system does not even require a proper display. Instead, it could be connected with a serial cable or KVM to the first one.
The best way to do kernel development is to have (at least) two separate computers.
One of these would contain the development environment and sources, and the other would be used to test the newly written code by network-booting and network-mounting filesystems from the first one.
This way if the new code contains bugs and crashes the machine, it will not mess up the sources (and other "live" data).
The second system does not even require a proper display.
Instead, it could be connected with a serial cable or KVM to the first one.
But, since not everybody has two or more computers handy, there are a few things that can be done to prepare an otherwise "live" system for developing kernel code. This setup is also applicable for developing in a http://www.vmware.com/[VMWare] or http://www.qemu.org/[QEmu] virtual machine (the next best thing after a dedicated development machine).
But, since not everybody has two or more computers handy, there are a few things that can be done to prepare an otherwise "live" system for developing kernel code.
This setup is also applicable for developing in a http://www.vmware.com/[VMWare] or http://www.qemu.org/[QEmu] virtual machine (the next best thing after a dedicated development machine).
[[prelim-system]]
=== Modifying a System for Development
@ -77,9 +84,12 @@ For debugging crash dumps, a kernel with debug symbols is needed:
makeoptions DEBUG=-g
....
With the usual way of installing the kernel (`make installkernel`) the debug kernel will not be automatically installed. It is called [.filename]#kernel.debug# and located in [.filename]#/usr/obj/usr/src/sys/KERNELNAME/#. For convenience it should be copied to [.filename]#/boot/kernel/#.
With the usual way of installing the kernel (`make installkernel`) the debug kernel will not be automatically installed.
It is called [.filename]#kernel.debug# and located in [.filename]#/usr/obj/usr/src/sys/KERNELNAME/#.
For convenience it should be copied to [.filename]#/boot/kernel/#.
Another convenience is enabling the kernel debugger so you can examine a kernel panic when it happens. For this, enter the following lines in your kernel configuration file:
Another convenience is enabling the kernel debugger so you can examine a kernel panic when it happens.
For this, enter the following lines in your kernel configuration file:
[.programlisting]
....
@ -95,7 +105,12 @@ For this to work you might need to set a sysctl (if it is not on by default):
debug.debugger_on_panic=1
....
Kernel panics will happen, so care should be taken with the filesystem cache. In particular, having softupdates might mean the latest file version could be lost if a panic occurs before it is committed to storage. Disabling softupdates yields a great performance hit, and still does not guarantee data consistency. Mounting filesystem with the "sync" option is needed for that. For a compromise, the softupdates cache delays can be shortened. There are three sysctl's that are useful for this (best to be set in [.filename]#/etc/sysctl.conf#):
Kernel panics will happen, so care should be taken with the filesystem cache.
In particular, having softupdates might mean the latest file version could be lost if a panic occurs before it is committed to storage.
Disabling softupdates yields a great performance hit, and still does not guarantee data consistency.
Mounting filesystem with the "sync" option is needed for that.
For a compromise, the softupdates cache delays can be shortened.
There are three sysctl's that are useful for this (best to be set in [.filename]#/etc/sysctl.conf#):
[.programlisting]
....
@ -106,7 +121,13 @@ kern.metadelay=3
The numbers represent seconds.
For debugging kernel panics, kernel core dumps are required. Since a kernel panic might make filesystems unusable, this crash dump is first written to a raw partition. Usually, this is the swap partition. This partition must be at least as large as the physical RAM in the machine. On the next boot, the dump is copied to a regular file. This happens after filesystems are checked and mounted, and before swap is enabled. This is controlled with two [.filename]#/etc/rc.conf# variables:
For debugging kernel panics, kernel core dumps are required.
Since a kernel panic might make filesystems unusable, this crash dump is first written to a raw partition.
Usually, this is the swap partition.
This partition must be at least as large as the physical RAM in the machine.
On the next boot, the dump is copied to a regular file.
This happens after filesystems are checked and mounted, and before swap is enabled.
This is controlled with two [.filename]#/etc/rc.conf# variables:
[.programlisting]
....
@ -116,7 +137,8 @@ dumpdir="/usr/core
The `dumpdev` variable specifies the swap partition and `dumpdir` tells the system where in the filesystem to relocate the core dump on reboot.
Writing kernel core dumps is slow and takes a long time so if you have lots of memory (>256M) and lots of panics it could be frustrating to sit and wait while it is done (twice - first to write it to swap, then to relocate it to filesystem). It is convenient then to limit the amount of RAM the system will use via a [.filename]#/boot/loader.conf# tunable:
Writing kernel core dumps is slow and takes a long time so if you have lots of memory (>256M) and lots of panics it could be frustrating to sit and wait while it is done (twice - first to write it to swap, then to relocate it to filesystem).
It is convenient then to limit the amount of RAM the system will use via a [.filename]#/boot/loader.conf# tunable:
[.programlisting]
....
@ -130,19 +152,23 @@ If the panics are frequent and filesystems large (or you simply do not trust sof
background_fsck="NO"
....
This way, the filesystems will always get checked when needed. Note that with background fsck, a new panic could happen while it is checking the disks. Again, the safest way is not to have many local filesystems by using another computer as an NFS server.
This way, the filesystems will always get checked when needed.
Note that with background fsck, a new panic could happen while it is checking the disks.
Again, the safest way is not to have many local filesystems by using another computer as an NFS server.
[[prelim-starting]]
=== Starting the Project
For the purpose of creating a new GEOM class, an empty subdirectory has to be created under an arbitrary user-accessible directory. You do not have to create the module directory under [.filename]#/usr/src#.
For the purpose of creating a new GEOM class, an empty subdirectory has to be created under an arbitrary user-accessible directory.
You do not have to create the module directory under [.filename]#/usr/src#.
[[prelim-makefile]]
=== The Makefile
It is good practice to create [.filename]#Makefiles# for every nontrivial coding project, which of course includes kernel modules.
Creating the [.filename]#Makefile# is simple thanks to an extensive set of helper routines provided by the system. In short, here is how a minimal [.filename]#Makefile# looks for a kernel module:
Creating the [.filename]#Makefile# is simple thanks to an extensive set of helper routines provided by the system.
In short, here is how a minimal [.filename]#Makefile# looks for a kernel module:
[.programlisting]
....
@ -152,7 +178,8 @@ KMOD=geom_journal
.include <bsd.kmod.mk>
....
This [.filename]#Makefile# (with changed filenames) will do for any kernel module, and a GEOM class can reside in just one kernel module. If more than one file is required, list it in the `SRCS` variable, separated with whitespace from other filenames.
This [.filename]#Makefile# (with changed filenames) will do for any kernel module, and a GEOM class can reside in just one kernel module.
If more than one file is required, list it in the `SRCS` variable, separated with whitespace from other filenames.
[[kernelprog]]
== On FreeBSD Kernel Programming
@ -160,7 +187,9 @@ This [.filename]#Makefile# (with changed filenames) will do for any kernel modul
[[kernelprog-memalloc]]
=== Memory Allocation
See man:malloc[9]. Basic memory allocation is only slightly different than its userland equivalent. Most notably, `malloc`() and `free`() accept additional parameters as is described in the man page.
See man:malloc[9].
Basic memory allocation is only slightly different than its userland equivalent.
Most notably, `malloc`() and `free`() accept additional parameters as is described in the man page.
A "malloc type" must be declared in the declaration section of a source file, like this:
@ -171,23 +200,33 @@ A "malloc type" must be declared in the declaration section of a source file, li
To use this macro, [.filename]#sys/param.h#, [.filename]#sys/kernel.h# and [.filename]#sys/malloc.h# headers must be included.
There is another mechanism for allocating memory, the UMA (Universal Memory Allocator). See man:uma[9] for details, but it is a special type of allocator mainly used for speedy allocation of lists comprised of same-sized items (for example, dynamic arrays of structs).
There is another mechanism for allocating memory, the UMA (Universal Memory Allocator).
See man:uma[9] for details, but it is a special type of allocator mainly used for speedy allocation of lists comprised of same-sized items (for example, dynamic arrays of structs).
[[kernelprog-lists]]
=== Lists and Queues
See man:queue[3]. There are a LOT of cases when a list of things needs to be maintained. Fortunately, this data structure is implemented (in several ways) by C macros included in the system. The most used list type is TAILQ because it is the most flexible. It is also the one with largest memory requirements (its elements are doubly-linked) and also the slowest (although the speed variation is on the order of several CPU instructions more, so it should not be taken seriously).
See man:queue[3].
There are a LOT of cases when a list of things needs to be maintained.
Fortunately, this data structure is implemented (in several ways) by C macros included in the system.
The most used list type is TAILQ because it is the most flexible.
It is also the one with largest memory requirements (its elements are doubly-linked) and also the slowest (although the speed variation is on the order of several CPU instructions more, so it should not be taken seriously).
If data retrieval speed is very important, see man:tree[3] and man:hashinit[9].
[[kernelprog-bios]]
=== BIOs
Structure `bio` is used for any and all Input/Output operations concerning GEOM. It basically contains information about what device ('provider') should satisfy the request, request type, offset, length, pointer to a buffer, and a bunch of "user-specific" flags and fields that can help implement various hacks.
Structure `bio` is used for any and all Input/Output operations concerning GEOM.
It basically contains information about what device ('provider') should satisfy the request, request type, offset, length, pointer to a buffer, and a bunch of "user-specific" flags and fields that can help implement various hacks.
The important thing here is that ``bio``s are handled asynchronously. That means that, in most parts of the code, there is no analogue to userland's man:read[2] and man:write[2] calls that do not return until a request is done. Rather, a developer-supplied function is called as a notification when the request gets completed (or results in error).
The important thing here is that ``bio``s are handled asynchronously.
That means that, in most parts of the code, there is no analogue to userland's man:read[2] and man:write[2] calls that do not return until a request is done.
Rather, a developer-supplied function is called as a notification when the request gets completed (or results in error).
The asynchronous programming model (also called "event-driven") is somewhat harder than the much more used imperative one used in userland (at least it takes a while to get used to it). In some cases the helper routines `g_write_data`() and `g_read_data`() can be used, but __not always__. In particular, they cannot be used when a mutex is held; for example, the GEOM topology mutex or the internal mutex held during the `.start`() and `.stop`() functions.
The asynchronous programming model (also called "event-driven") is somewhat harder than the much more used imperative one used in userland (at least it takes a while to get used to it).
In some cases the helper routines `g_write_data`() and `g_read_data`() can be used, but __not always__.
In particular, they cannot be used when a mutex is held; for example, the GEOM topology mutex or the internal mutex held during the `.start`() and `.stop`() functions.
[[geom]]
== On GEOM Programming
@ -195,12 +234,15 @@ The asynchronous programming model (also called "event-driven") is somewhat hard
[[geom-ggate]]
=== Ggate
If maximum performance is not needed, a much simpler way of making a data transformation is to implement it in userland via the ggate (GEOM gate) facility. Unfortunately, there is no easy way to convert between, or even share code between the two approaches.
If maximum performance is not needed, a much simpler way of making a data transformation is to implement it in userland via the ggate (GEOM gate) facility.
Unfortunately, there is no easy way to convert between, or even share code between the two approaches.
[[geom-class]]
=== GEOM Class
GEOM classes are transformations on the data. These transformations can be combined in a tree-like fashion. Instances of GEOM classes are called __geoms__.
GEOM classes are transformations on the data.
These transformations can be combined in a tree-like fashion.
Instances of GEOM classes are called __geoms__.
Each GEOM class has several "class methods" that get called when there is no geom instance available (or they are simply not bound to a single instance):
@ -219,13 +261,17 @@ These functions are called from the g_event kernel thread.
[[geom-softc]]
=== Softc
The name "softc" is a legacy term for "driver private data". The name most probably comes from the archaic term "software control block". In GEOM, it is a structure (more precise: pointer to a structure) that can be attached to a geom instance to hold whatever data is private to the geom instance. Most GEOM classes have the following members:
The name "softc" is a legacy term for "driver private data".
The name most probably comes from the archaic term "software control block".
In GEOM, it is a structure (more precise: pointer to a structure) that can be attached to a geom instance to hold whatever data is private to the geom instance.
Most GEOM classes have the following members:
* `struct g_provider *provider` : The "provider" this geom instantiates
* `uint16_t n_disks` : Number of consumer this geom consumes
* `struct g_consumer \**disks` : Array of `struct g_consumer*`. (It is not possible to use just single indirection because struct g_consumer* are created on our behalf by GEOM).
The `softc` structure contains all the state of geom instance. Every geom instance has its own softc.
The `softc` structure contains all the state of geom instance.
Every geom instance has its own softc.
[[geom-metadata]]
=== Metadata
@ -261,7 +307,8 @@ In the case of creating/labeling a new geom, this is what happens:
[[geom-command]]
=== GEOM Command Structure
The helper [.filename]#geom_CLASSNAME.so# library exports `class_commands` structure, which is an array of `struct g_command` elements. Commands are of uniform format and look like:
The helper [.filename]#geom_CLASSNAME.so# library exports `class_commands` structure, which is an array of `struct g_command` elements.
Commands are of uniform format and look like:
[.programlisting]
....
@ -278,12 +325,15 @@ Common options are:
* `-v` : be verbose
* `-f` : force
Many actions, such as labeling and destroying metadata can be performed in userland. For this, `struct g_command` provides field `gc_func` that can be set to a function (in the same [.filename]#.so#) that will be called to process a verb. If `gc_func` is NULL, the command will be passed to kernel module, to `.ctlreq` function of the geom class.
Many actions, such as labeling and destroying metadata can be performed in userland.
For this, `struct g_command` provides field `gc_func` that can be set to a function (in the same [.filename]#.so#) that will be called to process a verb.
If `gc_func` is NULL, the command will be passed to kernel module, to `.ctlreq` function of the geom class.
[[geom-geoms]]
=== Geoms
Geoms are instances of GEOM classes. They have internal data (a softc structure) and some functions with which they respond to external events.
Geoms are instances of GEOM classes.
They have internal data (a softc structure) and some functions with which they respond to external events.
The event functions are:
@ -317,7 +367,8 @@ When a user process issues "read data X at offset Y of a file" request, this is
See man:g_bio[9] man page for information how the data is passed back and forth in the `bio` structure (note in particular the `bio_parent` and `bio_children` fields and how they are handled).
One important feature is: __THERE CAN BE NO SLEEPING IN G_UP AND G_DOWN THREADS__. This means that none of the following things can be done in those threads (the list is of course not complete, but only informative):
One important feature is: __THERE CAN BE NO SLEEPING IN G_UP AND G_DOWN THREADS__.
This means that none of the following things can be done in those threads (the list is of course not complete, but only informative):
* Calls to `msleep`() and `tsleep`(), obviously.
* Calls to `g_write_data`() and `g_read_data`(), because these sleep between passing the data to consumers and returning.
@ -325,17 +376,27 @@ One important feature is: __THERE CAN BE NO SLEEPING IN G_UP AND G_DOWN THREADS_
* Calls to man:malloc[9] and `uma_zalloc`() with `M_WAITOK` flag set
* sx and other sleepable locks
This restriction is here to stop GEOM code clogging the I/O request path, since sleeping is usually not time-bound and there can be no guarantees on how long will it take (there are some other, more technical reasons also). It also means that there is not much that can be done in those threads; for example, almost any complex thing requires memory allocation. Fortunately, there is a way out: creating additional kernel threads.
This restriction is here to stop GEOM code clogging the I/O request path, since sleeping is usually not time-bound and there can be no guarantees on how long will it take (there are some other, more technical reasons also).
It also means that there is not much that can be done in those threads; for example, almost any complex thing requires memory allocation.
Fortunately, there is a way out: creating additional kernel threads.
[[geom-kernelthreads]]
=== Kernel Threads for Use in GEOM Code
Kernel threads are created with man:kthread_create[9] function, and they are sort of similar to userland threads in behavior, only they cannot return to caller to signify termination, but must call man:kthread_exit[9].
In GEOM code, the usual use of threads is to offload processing of requests from `g_down` thread (the `.start`() function). These threads look like "event handlers": they have a linked list of event associated with them (on which events can be posted by various functions in various threads so it must be protected by a mutex), take the events from the list one by one and process them in a big `switch`() statement.
In GEOM code, the usual use of threads is to offload processing of requests from `g_down` thread (the `.start`() function).
These threads look like "event handlers": they have a linked list of event associated with them (on which events can be posted by various functions in various threads so it must be protected by a mutex), take the events from the list one by one and process them in a big `switch`() statement.
The main benefit of using a thread to handle I/O requests is that it can sleep when needed. Now, this sounds good, but should be carefully thought out. Sleeping is well and very convenient but can very effectively destroy performance of the geom transformation. Extremely performance-sensitive classes probably should do all the work in `.start`() function call, taking great care to handle out-of-memory and similar errors.
The main benefit of using a thread to handle I/O requests is that it can sleep when needed.
Now, this sounds good, but should be carefully thought out.
Sleeping is well and very convenient but can very effectively destroy performance of the geom transformation.
Extremely performance-sensitive classes probably should do all the work in `.start`() function call, taking great care to handle out-of-memory and similar errors.
The other benefit of having a event-handler thread like that is to serialize all the requests and responses coming from different geom threads into one thread. This is also very convenient but can be slow. In most cases, handling of `.done`() requests can be left to the `g_up` thread.
The other benefit of having a event-handler thread like that is to serialize all the requests and responses coming from different geom threads into one thread.
This is also very convenient but can be slow.
In most cases, handling of `.done`() requests can be left to the `g_up` thread.
Mutexes in FreeBSD kernel (see man:mutex[9]) have one distinction from their more common userland cousins - the code cannot sleep while holding a mutex). If the code needs to sleep a lot, man:sx[9] locks may be more appropriate. On the other hand, if you do almost everything in a single thread, you may get away with no mutexes at all.
Mutexes in FreeBSD kernel (see man:mutex[9]) have one distinction from their more common userland cousins - the code cannot sleep while holding a mutex).
If the code needs to sleep a lot, man:sx[9] locks may be more appropriate.
On the other hand, if you do almost everything in a single thread, you may get away with no mutexes at all.

152
documentation/content/en/articles/gjournal-desktop/_index.adoc
View file

@ -22,7 +22,7 @@ include::shared/en/mailing-lists.adoc[]
include::shared/en/urls.adoc[]
ifeval::["{backend}" == "html5"]
:imagesdir: ../../images/articles/gjournal-desktop/
:imagesdir: ../../../images/articles/gjournal-desktop/
endif::[]
ifeval::["{backend}" == "pdf"]
@ -36,7 +36,11 @@ endif::[]
[.abstract-title]
Abstract
A journaling file system uses a log to record all transactions that take place in the file system, and preserves its integrity in the event of a system crash or power failure. Although it is still possible to lose unsaved changes to files, journaling almost completely eliminates the possibility of file system corruption caused by an unclean shutdown. It also shortens to a minimum the time required for after-failure file system checking. Although the UFS file system employed by FreeBSD does not implement journaling itself, the new journal class of the GEOM framework in FreeBSD 7._X_ can be used to provide file system independent journaling. This article explains how to implement UFS journaling on a typical desktop PC scenario.
A journaling file system uses a log to record all transactions that take place in the file system, and preserves its integrity in the event of a system crash or power failure.
Although it is still possible to lose unsaved changes to files, journaling almost completely eliminates the possibility of file system corruption caused by an unclean shutdown.
It also shortens to a minimum the time required for after-failure file system checking.
Although the UFS file system employed by FreeBSD does not implement journaling itself, the new journal class of the GEOM framework in FreeBSD 7._X_ can be used to provide file system independent journaling.
This article explains how to implement UFS journaling on a typical desktop PC scenario.
'''
@ -45,11 +49,15 @@ toc::[]
[[introduction]]
== Introduction
While professional servers are usually well protected from unforeseen shutdowns, the typical desktop is at the mercy of power failures, accidental resets, and other user related incidents that can lead to unclean shutdowns. Soft Updates usually protect the file system efficiently in such cases, although most of the times a lengthy background check is required. On rare occasions, file system corruption reaches a point where user intervention is required and data may be lost.
While professional servers are usually well protected from unforeseen shutdowns, the typical desktop is at the mercy of power failures, accidental resets, and other user related incidents that can lead to unclean shutdowns.
Soft Updates usually protect the file system efficiently in such cases, although most of the times a lengthy background check is required.
On rare occasions, file system corruption reaches a point where user intervention is required and data may be lost.
The new journaling capability provided by GEOM can greatly assist in such scenarios, by virtually eliminating the time required for file system checking, and ensuring that the file system is quickly restored to a consistent state.
This article describes a procedure for implementing UFS journaling on a typical desktop PC scenario (one hard disk used for both operating system and data). It should be followed during a fresh installation of FreeBSD. The steps are simple enough and do not require overly complex interaction with the command line.
This article describes a procedure for implementing UFS journaling on a typical desktop PC scenario (one hard disk used for both operating system and data).
It should be followed during a fresh installation of FreeBSD.
The steps are simple enough and do not require overly complex interaction with the command line.
After reading this article, you will know:
@ -66,28 +74,42 @@ Before reading this article, you should be able to:
[WARNING]
====
The procedure described here is intended for preparing a new installation where no actual user data is stored on the disk yet. While it is possible to modify and extend this procedure for systems already in production, you should _backup_ all important data before doing so. Messing around with disks and partitions at a low level can lead to fatal mistakes and data loss.
The procedure described here is intended for preparing a new installation where no actual user data is stored on the disk yet.
While it is possible to modify and extend this procedure for systems already in production, you should _backup_ all important data before doing so.
Messing around with disks and partitions at a low level can lead to fatal mistakes and data loss.
====
[[understanding-journaling]]
== Understanding Journaling in FreeBSD
The journaling provided by GEOM in FreeBSD 7._X_ is not file system specific (unlike for example the ext3 file system in Linux(R)) but is functioning at the block level. Though this means it can be applied to different file systems, for FreeBSD 7.0-RELEASE, it can only be used on UFS2.
The journaling provided by GEOM in FreeBSD 7._X_ is not file system specific (unlike for example the ext3 file system in Linux(R)) but is functioning at the block level.
Though this means it can be applied to different file systems, for FreeBSD 7.0-RELEASE, it can only be used on UFS2.
This functionality is provided by loading the [.filename]#geom_journal.ko# module into the kernel (or building it into a custom kernel) and using the `gjournal` command to configure the file systems. In general, you would like to journal large file systems, like [.filename]#/usr#. You will need however (see the following section) to reserve some free disk space.
This functionality is provided by loading the [.filename]#geom_journal.ko# module into the kernel (or building it into a custom kernel) and using the `gjournal` command to configure the file systems.
In general, you would like to journal large file systems, like [.filename]#/usr#.
You will need however (see the following section) to reserve some free disk space.
When a file system is journaled, some disk space is needed to keep the journal itself. The disk space that holds the actual data is referred to as the __data provider__, while the one that holds the journal is referred to as the __journal provider__. The data and journal providers need to be on different partitions when journaling an existing (non-empty) partition. When journaling a new partition, you have the option to use a single provider for both data and journal. In any case, the `gjournal` command combines both providers to create the final journaled file system. For example:
When a file system is journaled, some disk space is needed to keep the journal itself.
The disk space that holds the actual data is referred to as the __data provider__, while the one that holds the journal is referred to as the __journal provider__.
The data and journal providers need to be on different partitions when journaling an existing (non-empty) partition.
When journaling a new partition, you have the option to use a single provider for both data and journal.
In any case, the `gjournal` command combines both providers to create the final journaled file system.
For example:
* You wish to journal your [.filename]#/usr# file system, stored in [.filename]#/dev/ad0s1f# (which already contains data).
* You reserved some free disk space in a partition in [.filename]#/dev/ad0s1g#.
* Using `gjournal`, a new [.filename]#/dev/ad0s1f.journal# device is created where [.filename]#/dev/ad0s1f# is the data provider, and [.filename]#/dev/ad0s1g# is the journal provider. This new device is then used for all subsequent file operations.
The amount of disk space you need to reserve for the journal provider depends on the usage load of the file system and not on the size of the data provider. For example on a typical office desktop, a 1 GB journal provider for the [.filename]#/usr# file system will suffice, while a machine that deals with heavy disk I/O (i.e. video editing) may need more. A kernel panic will occur if the journal space is exhausted before it has a chance to be committed.
The amount of disk space you need to reserve for the journal provider depends on the usage load of the file system and not on the size of the data provider.
For example on a typical office desktop, a 1 GB journal provider for the [.filename]#/usr# file system will suffice, while a machine that deals with heavy disk I/O (i.e. video editing) may need more.
A kernel panic will occur if the journal space is exhausted before it has a chance to be committed.
[NOTE]
====
The journal sizes suggested here, are highly unlikely to cause problems in typical desktop use (such as web browsing, word processing and playback of media files). If your workload includes intense disk activity, use the following rule for maximum reliability: Your RAM size should fit in 30% of the journal provider's space. For example, if your system has 1 GB RAM, create an approximately 3.3 GB journal provider. (Multiply your RAM size with 3.3 to obtain the size of the journal).
The journal sizes suggested here, are highly unlikely to cause problems in typical desktop use (such as web browsing, word processing and playback of media files).
If your workload includes intense disk activity, use the following rule for maximum reliability: Your RAM size should fit in 30% of the journal provider's space.
For example, if your system has 1 GB RAM, create an approximately 3.3 GB journal provider.
(Multiply your RAM size with 3.3 to obtain the size of the journal).
====
For more information about journaling, please read the manual page of man:gjournal[8].
@ -97,34 +119,55 @@ For more information about journaling, please read the manual page of man:gjourn
=== Reserving Space for Journaling
A typical desktop machine usually has one hard disk that stores both the OS and user data. Arguably, the default partitioning scheme selected by sysinstall is more or less suitable: A desktop machine does not need a large [.filename]#/var# partition, while [.filename]#/usr# is allocated the bulk of the disk space, since user data and a lot of packages are installed into its subdirectories.
A typical desktop machine usually has one hard disk that stores both the OS and user data.
Arguably, the default partitioning scheme selected by sysinstall is more or less suitable: A desktop machine does not need a large [.filename]#/var# partition, while [.filename]#/usr# is allocated the bulk of the disk space, since user data and a lot of packages are installed into its subdirectories.
The default partitioning (the one obtained by pressing kbd:[A] at the FreeBSD partition editor, called Disklabel) does not leave any unallocated space. Each partition that will be journaled, requires another partition for the journal. Since the [.filename]#/usr# partition is the largest, it makes sense to shrink this partition slightly, to obtain the space required for journaling.
The default partitioning (the one obtained by pressing kbd:[A] at the FreeBSD partition editor, called Disklabel) does not leave any unallocated space.
Each partition that will be journaled, requires another partition for the journal.
Since the [.filename]#/usr# partition is the largest, it makes sense to shrink this partition slightly, to obtain the space required for journaling.
In our example, an 80 GB disk is used. The following screenshot shows the default partitions created by Disklabel during installation:
In our example, an 80 GB disk is used.
The following screenshot shows the default partitions created by Disklabel during installation:
image::disklabel1.png[]
If this is more or less what you need, it is very easy to adjust for journaling. Simply use the arrow keys to move the highlight to the [.filename]#/usr# partition and press kbd:[D] to delete it.
If this is more or less what you need, it is very easy to adjust for journaling.
Simply use the arrow keys to move the highlight to the [.filename]#/usr# partition and press kbd:[D] to delete it.
Now, move the highlight to the disk name at the top of the screen and press kbd:[C] to create a new partition for [.filename]#/usr#. This new partition should be smaller by 1 GB (if you intend to journal [.filename]#/usr# only), or 2 GB (if you intend to journal both [.filename]#/usr# and [.filename]#/var#). From the pop-up that appears, opt to create a file system, and type [.filename]#/usr# as the mount point.
Now, move the highlight to the disk name at the top of the screen and press kbd:[C] to create a new partition for [.filename]#/usr#.
This new partition should be smaller by 1 GB (if you intend to journal [.filename]#/usr# only), or 2 GB (if you intend to journal both [.filename]#/usr# and [.filename]#/var#).
From the pop-up that appears, opt to create a file system, and type [.filename]#/usr# as the mount point.
[NOTE]
====
Should you journal the [.filename]#/var# partition? Normally, journaling makes sense on quite large partitions. You may decide not to journal [.filename]#/var#, although doing so on a typical desktop will cause no harm. If the file system is lightly used (quite probable for a desktop) you may wish to allocate less disk space for its journal.
Should you journal the [.filename]#/var# partition? Normally, journaling makes sense on quite large partitions.
You may decide not to journal [.filename]#/var#, although doing so on a typical desktop will cause no harm.
If the file system is lightly used (quite probable for a desktop) you may wish to allocate less disk space for its journal.
In our example, we journal both [.filename]#/usr# and [.filename]#/var#. You may of course adjust the procedure to your own needs.
In our example, we journal both [.filename]#/usr# and [.filename]#/var#.
You may of course adjust the procedure to your own needs.
====
To keep things as easy going as possible, we are going to use sysinstall to create the partitions required for journaling. However, during installation, sysinstall insists on asking a mount point for each partition you create. At this point, you do not have any mount points for the partitions that will hold the journals, and in reality you __do not even need them__. These are not partitions that we are ever going to mount somewhere.
To keep things as easy going as possible, we are going to use sysinstall to create the partitions required for journaling.
However, during installation, sysinstall insists on asking a mount point for each partition you create.
At this point, you do not have any mount points for the partitions that will hold the journals, and in reality you __do not even need them__.
These are not partitions that we are ever going to mount somewhere.
To avoid these problems with sysinstall, we are going to create the journal partitions as swap space. Swap is never mounted, and sysinstall has no problem creating as many swap partitions as needed. After the first reboot, [.filename]#/etc/fstab# will have to be edited, and the extra swap space entries removed.
To avoid these problems with sysinstall, we are going to create the journal partitions as swap space.
Swap is never mounted, and sysinstall has no problem creating as many swap partitions as needed.
After the first reboot, [.filename]#/etc/fstab# will have to be edited, and the extra swap space entries removed.
To create the swap, again use the arrow keys to move the highlight to the top of Disklabel screen, so that the disk name itself is highlighted. Then press kbd:[N], enter the desired size (_1024M_), and select "swap space" from the pop-up menu that appears. Repeat for every journal you wish to create. In our example, we create two partitions to provide for the journals of [.filename]#/usr# and [.filename]#/var#. The final result is shown in the following screenshot:
To create the swap, again use the arrow keys to move the highlight to the top of Disklabel screen, so that the disk name itself is highlighted.
Then press kbd:[N], enter the desired size (_1024M_), and select "swap space" from the pop-up menu that appears.
Repeat for every journal you wish to create.
In our example, we create two partitions to provide for the journals of [.filename]#/usr# and [.filename]#/var#.
The final result is shown in the following screenshot:
image::disklabel2.png[]
When you have completed creating the partitions, we suggest you write down the partition names, and mount points, so you can easily refer to this information during the configuration phase. This will help alleviate mistakes that may damage your installation. The following table shows our notes for the sample configuration:
When you have completed creating the partitions, we suggest you write down the partition names, and mount points, so you can easily refer to this information during the configuration phase.
This will help alleviate mistakes that may damage your installation.
The following table shows our notes for the sample configuration:
.Partitions and Journals
[cols="1,1,1", options="header"]
@ -142,12 +185,15 @@ When you have completed creating the partitions, we suggest you write down the p
|ad0s1g
|===
Continue the installation as you would normally do. We would however suggest you postpone installation of third party software (packages) until you have completely setup journaling.
Continue the installation as you would normally do.
We would however suggest you postpone installation of third party software (packages) until you have completely setup journaling.
[[first-boot]]
=== Booting for the first time
Your system will come up normally, but you will need to edit [.filename]#/etc/fstab# and remove the extra swap partitions you created for the journals. Normally, the swap partition you will actually use is the one with the "b" suffix (i.e. ad0s1b in our example). Remove all other swap space entries and reboot so that FreeBSD will stop using them.
Your system will come up normally, but you will need to edit [.filename]#/etc/fstab# and remove the extra swap partitions you created for the journals.
Normally, the swap partition you will actually use is the one with the "b" suffix (i.e. ad0s1b in our example).
Remove all other swap space entries and reboot so that FreeBSD will stop using them.
When the system comes up again, we will be ready to configure journaling.
@ -157,14 +203,16 @@ When the system comes up again, we will be ready to configure journaling.
[[running-gjournal]]
=== Executing `gjournal`
Having prepared all the required partitions, it is quite easy to configure journaling. We will need to switch to single user mode, so login as `root` and type:
Having prepared all the required partitions, it is quite easy to configure journaling.
We will need to switch to single user mode, so login as `root` and type:
[source,shell]
....
# shutdown now
....
Press kbd:[Enter] to get the default shell. We will need to unmount the partitions that will be journaled, in our example [.filename]#/usr# and [.filename]#/var#:
Press kbd:[Enter] to get the default shell.
We will need to unmount the partitions that will be journaled, in our example [.filename]#/usr# and [.filename]#/var#:
[source,shell]
....
@ -178,7 +226,9 @@ Load the module required for journaling:
# gjournal load
....
Now, use your notes to determine which partition will be used for each journal. In our example, [.filename]#/usr# is [.filename]#ad0s1f# and its journal will be [.filename]#ad0s1g#, while [.filename]#/var# is [.filename]#ad0s1d# and will be journaled to [.filename]#ad0s1h#. The following commands are required:
Now, use your notes to determine which partition will be used for each journal.
In our example, [.filename]#/usr# is [.filename]#ad0s1f# and its journal will be [.filename]#ad0s1g#, while [.filename]#/var# is [.filename]#ad0s1d# and will be journaled to [.filename]#ad0s1h#.
The following commands are required:
[source,shell]
....
@ -193,7 +243,8 @@ GEOM_JOURNAL: Journal 3193218002: ad0s1h contains journal.
[NOTE]
====
If the last sector of either partition is used, `gjournal` will return an error. You will have to run the command using the `-f` flag to force an overwrite, i.e.:
If the last sector of either partition is used, `gjournal` will return an error.
You will have to run the command using the `-f` flag to force an overwrite, i.e.:
[source,shell]
....
@ -203,7 +254,9 @@ If the last sector of either partition is used, `gjournal` will return an error.
Since this is a new installation, it is highly unlikely that anything will be actually overwritten.
====
At this point, two new devices are created, namely [.filename]#ad0s1d.journal# and [.filename]#ad0s1f.journal#. These represent the [.filename]#/var# and [.filename]#/usr# partitions we have to mount. Before mounting, we must however set the journal flag on them and clear the Soft Updates flag:
At this point, two new devices are created, namely [.filename]#ad0s1d.journal# and [.filename]#ad0s1f.journal#.
These represent the [.filename]#/var# and [.filename]#/usr# partitions we have to mount.
Before mounting, we must however set the journal flag on them and clear the Soft Updates flag:
[source,shell]
....
@ -234,7 +287,6 @@ Edit [.filename]#/etc/fstab# and update the entries for [.filename]#/usr# and [.
[WARNING]
====
Make sure the above entries are correct, or you will have trouble starting up normally after you reboot!
====
@ -245,7 +297,9 @@ Finally, edit [.filename]#/boot/loader.conf# and add the following line so the m
geom_journal_load="YES"
....
Congratulations! Your system is now set for journaling. You can either type `exit` to return to multi-user mode, or reboot to test your configuration (recommended). During the boot you will see messages like the following:
Congratulations! Your system is now set for journaling.
You can either type `exit` to return to multi-user mode, or reboot to test your configuration (recommended).
During the boot you will see messages like the following:
[source,shell]
....
@ -270,14 +324,19 @@ This usually means that man:gjournal[8] used the information in the journal prov
[[gjournal-new]]
=== Journaling Newly Created Partitions
While the above procedure is necessary for journaling partitions that already contain data, journaling an empty partition is somewhat easier, since both the data and the journal provider can be stored in the same partition. For example, assume a new disk was installed, and a new partition [.filename]#/dev/ad1s1d# was created. Creating the journal would be as simple as:
While the above procedure is necessary for journaling partitions that already contain data, journaling an empty partition is somewhat easier, since both the data and the journal provider can be stored in the same partition.
For example, assume a new disk was installed, and a new partition [.filename]#/dev/ad1s1d# was created.
Creating the journal would be as simple as:
[source,shell]
....
# gjournal label ad1s1d
....
The journal size will be 1 GB by default. You may adjust it by using the `-s` option. The value can be given in bytes, or appended by `K`, `M` or `G` to denote Kilobytes, Megabytes or Gigabytes respectively. Note that `gjournal` will not allow you to create unsuitably small journal sizes.
The journal size will be 1 GB by default.
You may adjust it by using the `-s` option.
The value can be given in bytes, or appended by `K`, `M` or `G` to denote Kilobytes, Megabytes or Gigabytes respectively.
Note that `gjournal` will not allow you to create unsuitably small journal sizes.
For example, to create a 2 GB journal, you could use the following command:
@ -296,7 +355,8 @@ You can then create a file system on your new partition, and enable journaling u
[[configure-kernel]]
=== Building Journaling into Your Custom Kernel
If you do not wish to load `geom_journal` as a module, you can build its functions right into your kernel. Edit your custom kernel configuration file, and make sure it includes these two lines:
If you do not wish to load `geom_journal` as a module, you can build its functions right into your kernel.
Edit your custom kernel configuration file, and make sure it includes these two lines:
[.programlisting]
....
@ -316,18 +376,25 @@ The following section covers frequently asked questions regarding problems relat
=== I am getting kernel panics during periods of high disk activity. How is this related to journaling?
The journal probably fills up before it has a chance to get committed (flushed) to disk. Keep in mind the size of the journal depends on the usage load, and not the size of the data provider. If your disk activity is high, you need a larger partition for the journal. See the note in the <<understanding-journaling>> section.
The journal probably fills up before it has a chance to get committed (flushed) to disk.
Keep in mind the size of the journal depends on the usage load, and not the size of the data provider.
If your disk activity is high, you need a larger partition for the journal.
See the note in the <<understanding-journaling>> section.
=== I made some mistake during configuration, and I cannot boot normally now. Can this be fixed some way?
You either forgot (or misspelled) the entry in [.filename]#/boot/loader.conf#, or there are errors in your [.filename]#/etc/fstab# file. These are usually easy to fix. Press kbd:[Enter] to get to the default single user shell. Then locate the root of the problem:
You either forgot (or misspelled) the entry in [.filename]#/boot/loader.conf#, or there are errors in your [.filename]#/etc/fstab# file.
These are usually easy to fix.
Press kbd:[Enter] to get to the default single user shell.
Then locate the root of the problem:
[source,shell]
....
# cat /boot/loader.conf
....
If the `geom_journal_load` entry is missing or misspelled, the journaled devices are never created. Load the module manually, mount all partitions, and continue with multi-user boot:
If the `geom_journal_load` entry is missing or misspelled, the journaled devices are never created.
Load the module manually, mount all partitions, and continue with multi-user boot:
[source,shell]
....
@ -345,11 +412,15 @@ GEOM_JOURNAL: Journal ad0s1f clean.
(boot continues)
....
If, on the other hand, this entry is correct, have a look at [.filename]#/etc/fstab#. You will probably find a misspelled or missing entry. In this case, mount all remaining partitions by hand and continue with the multi-user boot.
If, on the other hand, this entry is correct, have a look at [.filename]#/etc/fstab#.
You will probably find a misspelled or missing entry.
In this case, mount all remaining partitions by hand and continue with the multi-user boot.
=== Can I remove journaling and return to my standard file system with Soft Updates?
Sure. Use the following procedure, which reverses the changes. The partitions you created for the journal providers can then be used for other purposes, if you so wish.
Sure.
Use the following procedure, which reverses the changes.
The partitions you created for the journal providers can then be used for other purposes, if you so wish.
Login as `root` and switch to single user mode:
@ -424,7 +495,8 @@ Finally, edit [.filename]#/boot/loader.conf#, remove the entry that loads the `g
[[further-reading]]
== Further Reading
Journaling is a fairly new feature of FreeBSD, and as such, it is not very well documented yet. You may however find the following additional references useful:
Journaling is a fairly new feature of FreeBSD, and as such, it is not very well documented yet.
You may however find the following additional references useful:
* A link:{handbook}#geom-gjournal[new section on journaling] is now part of the FreeBSD Handbook.
* https://lists.freebsd.org/pipermail/freebsd-current/2006-June/064043.html[This post] in {freebsd-current} by man:gjournal[8]'s developer, `{pjd}`.

150
documentation/content/en/articles/hubs/_index.adoc
View file

@ -44,7 +44,8 @@ We are not accepting new mirrors at this time.
[[mirror-contact]]
== Contact Information
The Mirror System Coordinators can be reached through email at mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org]. There is also a {freebsd-hubs}.
The Mirror System Coordinators can be reached through email at mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org].
There is also a {freebsd-hubs}.
[[mirror-requirements]]
== Requirements for FreeBSD Mirrors
@ -52,7 +53,14 @@ The Mirror System Coordinators can be reached through email at mailto:mirror-adm
[[mirror-diskspace]]
=== Disk Space
Disk space is one of the most important requirements. Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind that _official_ mirrors are probably required to be complete. The web pages should always be mirrored completely. Also note that the numbers stated here are reflecting the current state (at {rel120-current}-RELEASE/{rel113-current}-RELEASE). Further development and releases will only increase the required amount. Also make sure to keep some (ca. 10-20%) extra space around just to be sure. Here are some approximate figures:
Disk space is one of the most important requirements.
Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed.
Also keep in mind that _official_ mirrors are probably required to be complete.
The web pages should always be mirrored completely.
Also note that the numbers stated here are reflecting the current state (at {rel120-current}-RELEASE/{rel113-current}-RELEASE).
Further development and releases will only increase the required amount.
Also make sure to keep some (ca. 10-20%) extra space around just to be sure.
Here are some approximate figures:
* Full FTP Distribution: 1.4 TB
* CTM deltas: 10 GB
@ -63,7 +71,11 @@ The current disk usage of FTP Distribution can be found at link:ftp://ftp.FreeBS
[[mirror-bandwidth]]
=== Network Connection/Bandwidth
Of course, you need to be connected to the Internet. The required bandwidth depends on your intended use of the mirror. If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller than if you want to make the files publicly available. If you intend to become an official mirror, the bandwidth required will be even higher. We can only give rough estimates here:
Of course, you need to be connected to the Internet.
The required bandwidth depends on your intended use of the mirror.
If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller than if you want to make the files publicly available.
If you intend to become an official mirror, the bandwidth required will be even higher.
We can only give rough estimates here:
* Local site, no public access: basically no minimum, but < 2 Mbps could make syncing too slow.
* Unofficial public site: 34 Mbps is probably a good start.
@ -72,23 +84,36 @@ Of course, you need to be connected to the Internet. The required bandwidth depe
[[mirror-system]]
=== System Requirements, CPU, RAM
One thing this depends on the expected number of clients, which is determined by the server's policy. It is also affected by the types of services you want to offer. Plain FTP or HTTP services may not require a huge amount of resources. Watch out if you provide rsync. This can have a huge impact on CPU and memory requirements as it is considered a memory hog. The following are just examples to give you a very rough hint.
One thing this depends on the expected number of clients, which is determined by the server's policy.
It is also affected by the types of services you want to offer.
Plain FTP or HTTP services may not require a huge amount of resources.
Watch out if you provide rsync.
This can have a huge impact on CPU and memory requirements as it is considered a memory hog.
The following are just examples to give you a very rough hint.
For a moderately visited site that offers rsync, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an _official_ site.
For a moderately visited site that offers rsync, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM.
This is probably the minimum you want for an _official_ site.
For a frequently used site you definitely need more RAM (consider 2GB as a good start) and possibly more CPU, which could also mean that you need to go for a SMP system.
You also want to consider a fast disk subsystem. Operations on the SVN repository require a fast disk subsystem (RAID is highly advised). A SCSI controller that has a cache of its own can also speed up things since most of these services incur a large number of small modifications to the disk.
You also want to consider a fast disk subsystem.
Operations on the SVN repository require a fast disk subsystem (RAID is highly advised).
A SCSI controller that has a cache of its own can also speed up things since most of these services incur a large number of small modifications to the disk.
[[mirror-services]]
=== Services to Offer
Every mirror site is required to have a set of core services available. In addition to these required services, there are a number of optional services that server administrators may choose to offer. This section explains which services you can provide and how to go about implementing them.
Every mirror site is required to have a set of core services available.
In addition to these required services, there are a number of optional services that server administrators may choose to offer.
This section explains which services you can provide and how to go about implementing them.
[[mirror-serv-ftp]]
==== FTP (required for FTP Fileset)
This is one of the most basic services, and it is required for each mirror offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and _must_ never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path [.filename]#/pub/FreeBSD#.
This is one of the most basic services, and it is required for each mirror offering public FTP distributions.
FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway).
Upload capability is not required (and _must_ never be allowed for the FreeBSD file space).
Also the FreeBSD archive should be available under the path [.filename]#/pub/FreeBSD#.
There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order).
@ -100,19 +125,31 @@ There is a lot of software available which can be set up to allow anonymous FTP
* package:ftp/twoftpd[]: As above.
* package:ftp/vsftpd[]: The "very secure" ftpd.
FreeBSD's `ftpd`, `proftpd` and maybe `ncftpd` are among the most commonly used FTPds. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed.
FreeBSD's `ftpd`, `proftpd` and maybe `ncftpd` are among the most commonly used FTPds.
The others do not have a large userbase among mirror sites.
One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed.
[[mirror-serv-rsync]]
==== Rsync (optional for FTP Fileset)
Rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The protocol is different from FTP in many ways. It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change. Rsync does require a significant amount of memory for each instance. The size depends on the size of the synced module in terms of the number of directories and files. Rsync can use `rsh` and `ssh` (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available:
Rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source.
The protocol is different from FTP in many ways.
It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change.
Rsync does require a significant amount of memory for each instance.
The size depends on the size of the synced module in terms of the number of directories and files.
Rsync can use `rsh` and `ssh` (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers).
Authentication, connection limits, and other restrictions may be applied.
There is just one software package available:
* package:net/rsync[]
[[mirror-serv-http]]
==== HTTP (required for Web Pages, Optional for FTP Fileset)
If you want to offer the FreeBSD web pages, you will need to install a web server. You may optionally offer the FTP fileset via HTTP. The choice of web server software is left up to the mirror administrator. Some of the most popular choices are:
If you want to offer the FreeBSD web pages, you will need to install a web server.
You may optionally offer the FTP fileset via HTTP.
The choice of web server software is left up to the mirror administrator.
Some of the most popular choices are:
* package:www/apache24[]: Apache is still one of the most widely deployed web servers on the Internet. It is used extensively by the FreeBSD Project.
* package:www/boa[]: Boa is a single-tasking HTTP server. Unlike traditional web servers, it does not fork for each incoming connection, nor does it fork many copies of itself to handle multiple connections. Although, it should provide considerably great performance for purely static content.
@ -124,18 +161,26 @@ If you want to offer the FreeBSD web pages, you will need to install a web serve
[[mirror-howto]]
== How to Mirror FreeBSD
Ok, now you know the requirements and how to offer the services, but not how to get it. :-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from.
Ok, now you know the requirements and how to offer the services, but not how to get it.
:-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from.
[[mirror-ftp-rsync]]
=== Mirroring the FTP Site
The FTP area is the largest amount of data that needs to be mirrored. It includes the _distribution sets_ required for network installation, the _branches_ which are actually snapshots of checked-out source trees, the _ISO Images_ to write CD-ROMs with the installation distribution, a live file system, and a snapshot of the ports tree. All of course for various FreeBSD versions, and various architectures.
The FTP area is the largest amount of data that needs to be mirrored.
It includes the _distribution sets_ required for network installation, the _branches_ which are actually snapshots of checked-out source trees, the _ISO Images_ to write CD-ROMs with the installation distribution, a live file system, and a snapshot of the ports tree.
All of course for various FreeBSD versions, and various architectures.
The best way to mirror the FTP area is rsync. You can install the port package:net/rsync[] and then use rsync to sync with your upstream host. rsync is already mentioned in <<mirror-serv-rsync>>. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access.
The best way to mirror the FTP area is rsync.
You can install the port package:net/rsync[] and then use rsync to sync with your upstream host.
rsync is already mentioned in <<mirror-serv-rsync>>.
Since rsync access is not required, your preferred upstream site may not allow it.
You may need to hunt around a little bit to find a site that allows rsync access.
[NOTE]
====
Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror).
Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server.
For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror).
====
A command line to mirror FreeBSD might look like:
@ -145,7 +190,8 @@ A command line to mirror FreeBSD might look like:
% rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/
....
Consult the documentation for rsync, which is also available at http://rsync.samba.org/[http://rsync.samba.org/], about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via man:cron[8].
Consult the documentation for rsync, which is also available at http://rsync.samba.org/[http://rsync.samba.org/], about the various options to be used with rsync.
If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via man:cron[8].
[[mirror-www]]
=== Mirroring the WWW Pages
@ -162,7 +208,9 @@ A command line to mirror the FreeBSD web site might look like:
[[mirror-pkgs]]
=== Mirroring Packages
Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the man:pkg[8] process. Alternatively specific packages and their dependencies can be fetched by running something like the following:
Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages.
For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the man:pkg[8] process.
Alternatively specific packages and their dependencies can be fetched by running something like the following:
[source,shell]
....
@ -176,12 +224,16 @@ Once those packages have been fetched, the repository metadata must be generated
% pkg repo /usr/local/mirror
....
Once the packages have been fetched and the metadata for the repository has been generated, serve the packages up to the client machines via HTTP. For additional information see the man pages for man:pkg[8], specifically the man:pkg-repo[8] page.
Once the packages have been fetched and the metadata for the repository has been generated, serve the packages up to the client machines via HTTP.
For additional information see the man pages for man:pkg[8], specifically the man:pkg-repo[8] page.
[[mirror-how-often]]
=== How Often Should I Mirror?
Every mirror should be updated at a minimum of once per day. Certainly a script with locking to prevent multiple runs happening at the same time will be needed to run from man:cron[8]. Since nearly every admin does this in their own way, specific instructions cannot be provided. It could work something like this:
Every mirror should be updated at a minimum of once per day.
Certainly a script with locking to prevent multiple runs happening at the same time will be needed to run from man:cron[8].
Since nearly every admin does this in their own way, specific instructions cannot be provided.
It could work something like this:
[.procedure]
====
@ -199,24 +251,42 @@ Here are some recommended schedules:
[[mirror-where]]
== Where to Mirror From
This is an important issue. So this section will spend some effort to explain the backgrounds. We will say this several times: under no circumstances should you mirror from `ftp.FreeBSD.org`.
This is an important issue.
So this section will spend some effort to explain the backgrounds.
We will say this several times: under no circumstances should you mirror from `ftp.FreeBSD.org`.
[[mirror-where-organization]]
=== A few Words About the Organization
Mirrors are organized by country. All official mirrors have a DNS entry of the form `ftpN.CC.FreeBSD.org`. _CC_ (i.e., country code) is the _top level domain_ (TLD) of the country where this mirror is located. _N_ is a number, telling that the host would be the _Nth_ mirror in that country. (Same applies to `wwwN.CC.FreeBSD.org`, etc.) There are mirrors with no _CC_ part. These are the mirror sites that are very well connected and allow a large number of concurrent users. `ftp.FreeBSD.org` is actually two machines, one currently located in Denmark and the other in the United States. It is _NOT_ a master site and should never be used to mirror from. Lots of online documentation leads "interactive"users to `ftp.FreeBSD.org` so automated mirroring systems should find a different machine to mirror from.
Mirrors are organized by country.
All official mirrors have a DNS entry of the form `ftpN.CC.FreeBSD.org`.
_CC_ (i.e., country code) is the _top level domain_ (TLD) of the country where this mirror is located.
_N_ is a number, telling that the host would be the _Nth_ mirror in that country.
(Same applies to `wwwN.CC.FreeBSD.org`, etc.) There are mirrors with no _CC_ part.
These are the mirror sites that are very well connected and allow a large number of concurrent users.
`ftp.FreeBSD.org` is actually two machines, one currently located in Denmark and the other in the United States.
It is _NOT_ a master site and should never be used to mirror from.
Lots of online documentation leads "interactive"users to `ftp.FreeBSD.org` so automated mirroring systems should find a different machine to mirror from.
Additionally there exists a hierarchy of mirrors, which is described in terms of __tiers__. The master sites are not referred to but can be described as __Tier-0__. Mirrors that mirror from these sites can be considered __Tier-1__, mirrors of __Tier-1__-mirrors, are __Tier-2__, etc. Official sites are encouraged to be of a low __tier__, but the lower the tier the higher the requirements in terms as described in <<mirror-requirements>>. Also access to low-tier-mirrors may be restricted, and access to master sites is definitely restricted. The __tier__-hierarchy is not reflected by DNS and generally not documented anywhere except for the master sites. However, official mirrors with low numbers like 1-4, are usually _Tier-1_ (this is just a rough hint, and there is no rule).
Additionally there exists a hierarchy of mirrors, which is described in terms of __tiers__.
The master sites are not referred to but can be described as __Tier-0__.
Mirrors that mirror from these sites can be considered __Tier-1__, mirrors of __Tier-1__-mirrors, are __Tier-2__, etc.
Official sites are encouraged to be of a low __tier__, but the lower the tier the higher the requirements in terms as described in <<mirror-requirements>>.
Also access to low-tier-mirrors may be restricted, and access to master sites is definitely restricted.
The __tier__-hierarchy is not reflected by DNS and generally not documented anywhere except for the master sites.
However, official mirrors with low numbers like 1-4, are usually _Tier-1_ (this is just a rough hint, and there is no rule).
[[mirror-where-where]]
=== Ok, but Where Should I get the Stuff Now?
Under no circumstances should you mirror from `ftp.FreeBSD.org`. The short answer is: from the site that is closest to you in Internet terms, or gives you the fastest access.
Under no circumstances should you mirror from `ftp.FreeBSD.org`.
The short answer is: from the site that is closest to you in Internet terms, or gives you the fastest access.
[[mirror-where-simple]]
==== I Just Want to Mirror from Somewhere!
If you have no special intentions or requirements, the statement in <<mirror-where-where>> applies. This means:
If you have no special intentions or requirements, the statement in <<mirror-where-where>> applies.
This means:
[.procedure]
====
@ -228,14 +298,21 @@ If you have no special intentions or requirements, the statement in <<mirror-whe
[[mirror-where-official]]
==== I am an Official Mirror, What is the Right Rite for Me?
In general the description in <<mirror-where-simple>> still applies. Of course you may want to put some weight on the fact that your upstream should be of a low tier. There are some other considerations about _official_ mirrors that are described in <<mirror-official>>.
In general the description in <<mirror-where-simple>> still applies.
Of course you may want to put some weight on the fact that your upstream should be of a low tier.
There are some other considerations about _official_ mirrors that are described in <<mirror-official>>.
[[mirror-where-master]]
==== I Want to Access the Master Sites!
If you have good reasons and good prerequisites, you may want and get access to one of the master sites. Access to these sites is generally restricted, and there are special policies for access. If you are already an _official_ mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. If it already has three or more, ask the "zone administrator" (mailto:hostmaster@CC.FreeBSD.org[hostmaster@CC.FreeBSD.org]) or {freebsd-hubs} first.
If you have good reasons and good prerequisites, you may want and get access to one of the master sites.
Access to these sites is generally restricted, and there are special policies for access.
If you are already an _official_ mirror, this certainly helps you getting access.
In any other case make sure your country really needs another mirror.
If it already has three or more, ask the "zone administrator" (mailto:hostmaster@CC.FreeBSD.org[hostmaster@CC.FreeBSD.org]) or {freebsd-hubs} first.
Whoever helped you become, an _official_ should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site. If not, you can send email to mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org] to request help with that.
Whoever helped you become, an _official_ should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site.
If not, you can send email to mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org] to request help with that.
There is one master site for the FTP fileset.
@ -244,7 +321,8 @@ There is one master site for the FTP fileset.
This is the master site for the FTP fileset.
`ftp-master.FreeBSD.org` provides rsync access, in addition to FTP. Refer to <<mirror-ftp-rsync>>.
`ftp-master.FreeBSD.org` provides rsync access, in addition to FTP.
Refer to <<mirror-ftp-rsync>>.
Mirrors are also encouraged to allow rsync access for the FTP contents, since they are __Tier-1__-mirrors.
@ -256,12 +334,15 @@ Official mirrors are mirrors that
* a) have a `FreeBSD.org` DNS entry (usually a CNAME).
* b) are listed as an official mirror in the FreeBSD documentation (like handbook).
So far to distinguish official mirrors. Official mirrors are not necessarily __Tier-1__-mirrors. However you probably will not find a __Tier-1__-mirror, that is not also official.
So far to distinguish official mirrors. Official mirrors are not necessarily __Tier-1__-mirrors.
However you probably will not find a __Tier-1__-mirror, that is not also official.
[[mirror-official-requirements]]
=== Special Requirements for Official (tier-1) Mirrors
It is not so easy to state requirements for all official mirrors, since the project is sort of tolerant here. It is more easy to say, what _official tier-1 mirrors_ are required to. All other official mirrors can consider this a big __should__.
It is not so easy to state requirements for all official mirrors, since the project is sort of tolerant here.
It is more easy to say, what _official tier-1 mirrors_ are required to.
All other official mirrors can consider this a big __should__.
Tier-1 mirrors are required to:
@ -269,13 +350,16 @@ Tier-1 mirrors are required to:
* allow access to other mirror sites
* provide FTP and rsync access
Furthermore, admins should be subscribed to the {freebsd-hubs}. See link:{handbook}#eresources-mail[this link] for details, how to subscribe.
Furthermore, admins should be subscribed to the {freebsd-hubs}.
See link:{handbook}#eresources-mail[this link] for details, how to subscribe.
[IMPORTANT]
====
It is _very_ important for a hub administrator, especially Tier-1 hub admins, to check the https://www.FreeBSD.org/releng/[release schedule] for the next FreeBSD release. This is important because it will tell you when the next release is scheduled to come out, and thus giving you time to prepare for the big spike of traffic which follows it.
It is _very_ important for a hub administrator, especially Tier-1 hub admins, to check the https://www.FreeBSD.org/releng/[release schedule] for the next FreeBSD release.
This is important because it will tell you when the next release is scheduled to come out, and thus giving you time to prepare for the big spike of traffic which follows it.
It is also important that hub administrators try to keep their mirrors as up-to-date as possible (again, even more crucial for Tier-1 mirrors). If Mirror1 does not update for a while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins a downward spiral... Keep your mirrors up to date!
It is also important that hub administrators try to keep their mirrors as up-to-date as possible (again, even more crucial for Tier-1 mirrors).
If Mirror1 does not update for a while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins a downward spiral... Keep your mirrors up to date!
====
[[mirror-official-become]]

37
documentation/content/en/articles/ipsec-must/_index.adoc
View file

@ -22,7 +22,8 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
You installed IPsec and it seems to be working. How do you know? I describe a method for experimentally verifying that IPsec is working.
You installed IPsec and it seems to be working.
How do you know? I describe a method for experimentally verifying that IPsec is working.
'''
@ -31,7 +32,9 @@ toc::[]
[[problem]]
== The Problem
First, lets assume you have <<ipsec-install>>. How do you know it is <<caveat>>? Sure, your connection will not work if it is misconfigured, and it will work when you finally get it right. man:netstat[1] will list it. But can you independently confirm it?
First, lets assume you have <<ipsec-install>>.
How do you know it is <<caveat>>? Sure, your connection will not work if it is misconfigured, and it will work when you finally get it right.
man:netstat[1] will list it. But can you independently confirm it?
[[solution]]
== The Solution
@ -41,17 +44,22 @@ First, some crypto-relevant info theory:
. Encrypted data is uniformly distributed, i.e., has maximal entropy per symbol;
. Raw, uncompressed data is typically redundant, i.e., has sub-maximal entropy.
Suppose you could measure the entropy of the data to- and from- your network interface. Then you could see the difference between unencrypted data and encrypted data. This would be true even if some of the data in "encrypted mode" was not encrypted---as the outermost IP header must be if the packet is to be routable.
Suppose you could measure the entropy of the data to- and from- your network interface.
Then you could see the difference between unencrypted data and encrypted data.
This would be true even if some of the data in "encrypted mode" was not encrypted---as the outermost IP header must be if the packet is to be routable.
[[MUST]]
=== MUST
Ueli Maurer's "Universal Statistical Test for Random Bit Generators"(https://web.archive.org/web/20011115002319/http://www.geocities.com/SiliconValley/Code/4704/universal.pdf[MUST]) quickly measures the entropy of a sample. It uses a compression-like algorithm. <<code>> for a variant which measures successive (~quarter megabyte) chunks of a file.
Ueli Maurer's "Universal Statistical Test for Random Bit Generators"(https://web.archive.org/web/20011115002319/http://www.geocities.com/SiliconValley/Code/4704/universal.pdf[MUST]) quickly measures the entropy of a sample.
It uses a compression-like algorithm.
<<code>> for a variant which measures successive (~quarter megabyte) chunks of a file.
[[tcpdump]]
=== Tcpdump
We also need a way to capture the raw network data. A program called man:tcpdump[1] lets you do this, if you have enabled the _Berkeley Packet Filter_ interface in your <<kernel>>.
We also need a way to capture the raw network data.
A program called man:tcpdump[1] lets you do this, if you have enabled the _Berkeley Packet Filter_ interface in your <<kernel>>.
The command:
@ -60,7 +68,8 @@ The command:
tcpdump -c 4000 -s 10000 -w dumpfile.bin
....
will capture 4000 raw packets to _dumpfile.bin_. Up to 10,000 bytes per packet will be captured in this example.
will capture 4000 raw packets to _dumpfile.bin_.
Up to 10,000 bytes per packet will be captured in this example.
[[experiment]]
== The Experiment
@ -95,24 +104,32 @@ Expected value for L=8 is 7.1836656
[[caveat]]
== Caveat
This experiment shows that IPsec _does_ seem to be distributing the payload data __uniformly__, as encryption should. However, the experiment described here _cannot_ detect many possible flaws in a system (none of which do I have any evidence for). These include poor key generation or exchange, data or keys being visible to others, use of weak algorithms, kernel subversion, etc. Study the source; know the code.
This experiment shows that IPsec _does_ seem to be distributing the payload data __uniformly__, as encryption should.
However, the experiment described here _cannot_ detect many possible flaws in a system (none of which do I have any evidence for).
These include poor key generation or exchange, data or keys being visible to others, use of weak algorithms, kernel subversion, etc.
Study the source; know the code.
[[IPsec]]
== IPsec---Definition
Internet Protocol security extensions to IPv4; required for IPv6. A protocol for negotiating encryption and authentication at the IP (host-to-host) level. SSL secures only one application socket; SSH secures only a login; PGP secures only a specified file or message. IPsec encrypts everything between two hosts.
Internet Protocol security extensions to IPv4; required for IPv6.
A protocol for negotiating encryption and authentication at the IP (host-to-host) level.
SSL secures only one application socket; SSH secures only a login; PGP secures only a specified file or message.
IPsec encrypts everything between two hosts.
[[ipsec-install]]
== Installing IPsec
Most of the modern versions of FreeBSD have IPsec support in their base source. So you will need to include the `IPSEC` option in your kernel config and, after kernel rebuild and reinstall, configure IPsec connections using man:setkey[8] command.
Most of the modern versions of FreeBSD have IPsec support in their base source.
So you will need to include the `IPSEC` option in your kernel config and, after kernel rebuild and reinstall, configure IPsec connections using man:setkey[8] command.
A comprehensive guide on running IPsec on FreeBSD is provided in link:{handbook}#ipsec[FreeBSD Handbook].
[[kernel]]
== src/sys/i386/conf/KERNELNAME
This needs to be present in the kernel config file in order to capture network data with man:tcpdump[1]. Be sure to run man:config[8] after adding this, and rebuild and reinstall.
This needs to be present in the kernel config file in order to capture network data with man:tcpdump[1].
Be sure to run man:config[8] after adding this, and rebuild and reinstall.
[.programlisting]
....

231
documentation/content/en/articles/ldap-auth/_index.adoc
View file

@ -21,7 +21,8 @@ trademarks: ["freebsd", "general"]
[.abstract-title]
Abstract
This document is intended as a guide for the configuration of an LDAP server (principally an OpenLDAP server) for authentication on FreeBSD. This is useful for situations where many servers need the same user accounts, for example as a replacement for NIS.
This document is intended as a guide for the configuration of an LDAP server (principally an OpenLDAP server) for authentication on FreeBSD.
This is useful for situations where many servers need the same user accounts, for example as a replacement for NIS.
'''
@ -30,27 +31,36 @@ toc::[]
[[preface]]
== Preface
This document is intended to give the reader enough of an understanding of LDAP to configure an LDAP server. This document will attempt to provide an explanation of package:net/nss_ldap[] and package:security/pam_ldap[] for use with client machines services for use with the LDAP server.
This document is intended to give the reader enough of an understanding of LDAP to configure an LDAP server.
This document will attempt to provide an explanation of package:net/nss_ldap[] and package:security/pam_ldap[] for use with client machines services for use with the LDAP server.
When finished, the reader should be able to configure and deploy a FreeBSD server that can host an LDAP directory, and to configure and deploy a FreeBSD server which can authenticate against an LDAP directory.
This article is not intended to be an exhaustive account of the security, robustness, or best practice considerations for configuring LDAP or the other services discussed herein. While the author takes care to do everything correctly, they do not address security issues beyond a general scope. This article should be considered to lay the theoretical groundwork only, and any actual implementation should be accompanied by careful requirement analysis.
This article is not intended to be an exhaustive account of the security, robustness, or best practice considerations for configuring LDAP or the other services discussed herein.
While the author takes care to do everything correctly, they do not address security issues beyond a general scope.
This article should be considered to lay the theoretical groundwork only, and any actual implementation should be accompanied by careful requirement analysis.
[[ldap]]
== Configuring LDAP
LDAP stands for "Lightweight Directory Access Protocol" and is a subset of the X.500 Directory Access Protocol. Its most recent specifications are in http://www.ietf.org/rfc/rfc4510.txt[RFC4510] and friends. Essentially it is a database that expects to be read from more often than it is written to.
LDAP stands for "Lightweight Directory Access Protocol" and is a subset of the X.500 Directory Access Protocol.
Its most recent specifications are in http://www.ietf.org/rfc/rfc4510.txt[RFC4510] and friends.
Essentially it is a database that expects to be read from more often than it is written to.
The LDAP server http://www.openldap.org/[OpenLDAP] will be used in the examples in this document; while the principles here should be generally applicable to many different servers, most of the concrete administration is OpenLDAP-specific. There are several server versions in ports, for example package:net/openldap24-server[]. Client servers will need the corresponding package:net/openldap24-client[] libraries.
The LDAP server http://www.openldap.org/[OpenLDAP] will be used in the examples in this document; while the principles here should be generally applicable to many different servers, most of the concrete administration is OpenLDAP-specific.
There are several server versions in ports, for example package:net/openldap24-server[].
Client servers will need the corresponding package:net/openldap24-client[] libraries.
There are (basically) two areas of the LDAP service which need configuration. The first is setting up a server to receive connections properly, and the second is adding entries to the server's directory so that FreeBSD tools know how to interact with it.
There are (basically) two areas of the LDAP service which need configuration.
The first is setting up a server to receive connections properly, and the second is adding entries to the server's directory so that FreeBSD tools know how to interact with it.
[[ldap-connect]]
=== Setting Up the Server for Connections
[NOTE]
====
This section is specific to OpenLDAP. If you are using another server, you will need to consult that server's documentation.
This section is specific to OpenLDAP.
If you are using another server, you will need to consult that server's documentation.
====
[[ldap-connect-install]]
@ -78,13 +88,18 @@ This installs the `slapd` and `slurpd` binaries, along with the required OpenLDA
Next we must configure OpenLDAP.
You will want to require encryption in your connections to the LDAP server; otherwise your users' passwords will be transferred in plain text, which is considered insecure. The tools we will be using support two very similar kinds of encryption, SSL and TLS.
You will want to require encryption in your connections to the LDAP server; otherwise your users' passwords will be transferred in plain text, which is considered insecure.
The tools we will be using support two very similar kinds of encryption, SSL and TLS.
TLS stands for "Transportation Layer Security". Services that employ TLS tend to connect on the _same_ ports as the same services without TLS; thus an SMTP server which supports TLS will listen for connections on port 25, and an LDAP server will listen on 389.
TLS stands for "Transportation Layer Security".
Services that employ TLS tend to connect on the _same_ ports as the same services without TLS; thus an SMTP server which supports TLS will listen for connections on port 25, and an LDAP server will listen on 389.
SSL stands for "Secure Sockets Layer", and services that implement SSL do _not_ listen on the same ports as their non-SSL counterparts. Thus SMTPS listens on port 465 (not 25), HTTPS listens on 443, and LDAPS on 636.
SSL stands for "Secure Sockets Layer", and services that implement SSL do _not_ listen on the same ports as their non-SSL counterparts.
Thus SMTPS listens on port 465 (not 25), HTTPS listens on 443, and LDAPS on 636.
The reason SSL uses a different port than TLS is because a TLS connection begins as plain text, and switches to encrypted traffic after the `STARTTLS` directive. SSL connections are encrypted from the beginning. Other than that there are no substantial differences between the two.
The reason SSL uses a different port than TLS is because a TLS connection begins as plain text, and switches to encrypted traffic after the `STARTTLS` directive.
SSL connections are encrypted from the beginning.
Other than that there are no substantial differences between the two.
[NOTE]
====
@ -102,9 +117,11 @@ TLSCertificateKeyFile /path/to/your/cert.key
TLSCACertificateFile /path/to/your/cacert.crt
....
Here, `ssf=128` tells OpenLDAP to require 128-bit encryption for all connections, both search and update. This parameter may be configured based on the security needs of your site, but rarely you need to weaken it, as most LDAP client libraries support strong encryption.
Here, `ssf=128` tells OpenLDAP to require 128-bit encryption for all connections, both search and update.
This parameter may be configured based on the security needs of your site, but rarely you need to weaken it, as most LDAP client libraries support strong encryption.
The [.filename]#cert.crt#, [.filename]#cert.key#, and [.filename]#cacert.crt# files are necessary for clients to authenticate _you_ as the valid LDAP server. If you simply want a server that runs, you can create a self-signed certificate with OpenSSL:
The [.filename]#cert.crt#, [.filename]#cert.key#, and [.filename]#cacert.crt# files are necessary for clients to authenticate _you_ as the valid LDAP server.
If you simply want a server that runs, you can create a self-signed certificate with OpenSSL:
[[genrsa]]
.Generating an RSA Key
@ -124,7 +141,11 @@ e is 65537 (0x10001)
====
At this point you should be prompted for some values. You may enter whatever values you like; however, it is important the "Common Name" value be the fully qualified domain name of the OpenLDAP server. In our case, and the examples here, the server is _server.example.org_. Incorrectly setting this value will cause clients to fail when making connections. This can the cause of great frustration, so ensure that you follow these steps closely.
At this point you should be prompted for some values.
You may enter whatever values you like; however, it is important the "Common Name" value be the fully qualified domain name of the OpenLDAP server.
In our case, and the examples here, the server is _server.example.org_.
Incorrectly setting this value will cause clients to fail when making connections.
This can the cause of great frustration, so ensure that you follow these steps closely.
Finally, the certificate signing request needs to be signed:
@ -143,7 +164,8 @@ Getting Private key
====
This will create a self-signed certificate that can be used for the directives in [.filename]#slapd.conf#, where [.filename]#cert.crt# and [.filename]#cacert.crt# are the same file. If you are going to use many OpenLDAP servers (for replication via `slurpd`) you will want to see <<ssl-ca>> to generate a CA key and use it to sign individual server certificates.
This will create a self-signed certificate that can be used for the directives in [.filename]#slapd.conf#, where [.filename]#cert.crt# and [.filename]#cacert.crt# are the same file.
If you are going to use many OpenLDAP servers (for replication via `slurpd`) you will want to see <<ssl-ca>> to generate a CA key and use it to sign individual server certificates.
Once this is done, put the following in [.filename]#/etc/rc.conf#:
@ -152,7 +174,9 @@ Once this is done, put the following in [.filename]#/etc/rc.conf#:
slapd_enable="YES"
....
Then run `/usr/local/etc/rc.d/slapd start`. This should start OpenLDAP. Confirm that it is listening on 389 with
Then run `/usr/local/etc/rc.d/slapd start`.
This should start OpenLDAP.
Confirm that it is listening on 389 with
[source,shell]
....
@ -163,9 +187,11 @@ ldap slapd 3261 7 tcp4 *:389 *:*
[[ldap-connect-client]]
==== Configuring the Client
Install the package:net/openldap24-client[] port for the OpenLDAP libraries. The client machines will always have OpenLDAP libraries since that is all package:security/pam_ldap[] and package:net/nss_ldap[] support, at least for the moment.
Install the package:net/openldap24-client[] port for the OpenLDAP libraries.
The client machines will always have OpenLDAP libraries since that is all package:security/pam_ldap[] and package:net/nss_ldap[] support, at least for the moment.
The configuration file for the OpenLDAP libraries is [.filename]#/usr/local/etc/openldap/ldap.conf#. Edit this file to contain the following values:
The configuration file for the OpenLDAP libraries is [.filename]#/usr/local/etc/openldap/ldap.conf#.
Edit this file to contain the following values:
[.programlisting]
....
@ -182,19 +208,27 @@ It is important that your clients have access to [.filename]#cacert.crt#, otherw
[NOTE]
====
There are two files called [.filename]#ldap.conf#. The first is this file, which is for the OpenLDAP libraries and defines how to talk to the server. The second is [.filename]#/usr/local/etc/ldap.conf#, and is for pam_ldap.
There are two files called [.filename]#ldap.conf#.
The first is this file, which is for the OpenLDAP libraries and defines how to talk to the server.
The second is [.filename]#/usr/local/etc/ldap.conf#, and is for pam_ldap.
====
At this point you should be able to run `ldapsearch -Z` on the client machine; `-Z` means "use TLS". If you encounter an error, then something is configured wrong; most likely it is your certificates. Use man:openssl[1]'s `s_client` and `s_server` to ensure you have them configured and signed properly.
At this point you should be able to run `ldapsearch -Z` on the client machine; `-Z` means "use TLS".
If you encounter an error, then something is configured wrong; most likely it is your certificates.
Use man:openssl[1]'s `s_client` and `s_server` to ensure you have them configured and signed properly.
[[ldap-database]]
=== Entries in the Database
Authentication against an LDAP directory is generally accomplished by attempting to bind to the directory as the connecting user. This is done by establishing a "simple" bind on the directory with the user name supplied. If there is an entry with the `uid` equal to the user name and that entry's `userPassword` attribute matches the password supplied, then the bind is successful.
Authentication against an LDAP directory is generally accomplished by attempting to bind to the directory as the connecting user.
This is done by establishing a "simple" bind on the directory with the user name supplied.
If there is an entry with the `uid` equal to the user name and that entry's `userPassword` attribute matches the password supplied, then the bind is successful.
The first thing we have to do is figure out is where in the directory our users will live.
The base entry for our database is `dc=example,dc=org`. The default location for users that most clients seem to expect is something like `ou=people,_base_`, so that is what will be used here. However keep in mind that this is configurable.
The base entry for our database is `dc=example,dc=org`.
The default location for users that most clients seem to expect is something like `ou=people,_base_`, so that is what will be used here.
However keep in mind that this is configurable.
So the ldif entry for the `people` organizational unit will look like:
@ -208,9 +242,13 @@ ou: people
All users will be created as subentries of this organizational unit.
Some thought might be given to the object class your users will belong to. Most tools by default will use `people`, which is fine if you simply want to provide entries against which to authenticate. However, if you are going to store user information in the LDAP database as well, you will probably want to use `inetOrgPerson`, which has many useful attributes. In either case, the relevant schemas need to be loaded in [.filename]#slapd.conf#.
Some thought might be given to the object class your users will belong to.
Most tools by default will use `people`, which is fine if you simply want to provide entries against which to authenticate.
However, if you are going to store user information in the LDAP database as well, you will probably want to use `inetOrgPerson`, which has many useful attributes.
In either case, the relevant schemas need to be loaded in [.filename]#slapd.conf#.
For this example we will use the `person` object class. If you are using `inetOrgPerson`, the steps are basically identical, except that the `sn` attribute is required.
For this example we will use the `person` object class.
If you are using `inetOrgPerson`, the steps are basically identical, except that the `sn` attribute is required.
To add a user `testuser`, the ldif would be:
@ -231,7 +269,8 @@ cn: tuser
I start my LDAP users' UIDs at 10000 to avoid collisions with system accounts; you can configure whatever number you wish here, as long as it is less than 65536.
We also need group entries. They are as configurable as user entries, but we will use the defaults below:
We also need group entries.
They are as configurable as user entries, but we will use the defaults below:
[.programlisting]
....
@ -247,9 +286,11 @@ gidNumber: 10000
cn: tuser
....
To enter these into your database, you can use `slapadd` or `ldapadd` on a file containing these entries. Alternatively, you can use package:sysutils/ldapvi[].
To enter these into your database, you can use `slapadd` or `ldapadd` on a file containing these entries.
Alternatively, you can use package:sysutils/ldapvi[].
The `ldapsearch` utility on the client machine should now return these entries. If it does, your database is properly configured to be used as an LDAP authentication server.
The `ldapsearch` utility on the client machine should now return these entries.
If it does, your database is properly configured to be used as an LDAP authentication server.
[[client]]
== Client Configuration
@ -265,12 +306,15 @@ package:security/pam_ldap[] is configured via [.filename]#/usr/local/etc/ldap.co
[NOTE]
====
This is a _different file_ than the OpenLDAP library functions' configuration file, [.filename]#/usr/local/etc/openldap/ldap.conf#; however, it takes many of the same options; in fact it is a superset of that file. For the rest of this section, references to [.filename]#ldap.conf# will mean [.filename]#/usr/local/etc/ldap.conf#.
This is a _different file_ than the OpenLDAP library functions' configuration file, [.filename]#/usr/local/etc/openldap/ldap.conf#; however, it takes many of the same options; in fact it is a superset of that file.
For the rest of this section, references to [.filename]#ldap.conf# will mean [.filename]#/usr/local/etc/ldap.conf#.
====
Thus, we will want to copy all of our original configuration parameters from [.filename]#openldap/ldap.conf# to the new [.filename]#ldap.conf#. Once this is done, we want to tell package:security/pam_ldap[] what to look for on the directory server.
Thus, we will want to copy all of our original configuration parameters from [.filename]#openldap/ldap.conf# to the new [.filename]#ldap.conf#.
Once this is done, we want to tell package:security/pam_ldap[] what to look for on the directory server.
We are identifying our users with the `uid` attribute. To configure this (though it is the default), set the `pam_login_attribute` directive in [.filename]#ldap.conf#:
We are identifying our users with the `uid` attribute.
To configure this (though it is the default), set the `pam_login_attribute` directive in [.filename]#ldap.conf#:
[[set-pam-login-attr]]
.Setting `pam_login_attribute`
@ -284,28 +328,39 @@ pam_login_attribute uid
====
With this set, package:security/pam_ldap[] will search the entire LDAP directory under `base` for the value `uid=_username_`. If it finds one and only one entry, it will attempt to bind as that user with the password it was given. If it binds correctly, then it will allow access. Otherwise it will fail.
With this set, package:security/pam_ldap[] will search the entire LDAP directory under `base` for the value `uid=_username_`.
If it finds one and only one entry, it will attempt to bind as that user with the password it was given.
If it binds correctly, then it will allow access.
Otherwise it will fail.
Users whose shell is not in [.filename]#/etc/shells# will not be able to log in. This is particularly important when Bash is set as the user shell on the LDAP server. Bash is not included with a default installation of FreeBSD. When installed from a package or port, it is located at [.filename]#/usr/local/bin/bash#. Verify that the path to the shell on the server is set correctly:
Users whose shell is not in [.filename]#/etc/shells# will not be able to log in.
This is particularly important when Bash is set as the user shell on the LDAP server.
Bash is not included with a default installation of FreeBSD.
When installed from a package or port, it is located at [.filename]#/usr/local/bin/bash#.
Verify that the path to the shell on the server is set correctly:
[source,shell]
....
% getent passwd username
....
There are two choices when the output shows `/bin/bash` in the last column. The first is to change the user's entry on the LDAP server to [.filename]#/usr/local/bin/bash#. The second option is to create a symlink on the LDAP client computer so Bash is found at the correct location:
There are two choices when the output shows `/bin/bash` in the last column.
The first is to change the user's entry on the LDAP server to [.filename]#/usr/local/bin/bash#.
The second option is to create a symlink on the LDAP client computer so Bash is found at the correct location:
[source,shell]
....
# ln -s /usr/local/bin/bash /bin/bash
....
Make sure that [.filename]#/etc/shells# contains entries for both `/usr/local/bin/bash` and `/bin/bash`. The user will then be able to log in to the system with Bash as their shell.
Make sure that [.filename]#/etc/shells# contains entries for both `/usr/local/bin/bash` and `/bin/bash`.
The user will then be able to log in to the system with Bash as their shell.
[[client-auth-pam]]
==== PAM
PAM, which stands for "Pluggable Authentication Modules", is the method by which FreeBSD authenticates most of its sessions. To tell FreeBSD we wish to use an LDAP server, we will have to add a line to the appropriate PAM file.
PAM, which stands for "Pluggable Authentication Modules", is the method by which FreeBSD authenticates most of its sessions.
To tell FreeBSD we wish to use an LDAP server, we will have to add a line to the appropriate PAM file.
Most of the time the appropriate PAM file is [.filename]#/etc/pam.d/sshd#, if you want to use SSH (remember to set the relevant options in [.filename]#/etc/ssh/sshd_config#, otherwise SSH will not use PAM).
@ -318,27 +373,36 @@ auth sufficient /usr/local/lib/pam_ldap.so no_warn
Exactly where this line shows up in the file and which options appear in the fourth column determine the exact behavior of the authentication mechanism; see man:pam[d]
With this configuration you should be able to authenticate a user against an LDAP directory. PAM will perform a bind with your credentials, and if successful will tell SSH to allow access.
With this configuration you should be able to authenticate a user against an LDAP directory.
PAM will perform a bind with your credentials, and if successful will tell SSH to allow access.
However it is not a good idea to allow _every_ user in the directory into _every_ client machine. With the current configuration, all that a user needs to log into a machine is an LDAP entry. Fortunately there are a few ways to restrict user access.
However it is not a good idea to allow _every_ user in the directory into _every_ client machine.
With the current configuration, all that a user needs to log into a machine is an LDAP entry.
Fortunately there are a few ways to restrict user access.
[.filename]#ldap.conf# supports a `pam_groupdn` directive; every account that connects to this machine needs to be a member of the group specified here. For example, if you have
[.filename]#ldap.conf# supports a `pam_groupdn` directive; every account that connects to this machine needs to be a member of the group specified here.
For example, if you have
[.programlisting]
....
pam_groupdn cn=servername,ou=accessgroups,dc=example,dc=org
....
in [.filename]#ldap.conf#, then only members of that group will be able to log in. There are a few things to bear in mind, however.
in [.filename]#ldap.conf#, then only members of that group will be able to log in.
There are a few things to bear in mind, however.
Members of this group are specified in one or more `memberUid` attributes, and each attribute must have the full distinguished name of the member. So `memberUid: someuser` will not work; it must be:
Members of this group are specified in one or more `memberUid` attributes, and each attribute must have the full distinguished name of the member.
So `memberUid: someuser` will not work; it must be:
[.programlisting]
....
memberUid: uid=someuser,ou=people,dc=example,dc=org
....
Additionally, this directive is not checked in PAM during authentication, it is checked during account management, so you will need a second line in your PAM files under `account`. This will require, in turn, _every_ user to be listed in the group, which is not necessarily what we want. To avoid blocking users that are not in LDAP, you should enable the `ignore_unknown_user` attribute. Finally, you should set the `ignore_authinfo_unavail` option so that you are not locked out of every computer when the LDAP server is unavailable.
Additionally, this directive is not checked in PAM during authentication, it is checked during account management, so you will need a second line in your PAM files under `account`.
This will require, in turn, _every_ user to be listed in the group, which is not necessarily what we want.
To avoid blocking users that are not in LDAP, you should enable the `ignore_unknown_user` attribute.
Finally, you should set the `ignore_authinfo_unavail` option so that you are not locked out of every computer when the LDAP server is unavailable.
Your [.filename]#pam.d/sshd# might then end up looking like this:
@ -363,17 +427,23 @@ account required /usr/local/lib/pam_ldap.so no_warn ignore_a
[NOTE]
====
Since we are adding these lines specifically to [.filename]#pam.d/sshd#, this will only have an effect on SSH sessions. LDAP users will be unable to log in at the console. To change this behavior, examine the other files in [.filename]#/etc/pam.d# and modify them accordingly.
Since we are adding these lines specifically to [.filename]#pam.d/sshd#, this will only have an effect on SSH sessions.
LDAP users will be unable to log in at the console.
To change this behavior, examine the other files in [.filename]#/etc/pam.d# and modify them accordingly.
====
[[client-nss]]
=== Name Service Switch
NSS is the service that maps attributes to names. So, for example, if a file is owned by user `1001`, an application will query NSS for the name of `1001`, and it might get `bob` or `ted` or whatever the user's name is.
NSS is the service that maps attributes to names.
So, for example, if a file is owned by user `1001`, an application will query NSS for the name of `1001`, and it might get `bob` or `ted` or whatever the user's name is.
Now that our user information is kept in LDAP, we need to tell NSS to look there when queried.
The package:net/nss_ldap[] port does this. It uses the same configuration file as package:security/pam_ldap[], and should not need any extra parameters once it is installed. Instead, what is left is simply to edit [.filename]#/etc/nsswitch.conf# to take advantage of the directory. Simply replace the following lines:
The package:net/nss_ldap[] port does this.
It uses the same configuration file as package:security/pam_ldap[], and should not need any extra parameters once it is installed.
Instead, what is left is simply to edit [.filename]#/etc/nsswitch.conf# to take advantage of the directory.
Simply replace the following lines:
[.programlisting]
....
@ -396,7 +466,10 @@ Congratulations! You should now have working LDAP authentication.
[[caveats]]
=== Caveats
Unfortunately, as of the time this was written FreeBSD did not support changing user passwords with man:passwd[1]. As a result of this, most administrators are left to implement a solution themselves. I provide some examples here. Note that if you write your own password change script, there are some security issues you should be made aware of; see <<security-passwd>>
Unfortunately, as of the time this was written FreeBSD did not support changing user passwords with man:passwd[1].
As a result of this, most administrators are left to implement a solution themselves.
I provide some examples here.
Note that if you write your own password change script, there are some security issues you should be made aware of; see <<security-passwd>>
[[chpw-shell]]
.Shell Script for Changing Passwords
@ -429,7 +502,8 @@ ldappasswd -D uid="$USER",ou=people,dc=example,dc=org \
[CAUTION]
====
This script does hardly any error checking, but more important it is very cavalier about how it stores your passwords. If you do anything like this, at least adjust the `security.bsd.see_other_uids` sysctl value:
This script does hardly any error checking, but more important it is very cavalier about how it stores your passwords.
If you do anything like this, at least adjust the `security.bsd.see_other_uids` sysctl value:
[source,shell]
....
@ -438,7 +512,9 @@ This script does hardly any error checking, but more important it is very cavali
====
A more flexible (and probably more secure) approach can be used by writing a custom program, or even a web interface. The following is part of a Ruby library that can change LDAP passwords. It sees use both on the command line, and on the web.
A more flexible (and probably more secure) approach can be used by writing a custom program, or even a web interface.
The following is part of a Ruby library that can change LDAP passwords.
It sees use both on the command line, and on the web.
[[chpw-ruby]]
.Ruby Script for Changing Passwords
@ -494,14 +570,18 @@ Although not guaranteed to be free of security holes (the password is kept in me
Now that your machines (and possibly other services) are authenticating against your LDAP server, this server needs to be protected at least as well as [.filename]#/etc/master.passwd# would be on a regular server, and possibly even more so since a broken or cracked LDAP server would break every client service.
Remember, this section is not exhaustive. You should continually review your configuration and procedures for improvements.
Remember, this section is not exhaustive.
You should continually review your configuration and procedures for improvements.
[[secure-readonly]]
=== Setting Attributes Read-only
Several attributes in LDAP should be read-only. If left writable by the user, for example, a user could change his `uidNumber` attribute to `0` and get `root` access!
Several attributes in LDAP should be read-only.
If left writable by the user, for example, a user could change his `uidNumber` attribute to `0` and get `root` access!
To begin with, the `userPassword` attribute should not be world-readable. By default, anyone who can connect to the LDAP server can read this attribute. To disable this, put the following in [.filename]#slapd.conf#:
To begin with, the `userPassword` attribute should not be world-readable.
By default, anyone who can connect to the LDAP server can read this attribute.
To disable this, put the following in [.filename]#slapd.conf#:
[[hide-userpass]]
.Hide Passwords
@ -525,7 +605,9 @@ access to *
This will disallow reading of the `userPassword` attribute, while still allowing users to change their own passwords.
Additionally, you'll want to keep users from changing some of their own attributes. By default, users can change any attribute (except for those which the LDAP schemas themselves deny changes), such as `uidNumber`. To close this hole, modify the above to
Additionally, you'll want to keep users from changing some of their own attributes.
By default, users can change any attribute (except for those which the LDAP schemas themselves deny changes), such as `uidNumber`.
To close this hole, modify the above to
[[attrib-readonly]]
.Read-only Attributes
@ -555,14 +637,19 @@ This will stop users from being able to masquerade as other users.
[[secure-root]]
=== `root` Account Definition
Often the `root` or manager account for the LDAP service will be defined in the configuration file. OpenLDAP supports this, for example, and it works, but it can lead to trouble if [.filename]#slapd.conf# is compromised. It may be better to use this only to bootstrap yourself into LDAP, and then define a `root` account there.
Often the `root` or manager account for the LDAP service will be defined in the configuration file.
OpenLDAP supports this, for example, and it works, but it can lead to trouble if [.filename]#slapd.conf# is compromised.
It may be better to use this only to bootstrap yourself into LDAP, and then define a `root` account there.
Even better is to define accounts that have limited permissions, and omit a `root` account entirely. For example, users that can add or remove user accounts are added to one group, but they cannot themselves change the membership of this group. Such a security policy would help mitigate the effects of a leaked password.
Even better is to define accounts that have limited permissions, and omit a `root` account entirely.
For example, users that can add or remove user accounts are added to one group, but they cannot themselves change the membership of this group.
Such a security policy would help mitigate the effects of a leaked password.
[[manager-acct]]
==== Creating a Management Group
Say you want your IT department to be able to change home directories for users, but you do not want all of them to be able to add or remove users. The way to do this is to add a group for these admins:
Say you want your IT department to be able to change home directories for users, but you do not want all of them to be able to add or remove users.
The way to do this is to add a group for these admins:
[[manager-acct-dn]]
.Creating a Management Group
@ -601,14 +688,18 @@ access to dn.subtree="ou=people,dc=example,dc=org"
Now `tuser` and `user2` can change other users' home directories.
In this example we have given a subset of administrative power to certain users without giving them power in other domains. The idea is that soon no single user account has the power of a `root` account, but every power root had is had by at least one user. The `root` account then becomes unnecessary and can be removed.
In this example we have given a subset of administrative power to certain users without giving them power in other domains.
The idea is that soon no single user account has the power of a `root` account, but every power root had is had by at least one user.
The `root` account then becomes unnecessary and can be removed.
[[security-passwd]]
=== Password Storage
By default OpenLDAP will store the value of the `userPassword` attribute as it stores any other data: in the clear. Most of the time it is base 64 encoded, which provides enough protection to keep an honest administrator from knowing your password, but little else.
By default OpenLDAP will store the value of the `userPassword` attribute as it stores any other data: in the clear.
Most of the time it is base 64 encoded, which provides enough protection to keep an honest administrator from knowing your password, but little else.
It is a good idea, then, to store passwords in a more secure format, such as SSHA (salted SHA). This is done by whatever program you use to change users' passwords.
It is a good idea, then, to store passwords in a more secure format, such as SSHA (salted SHA).
This is done by whatever program you use to change users' passwords.
:sectnums!:
@ -618,13 +709,19 @@ It is a good idea, then, to store passwords in a more secure format, such as SSH
There are a few other programs that might be useful, particularly if you have many users and do not want to configure everything manually.
package:security/pam_mkhomedir[] is a PAM module that always succeeds; its purpose is to create home directories for users which do not have them. If you have dozens of client servers and hundreds of users, it is much easier to use this and set up skeleton directories than to prepare every home directory.
package:security/pam_mkhomedir[] is a PAM module that always succeeds; its purpose is to create home directories for users which do not have them.
If you have dozens of client servers and hundreds of users, it is much easier to use this and set up skeleton directories than to prepare every home directory.
package:sysutils/cpu[] is a man:pw[8]-like utility that can be used to manage users in the LDAP directory. You can call it directly, or wrap scripts around it. It can handle both TLS (with the `-x` flag) and SSL (directly).
package:sysutils/cpu[] is a man:pw[8]-like utility that can be used to manage users in the LDAP directory.
You can call it directly, or wrap scripts around it.
It can handle both TLS (with the `-x` flag) and SSL (directly).
package:sysutils/ldapvi[] is a great utility for editing LDAP values in an LDIF-like syntax. The directory (or subsection of the directory) is presented in the editor chosen by the `EDITOR` environment variable. This makes it easy to enable large-scale changes in the directory without having to write a custom tool.
package:sysutils/ldapvi[] is a great utility for editing LDAP values in an LDIF-like syntax.
The directory (or subsection of the directory) is presented in the editor chosen by the `EDITOR` environment variable.
This makes it easy to enable large-scale changes in the directory without having to write a custom tool.
package:security/openssh-portable[] has the ability to contact an LDAP server to verify SSH keys. This is extremely nice if you have many servers and do not want to copy your public keys across all of them.
package:security/openssh-portable[] has the ability to contact an LDAP server to verify SSH keys.
This is extremely nice if you have many servers and do not want to copy your public keys across all of them.
:sectnums!:
@ -632,11 +729,13 @@ package:security/openssh-portable[] has the ability to contact an LDAP server to
[[ssl-ca]]
== OpenSSL Certificates for LDAP
If you are hosting two or more LDAP servers, you will probably not want to use self-signed certificates, since each client will have to be configured to work with each certificate. While this is possible, it is not nearly as simple as creating your own certificate authority, and signing your servers' certificates with that.
If you are hosting two or more LDAP servers, you will probably not want to use self-signed certificates, since each client will have to be configured to work with each certificate.
While this is possible, it is not nearly as simple as creating your own certificate authority, and signing your servers' certificates with that.
The steps here are presented as they are with very little attempt at explaining what is going on-further explanation can be found in man:openssl[1] and its friends.
To create a certificate authority, we simply need a self-signed certificate and key. The steps for this again are
To create a certificate authority, we simply need a self-signed certificate and key.
The steps for this again are
[[make-cert]]
.Creating a Certificate
@ -652,9 +751,11 @@ To create a certificate authority, we simply need a self-signed certificate and
====
These will be your root CA key and certificate. You will probably want to encrypt the key and store it in a cool, dry place; anyone with access to it can masquerade as one of your LDAP servers.
These will be your root CA key and certificate.
You will probably want to encrypt the key and store it in a cool, dry place; anyone with access to it can masquerade as one of your LDAP servers.
Next, using the first two steps above create a key [.filename]#ldap-server-one.key# and certificate signing request [.filename]#ldap-server-one.csr#. Once you sign the signing request with [.filename]#root.key#, you will be able to use [.filename]#ldap-server-one.*# on your LDAP servers.
Next, using the first two steps above create a key [.filename]#ldap-server-one.key# and certificate signing request [.filename]#ldap-server-one.csr#.
Once you sign the signing request with [.filename]#root.key#, you will be able to use [.filename]#ldap-server-one.*# on your LDAP servers.
[NOTE]
====

30
documentation/content/en/articles/leap-seconds/_index.adoc
View file

@ -22,18 +22,23 @@ toc::[]
[[leapseconds-definition]]
== Introduction
A _leap second_ is an one second adjustment made at specific times of year to UTC to synchronize atomic time scales with variations in the rotation of the Earth. This article describes how FreeBSD interacts with leap seconds.
A _leap second_ is an one second adjustment made at specific times of year to UTC to synchronize atomic time scales with variations in the rotation of the Earth.
This article describes how FreeBSD interacts with leap seconds.
As of this writing, the next leap second will occur at 2015-Jun-30 23:59:60 UTC. This leap second will occur during a business day for North and South America and the Asia/Pacific region.
As of this writing, the next leap second will occur at 2015-Jun-30 23:59:60 UTC.
This leap second will occur during a business day for North and South America and the Asia/Pacific region.
Leap seconds are announced by http://datacenter.iers.org/[IERS] on http://datacenter.iers.org/web/guest/bulletins/-/somos/5Rgv/product/16[Bulletin C].
Standard leap second behavior is described in https://tools.ietf.org/html/rfc7164#section-3[RFC 7164]. Also see man:time2posix[3].
Standard leap second behavior is described in https://tools.ietf.org/html/rfc7164#section-3[RFC 7164].
Also see man:time2posix[3].
[[leapseconds-posix]]
== Default Leap Second Handling on FreeBSD
The easiest way to handle leap seconds is with the POSIX time rules FreeBSD uses by default, combined with link:{handbook}#network-ntp[NTP]. When man:ntpd[8] is running and the time is synchronized with upstream NTP servers that handle leap seconds correctly, the leap second will cause the system time to automatically repeat the last second of the day. No other adjustments are necessary.
The easiest way to handle leap seconds is with the POSIX time rules FreeBSD uses by default, combined with link:{handbook}#network-ntp[NTP].
When man:ntpd[8] is running and the time is synchronized with upstream NTP servers that handle leap seconds correctly, the leap second will cause the system time to automatically repeat the last second of the day.
No other adjustments are necessary.
If the upstream NTP servers do not handle leap seconds correctly, man:ntpd[8] will step the time by one second after the errant upstream server has noticed and stepped itself.
@ -42,11 +47,13 @@ If NTP is not being used, manual adjustment of the system clock will be required
[[leapseconds-cautions]]
== Cautions
Leap seconds are inserted at the same instant all over the world: UTC midnight. In Japan that is mid-morning, in the Pacific mid-day, in the Americas late afternoon, and in Europe at night.
Leap seconds are inserted at the same instant all over the world: UTC midnight.
In Japan that is mid-morning, in the Pacific mid-day, in the Americas late afternoon, and in Europe at night.
We believe and expect that FreeBSD, if provided correct and stable NTP service, will work as designed during this leap second, as it did during the previous ones.
However, we caution that practically no applications have ever asked the kernel about leap seconds. Our experience is that, as designed, leap seconds are essentially a replay of the second before the leap second, and this is a surprise to most application programmers.
However, we caution that practically no applications have ever asked the kernel about leap seconds.
Our experience is that, as designed, leap seconds are essentially a replay of the second before the leap second, and this is a surprise to most application programmers.
Other operating systems and other computers may or may not handle the leap-second the same way as FreeBSD, and systems without correct and stable NTP service will not know anything about leap seconds at all.
@ -57,16 +64,21 @@ Please try to make sure nothing horrible happens because of the leap second.
[[leapseconds-testing]]
== Testing
It is possible to test whether a leap second will be used. Due to the nature of NTP, the test might work up to 24 hours before the leap second. Some major reference clock sources only announce leap seconds one hour ahead of the event. Query the NTP daemon:
It is possible to test whether a leap second will be used.
Due to the nature of NTP, the test might work up to 24 hours before the leap second.
Some major reference clock sources only announce leap seconds one hour ahead of the event.
Query the NTP daemon:
[source,shell]
....
% ntpq -c 'rv 0 leap'
....
Output that includes `leap_add_sec` indicates proper support of the leap second. Before the 24 hours leading up to the leap second, or after the leap second has passed, `leap_none` will be shown.
Output that includes `leap_add_sec` indicates proper support of the leap second.
Before the 24 hours leading up to the leap second, or after the leap second has passed, `leap_none` will be shown.
[[leapseconds-conclusion]]
== Conclusion
In practice, leap seconds are usually not a problem on FreeBSD. We hope that this overview helps clarify what to expect and how to make the leap second event proceed more smoothly.
In practice, leap seconds are usually not a problem on FreeBSD.
We hope that this overview helps clarify what to expect and how to make the leap second event proceed more smoothly.

729
documentation/content/en/articles/linux-emulation/_index.adoc
View file

File diff suppressed because it is too large Load diff

87
documentation/content/en/articles/linux-users/_index.adoc
View file

@ -33,21 +33,30 @@ toc::[]
This document highlights some of the technical differences between FreeBSD and Linux(R) so that intermediate to advanced Linux(R) users can quickly familiarize themselves with the basics of FreeBSD.
This document assumes that FreeBSD is already installed. Refer to the link:{handbook}#bsdinstall[Installing FreeBSD] chapter of the FreeBSD Handbook for help with the installation process.
This document assumes that FreeBSD is already installed.
Refer to the link:{handbook}#bsdinstall[Installing FreeBSD] chapter of the FreeBSD Handbook for help with the installation process.
[[shells]]
== Default Shell
Linux(R) users are often surprised to find that Bash is not the default shell in FreeBSD. In fact, Bash is not included in the default installation. Instead, FreeBSD uses man:tcsh[1] as the default root shell, and the Bourne shell-compatible man:sh[1] as the default user shell. man:sh[1] is very similar to Bash but with a much smaller feature-set. Generally shell scripts written for man:sh[1] will run in Bash, but the reverse is not always true.
Linux(R) users are often surprised to find that Bash is not the default shell in FreeBSD.
In fact, Bash is not included in the default installation.
Instead, FreeBSD uses man:tcsh[1] as the default root shell, and the Bourne shell-compatible man:sh[1] as the default user shell.
man:sh[1] is very similar to Bash but with a much smaller feature-set.
Generally shell scripts written for man:sh[1] will run in Bash, but the reverse is not always true.
However, Bash and other shells are available for installation using the FreeBSD link:{handbook}#ports[Packages and Ports Collection].
After installing another shell, use man:chsh[1] to change a user's default shell. It is recommended that the `root` user's default shell remain unchanged since shells which are not included in the base distribution are installed to [.filename]#/usr/local/bin#. In the event of a problem, the file system where [.filename]#/usr/local/bin# is located may not be mounted. In this case, `root` would not have access to its default shell, preventing `root` from logging in and fixing the problem.
After installing another shell, use man:chsh[1] to change a user's default shell.
It is recommended that the `root` user's default shell remain unchanged since shells which are not included in the base distribution are installed to [.filename]#/usr/local/bin#.
In the event of a problem, the file system where [.filename]#/usr/local/bin# is located may not be mounted.
In this case, `root` would not have access to its default shell, preventing `root` from logging in and fixing the problem.
[[software]]
== Packages and Ports: Adding Software in FreeBSD
FreeBSD provides two methods for installing applications: binary packages and compiled ports. Each method has its own benefits:
FreeBSD provides two methods for installing applications: binary packages and compiled ports.
Each method has its own benefits:
.Binary Packages
* Faster installation as compared to compiling large applications.
@ -58,14 +67,18 @@ FreeBSD provides two methods for installing applications: binary packages and co
* Ability to customize installation options.
* Custom patches can be applied.
If an application installation does not require any customization, installing the package is sufficient. Compile the port instead whenever an application requires customization of the default options. If needed, a custom package can be compiled from ports using `make package`.
If an application installation does not require any customization, installing the package is sufficient.
Compile the port instead whenever an application requires customization of the default options.
If needed, a custom package can be compiled from ports using `make package`.
A complete list of all available ports and packages can be found https://www.freebsd.org/ports/[here].
[[packages]]
=== Packages
Packages are pre-compiled applications, the FreeBSD equivalents of [.filename]#.deb# files on Debian/Ubuntu based systems and [.filename]#.rpm# files on Red Hat/Fedora based systems. Packages are installed using `pkg`. For example, the following command installs Apache 2.4:
Packages are pre-compiled applications, the FreeBSD equivalents of [.filename]#.deb# files on Debian/Ubuntu based systems and [.filename]#.rpm# files on Red Hat/Fedora based systems.
Packages are installed using `pkg`.
For example, the following command installs Apache 2.4:
[source,shell]
....
@ -77,9 +90,11 @@ For more information on packages refer to section 5.4 of the FreeBSD Handbook: l
[[ports]]
=== Ports
The FreeBSD Ports Collection is a framework of [.filename]#Makefiles# and patches specifically customized for installing applications from source on FreeBSD. When installing a port, the system will fetch the source code, apply any required patches, compile the code, and install the application and any required dependencies.
The FreeBSD Ports Collection is a framework of [.filename]#Makefiles# and patches specifically customized for installing applications from source on FreeBSD.
When installing a port, the system will fetch the source code, apply any required patches, compile the code, and install the application and any required dependencies.
The Ports Collection, sometimes referred to as the ports tree, can be installed to [.filename]#/usr/ports# using man:portsnap[8]. Detailed instructions for installing the Ports Collection can be found in link:{handbook}#ports-using[section 5.5] of the FreeBSD Handbook.
The Ports Collection, sometimes referred to as the ports tree, can be installed to [.filename]#/usr/ports# using man:portsnap[8].
Detailed instructions for installing the Ports Collection can be found in link:{handbook}#ports-using[section 5.5] of the FreeBSD Handbook.
To compile a port, change to the port's directory and start the build process. The following example installs Apache 2.4 from the Ports Collection:
@ -89,7 +104,8 @@ To compile a port, change to the port's directory and start the build process. T
# make install clean
....
A benefit of using ports to install software is the ability to customize the installation options. This example specifies that the mod_ldap module should also be installed:
A benefit of using ports to install software is the ability to customize the installation options.
This example specifies that the mod_ldap module should also be installed:
[source,shell]
....
@ -102,13 +118,24 @@ Refer to link:{handbook}#ports-using[Using the Ports Collection] for more inform
[[startup]]
== System Startup
Many Linux(R) distributions use the SysV init system, whereas FreeBSD uses the traditional BSD-style man:init[8]. Under the BSD-style man:init[8], there are no run-levels and [.filename]#/etc/inittab# does not exist. Instead, startup is controlled by man:rc[8] scripts. At system boot, [.filename]#/etc/rc# reads [.filename]#/etc/rc.conf# and [.filename]#/etc/defaults/rc.conf# to determine which services are to be started. The specified services are then started by running the corresponding service initialization scripts located in [.filename]#/etc/rc.d/# and [.filename]#/usr/local/etc/rc.d/#. These scripts are similar to the scripts located in [.filename]#/etc/init.d/# on Linux(R) systems.
Many Linux(R) distributions use the SysV init system, whereas FreeBSD uses the traditional BSD-style man:init[8].
Under the BSD-style man:init[8], there are no run-levels and [.filename]#/etc/inittab# does not exist.
Instead, startup is controlled by man:rc[8] scripts.
At system boot, [.filename]#/etc/rc# reads [.filename]#/etc/rc.conf# and [.filename]#/etc/defaults/rc.conf# to determine which services are to be started.
The specified services are then started by running the corresponding service initialization scripts located in [.filename]#/etc/rc.d/# and [.filename]#/usr/local/etc/rc.d/#. These scripts are similar to the scripts located in [.filename]#/etc/init.d/# on Linux(R) systems.
The scripts found in [.filename]#/etc/rc.d/# are for applications that are part of the "base" system, such as man:cron[8], man:sshd[8], and man:syslog[3]. The scripts in [.filename]#/usr/local/etc/rc.d/# are for user-installed applications such as Apache and Squid.
The scripts found in [.filename]#/etc/rc.d/# are for applications that are part of the "base" system, such as man:cron[8], man:sshd[8], and man:syslog[3].
The scripts in [.filename]#/usr/local/etc/rc.d/# are for user-installed applications such as Apache and Squid.
Since FreeBSD is developed as a complete operating system, user-installed applications are not considered to be part of the "base" system. User-installed applications are generally installed using link:{handbook}#ports-using[Packages or Ports]. In order to keep them separate from the base system, user-installed applications are installed under [.filename]#/usr/local/#. Therefore, user-installed binaries reside in [.filename]#/usr/local/bin/#, configuration files are in [.filename]#/usr/local/etc/#, and so on.
Since FreeBSD is developed as a complete operating system, user-installed applications are not considered to be part of the "base" system.
User-installed applications are generally installed using link:{handbook}#ports-using[Packages or Ports].
In order to keep them separate from the base system, user-installed applications are installed under [.filename]#/usr/local/#.
Therefore, user-installed binaries reside in [.filename]#/usr/local/bin/#, configuration files are in [.filename]#/usr/local/etc/#, and so on.
Services are enabled by adding an entry for the service in [.filename]#/etc/rc.conf# . The system defaults are found in [.filename]#/etc/defaults/rc.conf# and these default settings are overridden by settings in [.filename]#/etc/rc.conf#. Refer to man:rc.conf[5] for more information about the available entries. When installing additional applications, review the application's install message to determine how to enable any associated services.
Services are enabled by adding an entry for the service in [.filename]#/etc/rc.conf#.
The system defaults are found in [.filename]#/etc/defaults/rc.conf# and these default settings are overridden by settings in [.filename]#/etc/rc.conf#.
Refer to man:rc.conf[5] for more information about the available entries.
When installing additional applications, review the application's install message to determine how to enable any associated services.
The following entries in [.filename]#/etc/rc.conf# enable man:sshd[8], enable Apache 2.4, and specify that Apache should be started with SSL.
@ -139,7 +166,8 @@ If a service has not been enabled, it can be started from the command line using
[[network]]
== Network Configuration
Instead of a generic _ethX_ identifier that Linux(R) uses to identify a network interface, FreeBSD uses the driver name followed by a number. The following output from man:ifconfig[8] shows two Intel(R) Pro 1000 network interfaces ([.filename]#em0# and [.filename]#em1#):
Instead of a generic _ethX_ identifier that Linux(R) uses to identify a network interface, FreeBSD uses the driver name followed by a number.
The following output from man:ifconfig[8] shows two Intel(R) Pro 1000 network interfaces ([.filename]#em0# and [.filename]#em1#):
[source,shell]
....
@ -158,7 +186,9 @@ em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
status: active
....
An IP address can be assigned to an interface using man:ifconfig[8]. To remain persistent across reboots, the IP configuration must be included in [.filename]#/etc/rc.conf#. The following [.filename]#/etc/rc.conf# entries specify the hostname, IP address, and default gateway:
An IP address can be assigned to an interface using man:ifconfig[8].
To remain persistent across reboots, the IP configuration must be included in [.filename]#/etc/rc.conf#.
The following [.filename]#/etc/rc.conf# entries specify the hostname, IP address, and default gateway:
[.programlisting]
....
@ -178,13 +208,16 @@ ifconfig_em0="DHCP"
[[firewall]]
== Firewall
FreeBSD does not use Linux(R) IPTABLES for its firewall. Instead, FreeBSD offers a choice of three kernel level firewalls:
FreeBSD does not use Linux(R) IPTABLES for its firewall.
Instead, FreeBSD offers a choice of three kernel level firewalls:
* link:{handbook}#firewalls-pf[PF]
* link:{handbook}#firewalls-ipf[IPFILTER]
* link:{handbook}#firewalls-ipfw[IPFW]
PF is developed by the OpenBSD project and ported to FreeBSD. PF was created as a replacement for IPFILTER and its syntax is similar to that of IPFILTER. PF can be paired with man:altq[4] to provide QoS features.
PF is developed by the OpenBSD project and ported to FreeBSD.
PF was created as a replacement for IPFILTER and its syntax is similar to that of IPFILTER.
PF can be paired with man:altq[4] to provide QoS features.
This sample PF entry allows inbound SSH:
@ -193,7 +226,8 @@ This sample PF entry allows inbound SSH:
pass in on $ext_if inet proto tcp from any to ($ext_if) port 22
....
IPFILTER is the firewall application developed by Darren Reed. It is not specific to FreeBSD and has been ported to several operating systems including NetBSD, OpenBSD, SunOS, HP/UX, and Solaris.
IPFILTER is the firewall application developed by Darren Reed.
It is not specific to FreeBSD and has been ported to several operating systems including NetBSD, OpenBSD, SunOS, HP/UX, and Solaris.
The IPFILTER syntax to allow inbound SSH is:
@ -202,7 +236,8 @@ The IPFILTER syntax to allow inbound SSH is:
pass in on $ext_if proto tcp from any to any port = 22
....
IPFW is the firewall developed and maintained by FreeBSD. It can be paired with man:dummynet[4] to provide traffic shaping capabilities and simulate different types of network connections.
IPFW is the firewall developed and maintained by FreeBSD.
It can be paired with man:dummynet[4] to provide traffic shaping capabilities and simulate different types of network connections.
The IPFW syntax to allow inbound SSH would be:
@ -216,9 +251,13 @@ ipfw add allow tcp from any to me 22 in via $ext_if
There are two methods for updating a FreeBSD system: from source or binary updates.
Updating from source is the most involved update method, but offers the greatest amount of flexibility. The process involves synchronizing a local copy of the FreeBSD source code with the FreeBSD Subversion servers. Once the local source code is up-to-date, a new version of the kernel and userland can be compiled.
Updating from source is the most involved update method, but offers the greatest amount of flexibility.
The process involves synchronizing a local copy of the FreeBSD source code with the FreeBSD Subversion servers.
Once the local source code is up-to-date, a new version of the kernel and userland can be compiled.
Binary updates are similar to using `yum` or `apt-get` to update a Linux(R) system. In FreeBSD, man:freebsd-update[8] can be used fetch new binary updates and install them. These updates can be scheduled using man:cron[8].
Binary updates are similar to using `yum` or `apt-get` to update a Linux(R) system.
In FreeBSD, man:freebsd-update[8] can be used fetch new binary updates and install them.
These updates can be scheduled using man:cron[8].
[NOTE]
====
@ -236,7 +275,8 @@ For more information on source and binary updates, refer to link:{handbook}#upda
[[procfs]]
== procfs: Gone But Not Forgotten
In some Linux(R) distributions, one could look at [.filename]#/proc/sys/net/ipv4/ip_forward# to determine if IP forwarding is enabled. In FreeBSD, man:sysctl[8] is instead used to view this and other system settings.
In some Linux(R) distributions, one could look at [.filename]#/proc/sys/net/ipv4/ip_forward# to determine if IP forwarding is enabled.
In FreeBSD, man:sysctl[8] is instead used to view this and other system settings.
For example, use the following to determine if IP forwarding is enabled on a FreeBSD system:
@ -313,4 +353,5 @@ Some common command equivalents are as follows:
[[conclusion]]
== Conclusion
This document has provided an overview of FreeBSD. Refer to the link:{handbook}[FreeBSD Handbook] for more in-depth coverage of these topics as well as the many topics not covered by this document.
This document has provided an overview of FreeBSD.
Refer to the link:{handbook}[FreeBSD Handbook] for more in-depth coverage of these topics as well as the many topics not covered by this document.

69
documentation/content/en/articles/mailing-list-faq/_index.adoc
View file

@ -23,7 +23,11 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
This is the FAQ for the FreeBSD mailing lists. If you are interested in helping with this project, send email to the {freebsd-doc}. The latest version of this document is always available from the link:.[FreeBSD World Wide Web server]. It may also be downloaded as one large link:.[HTML] file with HTTP or as plain text, PostScript, PDF, etc. from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server]. You may also want to link:https://www.FreeBSD.org/search/[Search the FAQ].
This is the FAQ for the FreeBSD mailing lists.
If you are interested in helping with this project, send email to the {freebsd-doc}.
The latest version of this document is always available from the link:.[FreeBSD World Wide Web server].
It may also be downloaded as one large link:.[HTML] file with HTTP or as plain text, PostScript, PDF, etc. from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server].
You may also want to link:https://www.FreeBSD.org/search/[Search the FAQ].
'''
@ -32,9 +36,12 @@ toc::[]
[[introduction]]
== Introduction
As is usual with FAQs, this document aims to cover the most frequently asked questions concerning the FreeBSD mailing lists (and of course answer them!). Although originally intended to reduce bandwidth and avoid the same old questions being asked over and over again, FAQs have become recognized as valuable information resources.
As is usual with FAQs, this document aims to cover the most frequently asked questions concerning the FreeBSD mailing lists (and of course answer them!).
Although originally intended to reduce bandwidth and avoid the same old questions being asked over and over again, FAQs have become recognized as valuable information resources.
This document attempts to represent a community consensus, and as such it can never really be __authoritative__. However, if you find technical errors within this document, or have suggestions about items that should be added, please either submit a PR, or email the {freebsd-doc}. Thanks.
This document attempts to represent a community consensus, and as such it can never really be __authoritative__.
However, if you find technical errors within this document, or have suggestions about items that should be added, please either submit a PR, or email the {freebsd-doc}.
Thanks.
=== What is the purpose of the FreeBSD mailing lists?
@ -42,15 +49,21 @@ The FreeBSD mailing lists serve as the primary communication channels for the Fr
=== Who is the audience for the FreeBSD mailing lists?
This depends on charter of each individual list. Some lists are more oriented to developers; some are more oriented towards the FreeBSD community as a whole. Please see http://lists.FreeBSD.org/mailman/listinfo[this list] for the current summary.
This depends on charter of each individual list.
Some lists are more oriented to developers; some are more oriented towards the FreeBSD community as a whole.
Please see http://lists.FreeBSD.org/mailman/listinfo[this list] for the current summary.
=== Are the FreeBSD mailing lists open for anyone to participate?
Again, this depends on charter of each individual list. Please read the charter of a mailing list before you post to it, and respect it when you post. This will help everyone to have a better experience with the lists.
Again, this depends on charter of each individual list.
Please read the charter of a mailing list before you post to it, and respect it when you post.
This will help everyone to have a better experience with the lists.
If after reading the above lists, you still do not know which mailing list to post a question to, you will probably want to post to freebsd-questions (but see below, first).
Also note that the mailing lists have traditionally been open to postings from non-subscribers. This has been a deliberate choice, to help make joining the FreeBSD community an easier process, and to encourage open sharing of ideas. However, due to past abuse by some individuals, certain lists now have a policy where postings from non-subscribers must be manually screened to ensure that they are appropriate.
Also note that the mailing lists have traditionally been open to postings from non-subscribers.
This has been a deliberate choice, to help make joining the FreeBSD community an easier process, and to encourage open sharing of ideas.
However, due to past abuse by some individuals, certain lists now have a policy where postings from non-subscribers must be manually screened to ensure that they are appropriate.
=== How can I subscribe?
@ -60,7 +73,9 @@ You can use http://lists.FreeBSD.org/mailman/listinfo[the Mailman web interface]
You can use the same interface as above; or, you can follow the instructions that are at the bottom of every mailing list message that is sent.
Please do not send unsubscribe messages directly to the public lists themselves. First, this will not accomplish your goal, and second, it will irritate the existing subscribers, and you will probably get flamed. This is a classical mistake when using mailing lists; please try to avoid it.
Please do not send unsubscribe messages directly to the public lists themselves.
First, this will not accomplish your goal, and second, it will irritate the existing subscribers, and you will probably get flamed.
This is a classical mistake when using mailing lists; please try to avoid it.
=== Are archives available?
@ -73,13 +88,19 @@ Yes. See http://lists.FreeBSD.org/mailman/listinfo[the Mailman web interface].
[[etiquette]]
== Mailing List Etiquette
Participation in the mailing lists, like participation in any community, requires a common basis for communication. Please make only appropriate postings, and follow common rules of etiquette.
Participation in the mailing lists, like participation in any community, requires a common basis for communication.
Please make only appropriate postings, and follow common rules of etiquette.
=== What should I do before I post?
You have already taken the most important step by reading this document. However, if you are new to FreeBSD, you may first need to familiarize yourself with the software, and all the social history around it, by reading the numerous link:https://www.FreeBSD.org/docs/books/[books and articles] that are available. Items of particular interest include the link:{faq}[FreeBSD Frequently Asked Questions (FAQ)] document, the link:{handbook}[FreeBSD Handbook], and the articles link:{freebsd-questions-article}[How to get best results from the FreeBSD-questions mailing list], link:{explaining-bsd}[Explaining BSD], and link:{new-users}[FreeBSD First Steps].
You have already taken the most important step by reading this document.
However, if you are new to FreeBSD, you may first need to familiarize yourself with the software, and all the social history around it, by reading the numerous link:https://www.FreeBSD.org/docs/books/[books and articles] that are available.
Items of particular interest include the link:{faq}[FreeBSD Frequently Asked Questions (FAQ)] document, the link:{handbook}[FreeBSD Handbook], and the articles link:{freebsd-questions-article}[How to get best results from the FreeBSD-questions mailing list], link:{explaining-bsd}[Explaining BSD], and link:{new-users}[FreeBSD First Steps].
It is always considered bad form to ask a question that is already answered in the above documents. This is not because the volunteers who work on this project are particularly mean people, but after a certain number of times answering the same questions over and over again, frustration begins to set in. This is particularly true if there is an existing answer to the question that is already available. Always keep in mind that almost all of the work done on FreeBSD is done by volunteers, and that we are only human.
It is always considered bad form to ask a question that is already answered in the above documents.
This is not because the volunteers who work on this project are particularly mean people, but after a certain number of times answering the same questions over and over again, frustration begins to set in.
This is particularly true if there is an existing answer to the question that is already available.
Always keep in mind that almost all of the work done on FreeBSD is done by volunteers, and that we are only human.
=== What constitutes an inappropriate posting?
@ -94,13 +115,14 @@ It is always considered bad form to ask a question that is already answered in t
* Format your message so that it is legible, and PLEASE DO NOT SHOUT!!!!!. Do not underestimate the effect that a poorly formatted mail message has, and not just on the FreeBSD mailing lists. Your mail message is all that people see of you, and if it is poorly formatted, badly spelled, full of errors, and/or has lots of exclamation points, it will give people a poor impression of you.
* Please use an appropriate human language for a particular mailing list. Many non-English mailing lists are link:https://www.FreeBSD.org/community/mailinglists/[available].
+
For the ones that are not, we do appreciate that many people do not speak English as their first language, and we try to make allowances for that. It is considered particularly poor form to criticize non-native speakers for spelling or grammatical errors. FreeBSD has an excellent track record in this regard; please, help us to uphold that tradition.
For the ones that are not, we do appreciate that many people do not speak English as their first language, and we try to make allowances for that.
It is considered particularly poor form to criticize non-native speakers for spelling or grammatical errors.
FreeBSD has an excellent track record in this regard; please, help us to uphold that tradition.
* Please use a standards-compliant Mail User Agent (MUA). A lot of badly formatted messages come from http://www.lemis.com/grog/email/email.php[bad mailers or badly configured mailers]. The following mailers are known to send out badly formatted messages without you finding out about them:
** exmh
** Microsoft(R) Exchange
** Microsoft(R) Outlook(R)
+
Try not to use MIME: a lot of people use mailers which do not get on very well with MIME.
* Make sure your time and time zone are set correctly. This may seem a little silly, since your message still gets there, but many of the people on these mailing lists get several hundred messages a day. They frequently sort the incoming messages by subject and by date, and if your message does not come before the first answer, they may assume that they missed it and not bother to look.
@ -131,22 +153,33 @@ This is especially important for postings of the type "yes, I see this too", whe
[[recurring]]
== Recurring Topics On The Mailing Lists
Participation in the mailing lists, like participation in any community, requires a common basis for communication. Many of the mailing lists presuppose a knowledge of the Project's history. In particular, there are certain topics that seem to regularly occur to newcomers to the community. It is the responsibility of each poster to ensure that their postings do not fall into one of these categories. By doing so, you will help the mailing lists to stay on-topic, and probably save yourself being flamed in the process.
Participation in the mailing lists, like participation in any community, requires a common basis for communication.
Many of the mailing lists presuppose a knowledge of the Project's history.
In particular, there are certain topics that seem to regularly occur to newcomers to the community.
It is the responsibility of each poster to ensure that their postings do not fall into one of these categories.
By doing so, you will help the mailing lists to stay on-topic, and probably save yourself being flamed in the process.
The best method to avoid this is to familiarize yourself with the http://docs.FreeBSD.org/mail/[mailing list archives], to help yourself understand the background of what has gone before. In this, the https://www.FreeBSD.org/search/#mailinglists[mailing list search interface] is invaluable. (If that method does not yield useful results, please supplement it with a search with your favorite major search engine).
The best method to avoid this is to familiarize yourself with the http://docs.FreeBSD.org/mail/[mailing list archives], to help yourself understand the background of what has gone before.
In this, the https://www.FreeBSD.org/search/#mailinglists[mailing list search interface] is invaluable.
(If that method does not yield useful results, please supplement it with a search with your favorite major search engine).
By familiarizing yourself with the archives, not only will you learn what topics have been discussed before, but also how discussion tends to proceed on that list, who the participants are, and who the target audience is. These are always good things to know before you post to any mailing list, not just a FreeBSD mailing list.
By familiarizing yourself with the archives, not only will you learn what topics have been discussed before, but also how discussion tends to proceed on that list, who the participants are, and who the target audience is.
These are always good things to know before you post to any mailing list, not just a FreeBSD mailing list.
There is no doubt that the archives are quite extensive, and some questions recur more often than others, sometimes as followups where the subject line no longer accurately reflects the new content. Nevertheless, the burden is on you, the poster, to do your homework to help avoid these recurring topics.
There is no doubt that the archives are quite extensive, and some questions recur more often than others, sometimes as followups where the subject line no longer accurately reflects the new content.
Nevertheless, the burden is on you, the poster, to do your homework to help avoid these recurring topics.
[[bikeshed]]
== What Is A "Bikeshed"?
Literally, a `bikeshed` is a small outdoor shelter into which one may store one's two-wheeled form of transportation. However, in FreeBSD parlance, the term refers to topics that are simple enough that (nearly) anyone can offer an opinion about, and often (nearly) everyone does. The genesis of this term is explained in more detail link:{faq}#bikeshed-painting[in this document]. You simply must have a working knowledge of this concept before posting to any FreeBSD mailing list.
Literally, a `bikeshed` is a small outdoor shelter into which one may store one's two-wheeled form of transportation.
However, in FreeBSD parlance, the term refers to topics that are simple enough that (nearly) anyone can offer an opinion about, and often (nearly) everyone does. The genesis of this term is explained in more detail link:{faq}#bikeshed-painting[in this document].
You simply must have a working knowledge of this concept before posting to any FreeBSD mailing list.
More generally, a bikeshed is a topic that will tend to generate immediate meta-discussions and flames if you have not read up on their past history.
Please help us to keep the mailing lists as useful for as many people as possible by avoiding bikesheds whenever you can. Thanks.
Please help us to keep the mailing lists as useful for as many people as possible by avoiding bikesheds whenever you can.
Thanks.
[[acknowledgments]]
== Acknowledgments

91
documentation/content/en/articles/nanobsd/_index.adoc
View file

@ -20,6 +20,8 @@ trademarks: ["freebsd", "general"]
include::shared/authors.adoc[]
[.abstract-title]
Abstract
This document provides information about the NanoBSD tools, which can be used to create FreeBSD system images for embedded applications, suitable for use on a USB key, memory card or other mass storage media.
'''
@ -29,9 +31,12 @@ toc::[]
[[intro]]
== Introduction to NanoBSD
NanoBSD is a tool developed by {phk} and now maintained by {imp}. It creates a FreeBSD system image for embedded applications, suitable for use on a USB key, memory card or other mass storage media.
NanoBSD is a tool developed by {phk} and now maintained by {imp}.
It creates a FreeBSD system image for embedded applications, suitable for use on a USB key, memory card or other mass storage media.
It can be used to build specialized install images, designed for easy installation and maintenance of systems commonly called "computer appliances". Computer appliances have their hardware and software bundled in the product, which means all applications are pre-installed. The appliance is plugged into an existing network and can begin working (almost) immediately.
It can be used to build specialized install images, designed for easy installation and maintenance of systems commonly called "computer appliances".
Computer appliances have their hardware and software bundled in the product, which means all applications are pre-installed.
The appliance is plugged into an existing network and can begin working (almost) immediately.
The features of NanoBSD include:
@ -46,7 +51,8 @@ The features of NanoBSD include:
[[design]]
=== The Design of NanoBSD
Once the image is present on the medium, it is possible to boot NanoBSD. The mass storage medium is divided into three parts by default:
Once the image is present on the medium, it is possible to boot NanoBSD.
The mass storage medium is divided into three parts by default:
* Two image partitions: `code#1` and `code#2`.
* The configuration file partition, which can be mounted under the [.filename]#/cfg# directory at run time.
@ -55,7 +61,8 @@ These partitions are normally mounted read-only.
The [.filename]#/etc# and [.filename]#/var# directories are man:md[4] (malloc) disks.
The configuration file partition persists under the [.filename]#/cfg# directory. It contains files for [.filename]#/etc# directory and is briefly mounted read-only right after the system boot, therefore it is required to copy modified files from [.filename]#/etc# back to the [.filename]#/cfg# directory if changes are expected to persist after the system restarts.
The configuration file partition persists under the [.filename]#/cfg# directory.
It contains files for [.filename]#/etc# directory and is briefly mounted read-only right after the system boot, therefore it is required to copy modified files from [.filename]#/etc# back to the [.filename]#/cfg# directory if changes are expected to persist after the system restarts.
.Making Persistent Changes to [.filename]#/etc/resolv.conf#
[example]
@ -81,7 +88,8 @@ Keeping [.filename]#/cfg# mounted at all times is not a good idea, especially if
=== Building a NanoBSD Image
A NanoBSD image is built using a simple [.filename]#nanobsd.sh# shell script, which can be found in the [.filename]#/usr/src/tools/tools/nanobsd# directory. This script creates an image, which can be copied on the storage medium using the man:dd[1] utility.
A NanoBSD image is built using a simple [.filename]#nanobsd.sh# shell script, which can be found in the [.filename]#/usr/src/tools/tools/nanobsd# directory.
This script creates an image, which can be copied on the storage medium using the man:dd[1] utility.
The necessary commands to build a NanoBSD image are:
@ -103,7 +111,8 @@ The necessary commands to build a NanoBSD image are:
==== Options When Building a NanoBSD Image
When building a NanoBSD image, several build options can be passed to [.filename]#nanobsd.sh# on the command line. These options can have a significant impact on the build process.
When building a NanoBSD image, several build options can be passed to [.filename]#nanobsd.sh# on the command line.
These options can have a significant impact on the build process.
Some options are for verbosity purposes:
@ -111,7 +120,8 @@ Some options are for verbosity purposes:
* `-q`: makes output quieter.
* `-v`: makes output more verbose
Some other options can be used to restrict the building process. Sometimes it is not necessary to rebuild everything from sources, especially if an image has already been built, and only little change is made.
Some other options can be used to restrict the building process.
Sometimes it is not necessary to rebuild everything from sources, especially if an image has already been built, and only little change is made.
* `-k`: do not build the kernel
* `-w`: do not build world
@ -120,7 +130,8 @@ Some other options can be used to restrict the building process. Sometimes it is
* `-f`: do not build a disk image of the first partition (which is useful for upgrade purposes)
* `-n`: add `-DNO_CLEAN` to `buildworld`, `buildkernel`. Also, all the files that have already been built in a previous run are kept.
A configuration file can be used to tweak as many elements as desired. Load it with `-c`
A configuration file can be used to tweak as many elements as desired.
Load it with `-c`
The last options are:
@ -128,7 +139,9 @@ The last options are:
==== The Complete Image Building Process
The complete image building process is going through a lot of steps. The exact steps taken will depend on the chosen options when starting the script. Assuming the script is run with no particular options, this is what will happen.
The complete image building process is going through a lot of steps.
The exact steps taken will depend on the chosen options when starting the script.
Assuming the script is run with no particular options, this is what will happen.
. `run_early_customize`: commands that are defined in a supplied configuration file.
. `clean_build`: Just cleans the build environment by deleting the previously built files.
@ -151,7 +164,8 @@ The complete image building process is going through a lot of steps. The exact s
=== Customizing a NanoBSD Image
This is probably the most important and most interesting feature of NanoBSD. This is also where you will be spending most of the time when developing with NanoBSD.
This is probably the most important and most interesting feature of NanoBSD.
This is also where you will be spending most of the time when developing with NanoBSD.
Invocation of the following command will force the [.filename]#nanobsd.sh# to read its configuration from [.filename]#myconf.nano# located in the current directory:
@ -167,9 +181,12 @@ Customization is done in two ways:
==== Configuration Options
With configuration settings, it is possible to configure options passed to both the `buildworld` and `installworld` stages of the NanoBSD build process, as well as internal options passed to the main build process of NanoBSD. Through these options it is possible to cut the system down, so it will fit on as little as 64MB. You can use the configuration options to trim down FreeBSD even more, until it will consists of just the kernel and two or three files in the userland.
With configuration settings, it is possible to configure options passed to both the `buildworld` and `installworld` stages of the NanoBSD build process, as well as internal options passed to the main build process of NanoBSD.
Through these options it is possible to cut the system down, so it will fit on as little as 64MB.
You can use the configuration options to trim down FreeBSD even more, until it will consists of just the kernel and two or three files in the userland.
The configuration file consists of configuration options, which override the default values. The most important directives are:
The configuration file consists of configuration options, which override the default values.
The most important directives are:
* `NANO_NAME` - Name of build (used to construct the workdir names).
* `NANO_SRC` - Path to the source tree used to build the image.
@ -191,28 +208,43 @@ There are three stages, by design, at which it is possible to make changes that
To customize a NanoBSD image, at any of these steps, it is best to add a specific value to one of the corresponding variables.
The `NANO_EARLY_CUSTOMIZE` variable is used at the first step of the building process. At this point, there is no example as to what can be done using that variable, but it may change in the future.
The `NANO_EARLY_CUSTOMIZE` variable is used at the first step of the building process.
At this point, there is no example as to what can be done using that variable, but it may change in the future.
The `NANO_CUSTOMIZE` variable is used after the kernel, world and etc configuration files have been installed, and the etc files have been setup as to be a NanoBSD installation. So it is the correct step in the building process to tweak configuration options and add packages, like in the cust_nobeastie example.
The `NANO_CUSTOMIZE` variable is used after the kernel, world and etc configuration files have been installed, and the etc files have been setup as to be a NanoBSD installation.
So it is the correct step in the building process to tweak configuration options and add packages, like in the cust_nobeastie example.
The `NANO_LATE_CUSTOMIZE` variable is used just before the disk image is created, so it is the very last moment to change anything. Remember that the `setup_nanobsd` routine already executed and that the [.filename]#etc#, [.filename]#conf# and [.filename]#cfg#directories and subdirectories are already modified, so it is not time to change them at this point. Rather, it is possible to add or remove specific files.
The `NANO_LATE_CUSTOMIZE` variable is used just before the disk image is created, so it is the very last moment to change anything.
Remember that the `setup_nanobsd` routine already executed and that the [.filename]#etc#, [.filename]#conf# and [.filename]#cfg# directories and subdirectories are already modified, so it is not time to change them at this point.
Rather, it is possible to add or remove specific files.
===== Booting Options
There are also variables that can change the way the NanoBSD image boots. Two options are passed to man:boot0cfg[8] to initialize the boot sector of the disk image:
There are also variables that can change the way the NanoBSD image boots.
Two options are passed to man:boot0cfg[8] to initialize the boot sector of the disk image:
* `NANO_BOOT0CFG`
* `NANO_BOOTLOADER`
With `NANO_BOOTLOADER` a bootloader file can be chosen. The most common possible options are between [.filename]#boot0sio# and [.filename]#boot0# depending on whether the appliance has a serial port or not. It is best to avoid supplying a different bootloader, but it is possible. To do so, it is best to have checked the link:{handbook}#boot/[FreeBSD Handbook] chapter on the boot process.
With `NANO_BOOTLOADER` a bootloader file can be chosen.
The most common possible options are between [.filename]#boot0sio# and [.filename]#boot0# depending on whether the appliance has a serial port or not.
It is best to avoid supplying a different bootloader, but it is possible.
To do so, it is best to have checked the link:{handbook}#boot/[FreeBSD Handbook] chapter on the boot process.
With `NANO_BOOT0CFG`, the booting process can be tweaked, like selecting on which partition the NanoBSD image will actually boot. It is best to check the man:boot0cfg[8] page before changing the default value of this variable. One option that could be interesting to change is the timeout of the booting procedure. To do so, the `NANO_BOOT0CFG` variable can be changed to `"-o packet -s 1 -m 3 -t 36"`. That way the booting process would start after approximately 2 seconds; because it is rare that waiting 10 seconds before actually booting is desired.
With `NANO_BOOT0CFG`, the booting process can be tweaked, like selecting on which partition the NanoBSD image will actually boot.
It is best to check the man:boot0cfg[8] page before changing the default value of this variable.
One option that could be interesting to change is the timeout of the booting procedure.
To do so, the `NANO_BOOT0CFG` variable can be changed to `"-o packet -s 1 -m 3 -t 36"`.
That way the booting process would start after approximately 2 seconds;
because it is rare that waiting 10 seconds before actually booting is desired.
Good to know: the `NANO_BOOT2CFG` variable is only used in the `cust_comconsole` routine that can be called at the `NANO_CUSTOMIZE` step if the appliance has a serial port and all console input and output has to take place through it. Be sure to check the relevant parameters of the serial port, as setting a bad parameter value can make it useless.
Good to know: the `NANO_BOOT2CFG` variable is only used in the `cust_comconsole` routine that can be called at the `NANO_CUSTOMIZE` step if the appliance has a serial port and all console input and output has to take place through it.
Be sure to check the relevant parameters of the serial port, as setting a bad parameter value can make it useless.
===== Disk Image Creation
In the end of the boot process is the disk image creation. With this step, the NanoBSD script provides a file that can simply be copied onto a disk for the appliance, and that will make it boot and start.
In the end of the boot process is the disk image creation.
With this step, the NanoBSD script provides a file that can simply be copied onto a disk for the appliance, and that will make it boot and start.
There are many variable that need to be set just right for the script to produce a usable disk image.
@ -225,7 +257,8 @@ There are many variable that need to be set just right for the script to produce
==== Custom Functions
It is possible to fine-tune NanoBSD using shell functions in the configuration file. The following example illustrates the basic model of custom functions:
It is possible to fine-tune NanoBSD using shell functions in the configuration file.
The following example illustrates the basic model of custom functions:
[.programlisting]
....
@ -260,9 +293,14 @@ Packages can be added to a NanoBSD image, to provide specific functionalities on
* Add the `cust_pkgng` to the `NANO_CUSTOMIZE` variable, or
* Add a `'customize_cmd cust_pkgng'` command in a customized configuration file.
Both methods achieve the same result: launching the `cust_pkgng` routine. This routine will go through `NANO_PACKAGE_DIR` directory to find either all packages or just the list of packages in the `NANO_PACKAGE_LIST` variable.
Both methods achieve the same result: launching the `cust_pkgng` routine.
This routine will go through `NANO_PACKAGE_DIR` directory to find either all packages or just the list of packages in the `NANO_PACKAGE_LIST` variable.
It is common, when installing applications through pkg on a standard FreeBSD environment, that the install process puts configuration files, in the [.filename]#usr/local/etc# directory, and startup scripts in the [.filename]#/usr/local/etc/rc.d# directory. So, after the required packages have been installed, they need to be configured in order for them to start right out of the box. To do so, the necessary configuration files have to be installed in the correct directories. This can be achieved by writing dedicated routines or the generic `cust_install_files` routine can be used to lay out files properly from the [.filename]#/usr/src/tools/tools/nanobsd/Files# directory. Usually a statement, sometimes multiple statements, in the [.filename]#/etc/rc.conf# also needs to be added for each package.
It is common, when installing applications through pkg on a standard FreeBSD environment, that the install process puts configuration files, in the [.filename]#usr/local/etc# directory, and startup scripts in the [.filename]#/usr/local/etc/rc.d# directory.
So, after the required packages have been installed, they need to be configured in order for them to start right out of the box.
To do so, the necessary configuration files have to be installed in the correct directories.
This can be achieved by writing dedicated routines or the generic `cust_install_files` routine can be used to lay out files properly from the [.filename]#/usr/src/tools/tools/nanobsd/Files# directory.
Usually a statement, sometimes multiple statements, in the [.filename]#/etc/rc.conf# also needs to be added for each package.
==== Configuration File Example
@ -321,9 +359,12 @@ customize_cmd cust_allow_ssh_root
customize_cmd cust_nobeastie
....
All the build and install compilation options can be found in the man:src.conf[5] man page, but not all options can or should be used when building a NanoBSD image. The build and install options should be defined according to the needs of the image being built.
All the build and install compilation options can be found in the man:src.conf[5] man page, but not all options can or should be used when building a NanoBSD image.
The build and install options should be defined according to the needs of the image being built.
For example, the ftp client and server might not be needed. Adding `WITHOUT_FTP=TRUE` to a configuration file in the `CONF_BUILD` section will avoid having them built. Also, if the NanoBSD appliance will not be used to build programs then it is possible to add the `WITHOUT_BINUTILS=TRUE` in the `CONF_INSTALL` section; but not in the `CONF_BUILD` section as they will be used to build the NanoBSD image.
For example, the ftp client and server might not be needed.
Adding `WITHOUT_FTP=TRUE` to a configuration file in the `CONF_BUILD` section will avoid having them built.
Also, if the NanoBSD appliance will not be used to build programs then it is possible to add the `WITHOUT_BINUTILS=TRUE` in the `CONF_INSTALL` section; but not in the `CONF_BUILD` section as they will be used to build the NanoBSD image.
Not building a particular set of programs - through a compilation option - shortens the overall building time and lowers the required size for the disk image, whereas not installing the same specific set of programs does not lower the overall building time.

189
documentation/content/en/articles/new-users/_index.adoc
View file

@ -29,7 +29,10 @@ toc::[]
[[in-and-out]]
== Logging in and Getting Out
Log in (when you see `login:`) as a user you created during installation or as `root`. (Your FreeBSD installation will already have an account for `root`; who can go anywhere and do anything, including deleting essential files, so be careful!) The symbols % and # in the following stand for the prompt (yours may be different), with % indicating an ordinary user and # indicating `root`.
Log in (when you see `login:`) as a user you created during installation or as `root`.
(Your FreeBSD installation will already have an account for `root`;
who can go anywhere and do anything, including deleting essential files, so be careful!)
The symbols % and # in the following stand for the prompt (yours may be different), with % indicating an ordinary user and # indicating `root`.
To log out (and get a new `login:` prompt) type
@ -38,7 +41,8 @@ To log out (and get a new `login:` prompt) type
# exit
....
as often as necessary. Yes, press kbd:[enter] after commands, and remember that UNIX(R) is case-sensitive-``exit``, not `EXIT`.
as often as necessary.
Yes, press kbd:[enter] after commands, and remember that UNIX(R) is case-sensitive-``exit``, not `EXIT`.
To shut down the machine type
@ -61,7 +65,10 @@ or
# /sbin/reboot
....
You can also reboot with kbd:[Ctrl+Alt+Delete]. Give it a little time to do its work. This is equivalent to `/sbin/reboot` in recent releases of FreeBSD and is much, much better than hitting the reset button. You do not want to have to reinstall this thing, do you?
You can also reboot with kbd:[Ctrl+Alt+Delete].
Give it a little time to do its work.
This is equivalent to `/sbin/reboot` in recent releases of FreeBSD and is much, much better than hitting the reset button.
You do not want to have to reinstall this thing, do you?
[[adding-a-user]]
== Adding a User with Root Privileges
@ -73,22 +80,31 @@ If you did not create any users when you installed the system and are thus logge
# adduser
....
The first time you use `adduser`, it might ask for some defaults to save. You might want to make the default shell man:csh[1] instead of man:sh[1], if it suggests `sh` as the default. Otherwise just press enter to accept each default. These defaults are saved in [.filename]#/etc/adduser.conf#, an editable file.
The first time you use `adduser`, it might ask for some defaults to save.
You might want to make the default shell man:csh[1] instead of man:sh[1], if it suggests `sh` as the default.
Otherwise just press enter to accept each default.
These defaults are saved in [.filename]#/etc/adduser.conf#, an editable file.
Suppose you create a user `jack` with full name _Jack Benimble_. Give `jack` a password if security (even kids around who might pound on the keyboard) is an issue. When it asks you if you want to invite `jack` into other groups, type `wheel`
Suppose you create a user `jack` with full name _Jack Benimble_.
Give `jack` a password if security (even kids around who might pound on the keyboard) is an issue.
When it asks you if you want to invite `jack` into other groups, type `wheel`
[source,shell]
....
Login group is "jack". Invite jack into other groups: wheel
....
This will make it possible to log in as `jack` and use the man:su[1] command to become `root`. Then you will not get scolded any more for logging in as `root`.
This will make it possible to log in as `jack` and use the man:su[1] command to become `root`.
Then you will not get scolded any more for logging in as `root`.
You can quit `adduser` any time by typing kbd:[Ctrl+C], and at the end you will have a chance to approve your new user or simply type kbd:[n] for no. You might want to create a second new user so that when you edit `jack`'s login files, you will have a hot spare in case something goes wrong.
You can quit `adduser` any time by typing kbd:[Ctrl+C], and at the end you will have a chance to approve your new user or simply type kbd:[n] for no.
You might want to create a second new user so that when you edit `jack`'s login files, you will have a hot spare in case something goes wrong.
Once you have done this, use `exit` to get back to a login prompt and log in as `jack`. In general, it is a good idea to do as much work as possible as an ordinary user who does not have the power-and risk-of `root`.
Once you have done this, use `exit` to get back to a login prompt and log in as `jack`.
In general, it is a good idea to do as much work as possible as an ordinary user who does not have the power-and risk-of `root`.
If you already created a user and you want the user to be able to `su` to `root`, you can log in as `root` and edit the file [.filename]#/etc/group#, adding `jack` to the first line (the group `wheel`). But first you need to practice man:vi[1], the text editor-or use the simpler text editor, man:ee[1], installed on recent versions of FreeBSD.
If you already created a user and you want the user to be able to `su` to `root`, you can log in as `root` and edit the file [.filename]#/etc/group#, adding `jack` to the first line (the group `wheel`).
But first you need to practice man:vi[1], the text editor-or use the simpler text editor, man:ee[1], installed on recent versions of FreeBSD.
To delete a user, use `rmuser`.
@ -115,29 +131,43 @@ Lists the files in the current directory with a * after executables, a `/` after
Lists the files in long format-size, date, permissions.
`ls -a`::
Lists hidden "dot" files with the others. If you are `root`, the "dot" files show up without the `-a` switch.
Lists hidden "dot" files with the others.
If you are `root`, the "dot" files show up without the `-a` switch.
`cd`::
Changes directories. `cd ..` backs up one level; note the space after `cd`. `cd /usr/local` goes there. `cd ~` goes to the home directory of the person logged in-e.g., [.filename]#/usr/home/jack#. Try `cd /cdrom`, and then `ls`, to find out if your CDROM is mounted and working.
Changes directories. `cd ..` backs up one level; note the space after `cd`.
`cd /usr/local` goes there. `cd ~` goes to the home directory of the person logged in-e.g., [.filename]#/usr/home/jack#.
Try `cd /cdrom`, and then `ls`, to find out if your CDROM is mounted and working.
`less _filename_`::
Lets you look at a file (named _filename_) without changing it. Try `less /etc/fstab`. Type `q` to quit.
Lets you look at a file (named _filename_) without changing it.
Try `less /etc/fstab`.
Type `q` to quit.
`cat _filename_`::
Displays _filename_ on screen. If it is too long and you can see only the end of it, press kbd:[ScrollLock] and use the kbd:[up-arrow] to move backward; you can use kbd:[ScrollLock] with manual pages too. Press kbd:[ScrollLock] again to quit scrolling. You might want to try `cat` on some of the dot files in your home directory-`cat .cshrc`, `cat .login`, `cat .profile`.
Displays _filename_ on screen.
If it is too long and you can see only the end of it, press kbd:[ScrollLock] and use the kbd:[up-arrow] to move backward; you can use kbd:[ScrollLock] with manual pages too.
Press kbd:[ScrollLock] again to quit scrolling.
You might want to try `cat` on some of the dot files in your home directory-`cat .cshrc`, `cat .login`, `cat .profile`.
You will notice aliases in [.filename]#.cshrc# for some of the `ls` commands (they are very convenient). You can create other aliases by editing [.filename]#.cshrc#. You can make these aliases available to all users on the system by putting them in the system-wide `csh` configuration file, [.filename]#/etc/csh.cshrc#.
You will notice aliases in [.filename]#.cshrc# for some of the `ls` commands (they are very convenient).
You can create other aliases by editing [.filename]#.cshrc#.
You can make these aliases available to all users on the system by putting them in the system-wide `csh` configuration file, [.filename]#/etc/csh.cshrc#.
[[getting-help]]
== Getting Help and Information
Here are some useful sources of help. _Text_ stands for something of your choice that you type in-usually a command or filename.
Here are some useful sources of help.
_Text_ stands for something of your choice that you type in-usually a command or filename.
`apropos _text_`::
Everything containing string _text_ in the `whatis database`.
`man _text_`::
The manual page for _text_. The major source of documentation for UNIX(R) systems. `man ls` will tell you all the ways to use `ls`. Press kbd:[Enter] to move through text, kbd:[Ctrl+B] to go back a page, kbd:[Ctrl+F] to go forward, kbd:[q] or kbd:[Ctrl+C] to quit.
The manual page for _text_.
The major source of documentation for UNIX(R) systems.
`man ls` will tell you all the ways to use `ls`.
Press kbd:[Enter] to move through text, kbd:[Ctrl+B] to go back a page, kbd:[Ctrl+F] to go forward, kbd:[q] or kbd:[Ctrl+C] to quit.
`which _text_`::
Tells you where in the user's path the command _text_ is found.
@ -146,14 +176,19 @@ Tells you where in the user's path the command _text_ is found.
All the paths where the string _text_ is found.
`whatis _text_`::
Tells you what the command _text_ does and its manual page. Typing `whatis *` will tell you about all the binaries in the current directory.
Tells you what the command _text_ does and its manual page.
Typing `whatis *` will tell you about all the binaries in the current directory.
`whereis _text_`::
Finds the file _text_, giving its full path.
You might want to try using `whatis` on some common useful commands like `cat`, `more`, `grep`, `mv`, `find`, `tar`, `chmod`, `chown`, `date`, and `script`. `more` lets you read a page at a time as it does in DOS, e.g., `ls -l | more` or `more _filename_`. The * works as a wildcard-e.g., `ls w*` will show you files beginning with `w`.
You might want to try using `whatis` on some common useful commands like `cat`, `more`, `grep`, `mv`, `find`, `tar`, `chmod`, `chown`, `date`, and `script`.
`more` lets you read a page at a time as it does in DOS, e.g., `ls -l | more` or `more _filename_`.
The * works as a wildcard-e.g., `ls w*` will show you files beginning with `w`.
Are some of these not working very well? Both man:locate[1] and man:whatis[1] depend on a database that is rebuilt weekly. If your machine is not going to be left on over the weekend (and running FreeBSD), you might want to run the commands for daily, weekly, and monthly maintenance now and then. Run them as `root` and, for now, give each one time to finish before you start the next one.
Are some of these not working very well? Both man:locate[1] and man:whatis[1] depend on a database that is rebuilt weekly.
If your machine is not going to be left on over the weekend (and running FreeBSD), you might want to run the commands for daily, weekly, and monthly maintenance now and then.
Run them as `root` and, for now, give each one time to finish before you start the next one.
[source,shell]
....
@ -165,23 +200,36 @@ output omitted
output omitted
....
If you get tired of waiting, press kbd:[Alt+F2] to get another _virtual console_, and log in again. After all, it is a multi-user, multi-tasking system. Nevertheless these commands will probably flash messages on your screen while they are running; you can type `clear` at the prompt to clear the screen. Once they have run, you might want to look at [.filename]#/var/mail/root# and [.filename]#/var/log/messages#.
If you get tired of waiting, press kbd:[Alt+F2] to get another _virtual console_, and log in again.
After all, it is a multi-user, multi-tasking system.
Nevertheless these commands will probably flash messages on your screen while they are running; you can type `clear` at the prompt to clear the screen.
Once they have run, you might want to look at [.filename]#/var/mail/root# and [.filename]#/var/log/messages#.
Running such commands is part of system administration-and as a single user of a UNIX(R) system, you are your own system administrator. Virtually everything you need to be `root` to do is system administration. Such responsibilities are not covered very well even in those big fat books on UNIX(R), which seem to devote a lot of space to pulling down menus in windows managers. You might want to get one of the two leading books on systems administration, either Evi Nemeth et.al.'s UNIX System Administration Handbook (Prentice-Hall, 1995, ISBN 0-13-15051-7)-the second edition with the red cover; or Æleen Frisch's Essential System Administration (O'Reilly & Associates, 2002, ISBN 0-596-00343-9). I used Nemeth.
Running such commands is part of system administration-and as a single user of a UNIX(R) system, you are your own system administrator.
Virtually everything you need to be `root` to do is system administration.
Such responsibilities are not covered very well even in those big fat books on UNIX(R), which seem to devote a lot of space to pulling down menus in windows managers.
You might want to get one of the two leading books on systems administration, either Evi Nemeth et.al.'s UNIX System Administration Handbook (Prentice-Hall, 1995, ISBN 0-13-15051-7)-the second edition with the red cover; or Æleen Frisch's Essential System Administration (O'Reilly & Associates, 2002, ISBN 0-596-00343-9).
I used Nemeth.
[[editing-text]]
== Editing Text
To configure your system, you need to edit text files. Most of them will be in the [.filename]#/etc# directory; and you will need to `su` to `root` to be able to change them. You can use the easy `ee`, but in the long run the text editor `vi` is worth learning. There is an excellent tutorial on vi in [.filename]#/usr/src/contrib/nvi/docs/tutorial#, if you have the system sources installed.
To configure your system, you need to edit text files.
Most of them will be in the [.filename]#/etc# directory; and you will need to `su` to `root` to be able to change them.
You can use the easy `ee`, but in the long run the text editor `vi` is worth learning.
There is an excellent tutorial on vi in [.filename]#/usr/src/contrib/nvi/docs/tutorial#, if you have the system sources installed.
Before you edit a file, you should probably back it up. Suppose you want to edit [.filename]#/etc/rc.conf#. You could just use `cd /etc` to get to the [.filename]#/etc# directory and do:
Before you edit a file, you should probably back it up.
Suppose you want to edit [.filename]#/etc/rc.conf#.
You could just use `cd /etc` to get to the [.filename]#/etc# directory and do:
[source,shell]
....
# cp rc.conf rc.conf.orig
....
This would copy [.filename]#rc.conf# to [.filename]#rc.conf.orig#, and you could later copy [.filename]#rc.conf.orig# to [.filename]#rc.conf# to recover the original. But even better would be moving (renaming) and then copying back:
This would copy [.filename]#rc.conf# to [.filename]#rc.conf.orig#, and you could later copy [.filename]#rc.conf.orig# to [.filename]#rc.conf# to recover the original.
But even better would be moving (renaming) and then copying back:
[source,shell]
....
@ -189,7 +237,9 @@ This would copy [.filename]#rc.conf# to [.filename]#rc.conf.orig#, and you could
# cp rc.conf.orig rc.conf
....
because `mv` preserves the original date and owner of the file. You can now edit [.filename]#rc.conf#. If you want the original back, you would then `mv rc.conf rc.conf.myedit` (assuming you want to preserve your edited version) and then
because `mv` preserves the original date and owner of the file.
You can now edit [.filename]#rc.conf#.
If you want the original back, you would then `mv rc.conf rc.conf.myedit` (assuming you want to preserve your edited version) and then
[source,shell]
....
@ -205,7 +255,9 @@ To edit a file, type
# vi filename
....
Move through the text with the arrow keys. kbd:[Esc] (the escape key) puts `vi` in command mode. Here are some commands:
Move through the text with the arrow keys.
kbd:[Esc] (the escape key) puts `vi` in command mode.
Here are some commands:
`x`::
delete letter the cursor is on
@ -219,7 +271,8 @@ insert text at the cursor
`a`::
insert text after the cursor
Once you type `i` or `a`, you can enter text. `Esc` puts you back in command mode where you can type
Once you type `i` or `a`, you can enter text.
`Esc` puts you back in command mode where you can type
`:w`::
to write your changes to disk and continue editing
@ -245,9 +298,14 @@ to redraw the screen
kbd:[Ctrl+b] and kbd:[Ctrl+f]::
go back and forward a screen, as they do with `more` and `view`.
Practice with `vi` in your home directory by creating a new file with `vi _filename_` and adding and deleting text, saving the file, and calling it up again. `vi` delivers some surprises because it is really quite complex, and sometimes you will inadvertently issue a command that will do something you do not expect. (Some people actually like `vi`-it is more powerful than DOS EDIT-find out about `:r`.) Use kbd:[Esc] one or more times to be sure you are in command mode and proceed from there when it gives you trouble, save often with `:w`, and use `:q!` to get out and start over (from your last `:w`) when you need to.
Practice with `vi` in your home directory by creating a new file with `vi _filename_` and adding and deleting text, saving the file, and calling it up again.
`vi` delivers some surprises because it is really quite complex, and sometimes you will inadvertently issue a command that will do something you do not expect.
(Some people actually like `vi`-it is more powerful than DOS EDIT-find out about `:r`.)
Use kbd:[Esc] one or more times to be sure you are in command mode and proceed from there when it gives you trouble, save often with `:w`, and use `:q!` to get out and start over (from your last `:w`) when you need to.
Now you can `cd` to [.filename]#/etc#, `su` to `root`, use `vi` to edit the file [.filename]#/etc/group#, and add a user to `wheel` so the user has root privileges. Just add a comma and the user's login name to the end of the first line in the file, press kbd:[Esc], and use `:wq` to write the file to disk and quit. Instantly effective. (You did not put a space after the comma, did you?)
Now you can `cd` to [.filename]#/etc#, `su` to `root`, use `vi` to edit the file [.filename]#/etc/group#, and add a user to `wheel` so the user has root privileges.
Just add a comma and the user's login name to the end of the first line in the file, press kbd:[Esc], and use `:wq` to write the file to disk and quit.
Instantly effective. (You did not put a space after the comma, did you?)
[[other-useful-commands]]
== Other Useful Commands
@ -280,18 +338,24 @@ Use `find` to locate [.filename]#filename# in [.filename]#/usr# or any of its su
% find /usr -name "filename"
....
You can use * as a wildcard in `"_filename_"` (which should be in quotes). If you tell `find` to search in [.filename]#/# instead of [.filename]#/usr# it will look for the file(s) on all mounted filesystems, including the CDROM and the DOS partition.
You can use * as a wildcard in `"_filename_"` (which should be in quotes).
If you tell `find` to search in [.filename]#/# instead of [.filename]#/usr# it will look for the file(s) on all mounted filesystems, including the CDROM and the DOS partition.
An excellent book that explains UNIX(R) commands and utilities is Abrahams & Larson, Unix for the Impatient (2nd ed., Addison-Wesley, 1996). There is also a lot of UNIX(R) information on the Internet.
[[next-steps]]
== Next Steps
You should now have the tools you need to get around and edit files, so you can get everything up and running. There is a great deal of information in the FreeBSD handbook (which is probably on your hard drive) and link:https://www.FreeBSD.org/[FreeBSD's web site]. A wide variety of packages and ports are on the CDROM as well as the web site. The handbook tells you more about how to use them (get the package if it exists, with `pkg add _packagename_`, where _packagename_ is the filename of the package). The CDROM has lists of the packages and ports with brief descriptions in [.filename]#cdrom/packages/index#, [.filename]#cdrom/packages/index.txt#, and [.filename]#cdrom/ports/index#, with fuller descriptions in [.filename]#/cdrom/ports/\*/*/pkg/DESCR#, where the *s represent subdirectories of kinds of programs and program names respectively.
You should now have the tools you need to get around and edit files, so you can get everything up and running.
There is a great deal of information in the FreeBSD handbook (which is probably on your hard drive) and link:https://www.FreeBSD.org/[FreeBSD's web site].
A wide variety of packages and ports are on the CDROM as well as the web site.
The handbook tells you more about how to use them (get the package if it exists, with `pkg add _packagename_`, where _packagename_ is the filename of the package).
The CDROM has lists of the packages and ports with brief descriptions in [.filename]#cdrom/packages/index#, [.filename]#cdrom/packages/index.txt#, and [.filename]#cdrom/ports/index#, with fuller descriptions in [.filename]#/cdrom/ports/\*/*/pkg/DESCR#, where the *s represent subdirectories of kinds of programs and program names respectively.
If you find the handbook too sophisticated (what with `lndir` and all) on installing ports from the CDROM, here is what usually works:
Find the port you want, say `kermit`. There will be a directory for it on the CDROM. Copy the subdirectory to [.filename]#/usr/local# (a good place for software you add that should be available to all users) with:
Find the port you want, say `kermit`. There will be a directory for it on the CDROM.
Copy the subdirectory to [.filename]#/usr/local# (a good place for software you add that should be available to all users) with:
[source,shell]
....
@ -300,25 +364,40 @@ Find the port you want, say `kermit`. There will be a directory for it on the CD
This should result in a [.filename]#/usr/local/kermit# subdirectory that has all the files that the `kermit` subdirectory on the CDROM has.
Next, create the directory [.filename]#/usr/ports/distfiles# if it does not already exist using `mkdir`. Now check [.filename]#/cdrom/ports/distfiles# for a file with a name that indicates it is the port you want. Copy that file to [.filename]#/usr/ports/distfiles#; in recent versions you can skip this step, as FreeBSD will do it for you. In the case of `kermit`, there is no distfile.
Next, create the directory [.filename]#/usr/ports/distfiles# if it does not already exist using `mkdir`.
Now check [.filename]#/cdrom/ports/distfiles# for a file with a name that indicates it is the port you want.
Copy that file to [.filename]#/usr/ports/distfiles#; in recent versions you can skip this step, as FreeBSD will do it for you.
In the case of `kermit`, there is no distfile.
Then `cd` to the subdirectory of [.filename]#/usr/local/kermit# that has the file [.filename]#Makefile#. Type
Then `cd` to the subdirectory of [.filename]#/usr/local/kermit# that has the file [.filename]#Makefile#.
Type
[source,shell]
....
# make all install
....
During this process the port will FTP to get any compressed files it needs that it did not find on the CDROM or in [.filename]#/usr/ports/distfiles#. If you do not have your network running yet and there was no file for the port in [.filename]#/cdrom/ports/distfiles#, you will have to get the distfile using another machine and copy it to [.filename]#/usr/ports/distfiles#. Read [.filename]#Makefile# (with `cat` or `more` or `view`) to find out where to go (the master distribution site) to get the file and what its name is. (Use binary file transfers!) Then go back to [.filename]#/usr/local/kermit#, find the directory with [.filename]#Makefile#, and type `make all install`.
During this process the port will FTP to get any compressed files it needs that it did not find on the CDROM or in [.filename]#/usr/ports/distfiles#.
If you do not have your network running yet and there was no file for the port in [.filename]#/cdrom/ports/distfiles#, you will have to get the distfile using another machine and copy it to [.filename]#/usr/ports/distfiles#.
Read [.filename]#Makefile# (with `cat` or `more` or `view`) to find out where to go (the master distribution site) to get the file and what its name is.
(Use binary file transfers!) Then go back to [.filename]#/usr/local/kermit#, find the directory with [.filename]#Makefile#, and type `make all install`.
[[your-working-environment]]
== Your Working Environment
Your shell is the most important part of your working environment. The shell is what interprets the commands you type on the command line, and thus communicates with the rest of the operating system. You can also write shell scripts a series of commands to be run without intervention.
Your shell is the most important part of your working environment.
The shell is what interprets the commands you type on the command line, and thus communicates with the rest of the operating system.
You can also write shell scripts a series of commands to be run without intervention.
Two shells come installed with FreeBSD: `csh` and `sh`. `csh` is good for command-line work, but scripts should be written with `sh` (or `bash`). You can find out what shell you have by typing `echo $SHELL`.
Two shells come installed with FreeBSD: `csh` and `sh`.
`csh` is good for command-line work, but scripts should be written with `sh` (or `bash`).
You can find out what shell you have by typing `echo $SHELL`.
The `csh` shell is okay, but `tcsh` does everything `csh` does and more. It allows you to recall commands with the arrow keys and edit them. It has tab-key completion of filenames (`csh` uses kbd:[Esc]), and it lets you switch to the directory you were last in with `cd -`. It is also much easier to alter your prompt with `tcsh`. It makes life a lot easier.
The `csh` shell is okay, but `tcsh` does everything `csh` does and more.
It allows you to recall commands with the arrow keys and edit them.
It has tab-key completion of filenames (`csh` uses kbd:[Esc]), and it lets you switch to the directory you were last in with `cd -`.
It is also much easier to alter your prompt with `tcsh`.
It makes life a lot easier.
Here are the three steps for installing a new shell:
@ -330,7 +409,10 @@ Here are the three steps for installing a new shell:
[NOTE]
====
It can be dangerous to change `root`'s shell to something other than `sh` or `csh` on early versions of FreeBSD and many other versions of UNIX(R); you may not have a working shell when the system puts you into single user mode. The solution is to use `su -m` to become `root`, which will give you the `tcsh` as `root`, because the shell is part of the environment. You can make this permanent by adding it to your [.filename]#.tcshrc# as an alias with:
It can be dangerous to change `root`'s shell to something other than `sh` or `csh` on early versions of FreeBSD and many other versions of UNIX(R);
you may not have a working shell when the system puts you into single user mode.
The solution is to use `su -m` to become `root`, which will give you the `tcsh` as `root`, because the shell is part of the environment.
You can make this permanent by adding it to your [.filename]#.tcshrc# as an alias with:
[.programlisting]
....
@ -339,26 +421,41 @@ alias su su -m
====
When `tcsh` starts up, it will read the [.filename]#/etc/csh.cshrc# and [.filename]#/etc/csh.login# files, as does `csh`. It will also read [.filename]#.login# in your home directory and [.filename]#.cshrc# as well, unless you provide a [.filename]#.tcshrc#. This you can do by simply copying [.filename]#.cshrc# to [.filename]#.tcshrc#.
When `tcsh` starts up, it will read the [.filename]#/etc/csh.cshrc# and [.filename]#/etc/csh.login# files, as does `csh`.
It will also read [.filename]#.login# in your home directory and [.filename]#.cshrc# as well, unless you provide a [.filename]#.tcshrc#.
This you can do by simply copying [.filename]#.cshrc# to [.filename]#.tcshrc#.
Now that you have installed `tcsh`, you can adjust your prompt. You can find the details in the manual page for `tcsh`, but here is a line to put in your [.filename]#.tcshrc# that will tell you how many commands you have typed, what time it is, and what directory you are in. It also produces a `>` if you are an ordinary user and a # if you are `root`, but tsch will do that in any case:
Now that you have installed `tcsh`, you can adjust your prompt.
You can find the details in the manual page for `tcsh`, but here is a line to put in your [.filename]#.tcshrc# that will tell you how many commands you have typed, what time it is, and what directory you are in.
It also produces a `>` if you are an ordinary user and a # if you are `root`, but tsch will do that in any case:
set prompt = "%h %t %~ %# "
This should go in the same place as the existing set prompt line if there is one, or under "if($?prompt) then" if not. Comment out the old line; you can always switch back to it if you prefer it. Do not forget the spaces and quotes. You can get the [.filename]#.tcshrc# reread by typing `source .tcshrc`.
This should go in the same place as the existing set prompt line if there is one, or under "if($?prompt) then" if not.
Comment out the old line; you can always switch back to it if you prefer it.
Do not forget the spaces and quotes.
You can get the [.filename]#.tcshrc# reread by typing `source .tcshrc`.
You can get a listing of other environmental variables that have been set by typing `env` at the prompt. The result will show you your default editor, pager, and terminal type, among possibly many others. A useful command if you log in from a remote location and cannot run a program because the terminal is not capable is `setenv TERM vt100`.
You can get a listing of other environmental variables that have been set by typing `env` at the prompt.
The result will show you your default editor, pager, and terminal type, among possibly many others.
A useful command if you log in from a remote location and cannot run a program because the terminal is not capable is `setenv TERM vt100`.
[[other]]
== Other
As `root`, you can unmount the CDROM with `/sbin/umount /cdrom`, take it out of the drive, insert another one, and mount it with `/sbin/mount_cd9660 /dev/cd0a /cdrom` assuming cd0a is the device name for your CDROM drive. The most recent versions of FreeBSD let you mount the CDROM with just `/sbin/mount /cdrom`.
As `root`, you can unmount the CDROM with `/sbin/umount /cdrom`, take it out of the drive, insert another one, and mount it with `/sbin/mount_cd9660 /dev/cd0a /cdrom` assuming cd0a is the device name for your CDROM drive.
The most recent versions of FreeBSD let you mount the CDROM with just `/sbin/mount /cdrom`.
Using the live filesystem-the second of FreeBSD's CDROM disks-is useful if you have got limited space. What is on the live filesystem varies from release to release. You might try playing games from the CDROM. This involves using `lndir`, which gets installed with the X Window System, to tell the program(s) where to find the necessary files, because they are in [.filename]#/cdrom# instead of in [.filename]#/usr# and its subdirectories, which is where they are expected to be. Read `man lndir`.
Using the live filesystem-the second of FreeBSD's CDROM disks-is useful if you have got limited space.
What is on the live filesystem varies from release to release.
You might try playing games from the CDROM.
This involves using `lndir`, which gets installed with the X Window System, to tell the program(s) where to find the necessary files, because they are in [.filename]#/cdrom# instead of in [.filename]#/usr# and its subdirectories, which is where they are expected to be.
Read `man lndir`.
[[comments-welcome]]
== Comments Welcome
If you use this guide I would be interested in knowing where it was unclear and what was left out that you think should be included, and if it was helpful. My thanks to Eugene W. Stark, professor of computer science at SUNY-Stony Brook, and John Fieber for helpful comments.
If you use this guide I would be interested in knowing where it was unclear and what was left out that you think should be included, and if it was helpful.
My thanks to Eugene W. Stark, professor of computer science at SUNY-Stony Brook, and John Fieber for helpful comments.
Annelise Anderson, mailto:andrsn@andrsn.stanford.edu[andrsn@andrsn.stanford.edu]

204
documentation/content/en/articles/pam/_index.adoc
View file

@ -31,7 +31,9 @@ toc::[]
The Pluggable Authentication Modules (PAM) library is a generalized API for authentication-related services which allows a system administrator to add new authentication methods simply by installing new PAM modules, and to modify authentication policies by editing configuration files.
PAM was defined and developed in 1995 by Vipin Samar and Charlie Lai of Sun Microsystems, and has not changed much since. In 1997, the Open Group published the X/Open Single Sign-on (XSSO) preliminary specification, which standardized the PAM API and added extensions for single (or rather integrated) sign-on. At the time of this writing, this specification has not yet been adopted as a standard.
PAM was defined and developed in 1995 by Vipin Samar and Charlie Lai of Sun Microsystems, and has not changed much since.
In 1997, the Open Group published the X/Open Single Sign-on (XSSO) preliminary specification, which standardized the PAM API and added extensions for single (or rather integrated) sign-on.
At the time of this writing, this specification has not yet been adopted as a standard.
Although this article focuses primarily on FreeBSD 5.x, which uses OpenPAM, it should be equally applicable to FreeBSD 4.x, which uses Linux-PAM, and other operating systems such as Linux and Solaris(TM).
@ -41,47 +43,54 @@ Although this article focuses primarily on FreeBSD 5.x, which uses OpenPAM, it s
[[pam-definitions]]
=== Definitions
The terminology surrounding PAM is rather confused. Neither Samar and Lai's original paper nor the XSSO specification made any attempt at formally defining terms for the various actors and entities involved in PAM, and the terms that they do use (but do not define) are sometimes misleading and ambiguous. The first attempt at establishing a consistent and unambiguous terminology was a whitepaper written by Andrew G. Morgan (author of Linux-PAM) in 1999. While Morgan's choice of terminology was a huge leap forward, it is in this author's opinion by no means perfect. What follows is an attempt, heavily inspired by Morgan, to define precise and unambiguous terms for all actors and entities involved in PAM.
The terminology surrounding PAM is rather confused.
Neither Samar and Lai's original paper nor the XSSO specification made any attempt at formally defining terms for the various actors and entities involved in PAM, and the terms that they do use (but do not define) are sometimes misleading and ambiguous.
The first attempt at establishing a consistent and unambiguous terminology was a whitepaper written by Andrew G. Morgan (author of Linux-PAM) in 1999.
While Morgan's choice of terminology was a huge leap forward, it is in this author's opinion by no means perfect.
What follows is an attempt, heavily inspired by Morgan, to define precise and unambiguous terms for all actors and entities involved in PAM.
[.glosslist]
account::
The set of credentials the applicant is requesting from the arbitrator.
The set of credentials the applicant is requesting from the arbitrator.
applicant::
The user or entity requesting authentication.
The user or entity requesting authentication.
arbitrator::
The user or entity who has the privileges necessary to verify the applicant's credentials and the authority to grant or deny the request.
The user or entity who has the privileges necessary to verify the applicant's credentials and the authority to grant or deny the request.
chain::
A sequence of modules that will be invoked in response to a PAM request. The chain includes information about the order in which to invoke the modules, what arguments to pass to them, and how to interpret the results.
A sequence of modules that will be invoked in response to a PAM request.
The chain includes information about the order in which to invoke the modules, what arguments to pass to them, and how to interpret the results.
client::
The application responsible for initiating an authentication request on behalf of the applicant and for obtaining the necessary authentication information from him.
The application responsible for initiating an authentication request on behalf of the applicant and for obtaining the necessary authentication information from him.
facility::
One of the four basic groups of functionality provided by PAM: authentication, account management, session management and authentication token update.
One of the four basic groups of functionality provided by PAM: authentication, account management, session management and authentication token update.
module::
A collection of one or more related functions implementing a particular authentication facility, gathered into a single (normally dynamically loadable) binary file and identified by a single name.
A collection of one or more related functions implementing a particular authentication facility, gathered into a single (normally dynamically loadable) binary file and identified by a single name.
policy::
The complete set of configuration statements describing how to handle PAM requests for a particular service. A policy normally consists of four chains, one for each facility, though some services do not use all four facilities.
The complete set of configuration statements describing how to handle PAM requests for a particular service.
A policy normally consists of four chains, one for each facility, though some services do not use all four facilities.
server::
The application acting on behalf of the arbitrator to converse with the client, retrieve authentication information, verify the applicant's credentials and grant or deny requests.
The application acting on behalf of the arbitrator to converse with the client, retrieve authentication information, verify the applicant's credentials and grant or deny requests.
service::
A class of servers providing similar or related functionality and requiring similar authentication. PAM policies are defined on a per-service basis, so all servers that claim the same service name will be subject to the same policy.
A class of servers providing similar or related functionality and requiring similar authentication.
PAM policies are defined on a per-service basis, so all servers that claim the same service name will be subject to the same policy.
session::
The context within which service is rendered to the applicant by the server. One of PAM's four facilities, session management, is concerned exclusively with setting up and tearing down this context.
The context within which service is rendered to the applicant by the server.
One of PAM's four facilities, session management, is concerned exclusively with setting up and tearing down this context.
token::
A chunk of information associated with the account, such as a password or passphrase, which the applicant must provide to prove his identity.
A chunk of information associated with the account, such as a password or passphrase, which the applicant must provide to prove his identity.
transaction::
A sequence of requests from the same applicant to the same instance of the same server, beginning with authentication and session set-up and ending with session tear-down.
A sequence of requests from the same applicant to the same instance of the same server, beginning with authentication and session set-up and ending with session tear-down.
[[pam-usage-examples]]
=== Usage Examples
@ -114,7 +123,8 @@ root
==== Client and Server Are Separate
The example below shows `eve` try to initiate an man:ssh[1] connection to `login.example.com`, ask to log in as `bob`, and succeed. Bob should have chosen a better password!
The example below shows `eve` try to initiate an man:ssh[1] connection to `login.example.com`, ask to log in as `bob`, and succeed.
Bob should have chosen a better password!
[source,shell]
....
@ -168,80 +178,103 @@ sshd password required pam_permit.so
The PAM API offers six different authentication primitives grouped in four facilities, which are described below.
`auth`::
_Authentication._ This facility concerns itself with authenticating the applicant and establishing the account credentials. It provides two primitives:
_Authentication._ This facility concerns itself with authenticating the applicant and establishing the account credentials.
It provides two primitives:
** man:pam_authenticate[3] authenticates the applicant, usually by requesting an authentication token and comparing it with a value stored in a database or obtained from an authentication server.
** man:pam_setcred[3] establishes account credentials such as user ID, group membership and resource limits.
`account`::
_Account management._ This facility handles non-authentication-related issues of account availability, such as access restrictions based on the time of day or the server's work load. It provides a single primitive:
_Account management._ This facility handles non-authentication-related issues of account availability, such as access restrictions based on the time of day or the server's work load.
It provides a single primitive:
** man:pam_acct_mgmt[3] verifies that the requested account is available.
`session`::
_Session management._ This facility handles tasks associated with session set-up and tear-down, such as login accounting. It provides two primitives:
_Session management._ This facility handles tasks associated with session set-up and tear-down, such as login accounting.
It provides two primitives:
** man:pam_open_session[3] performs tasks associated with session set-up: add an entry in the [.filename]#utmp# and [.filename]#wtmp# databases, start an SSH agent, etc.
** man:pam_close_session[3] performs tasks associated with session tear-down: add an entry in the [.filename]#utmp# and [.filename]#wtmp# databases, stop the SSH agent, etc.
`password`::
_Password management._ This facility is used to change the authentication token associated with an account, either because it has expired or because the user wishes to change it. It provides a single primitive:
_Password management._ This facility is used to change the authentication token associated with an account, either because it has expired or because the user wishes to change it.
It provides a single primitive:
** man:pam_chauthtok[3] changes the authentication token, optionally verifying that it is sufficiently hard to guess, has not been used previously, etc.
[[pam-modules]]
=== Modules
Modules are a very central concept in PAM; after all, they are the "M" in "PAM". A PAM module is a self-contained piece of program code that implements the primitives in one or more facilities for one particular mechanism; possible mechanisms for the authentication facility, for instance, include the UNIX(R) password database, NIS, LDAP and Radius.
Modules are a very central concept in PAM; after all, they are the "M" in "PAM".
A PAM module is a self-contained piece of program code that implements the primitives in one or more facilities for one particular mechanism;
possible mechanisms for the authentication facility, for instance, include the UNIX(R) password database, NIS, LDAP and Radius.
[[pam-module-naming]]
==== Module Naming
FreeBSD implements each mechanism in a single module, named `pam_mechanism.so` (for instance, `pam_unix.so` for the UNIX(R) mechanism.) Other implementations sometimes have separate modules for separate facilities, and include the facility name as well as the mechanism name in the module name. To name one example, Solaris(TM) has a `pam_dial_auth.so.1` module which is commonly used to authenticate dialup users.
FreeBSD implements each mechanism in a single module, named `pam_mechanism.so` (for instance, `pam_unix.so` for the UNIX(R) mechanism.)
Other implementations sometimes have separate modules for separate facilities, and include the facility name as well as the mechanism name in the module name.
To name one example, Solaris(TM) has a `pam_dial_auth.so.1` module which is commonly used to authenticate dialup users.
[[pam-module-versioning]]
==== Module Versioning
FreeBSD's original PAM implementation, based on Linux-PAM, did not use version numbers for PAM modules. This would commonly cause problems with legacy applications, which might be linked against older versions of the system libraries, as there was no way to load a matching version of the required modules.
FreeBSD's original PAM implementation, based on Linux-PAM, did not use version numbers for PAM modules.
This would commonly cause problems with legacy applications, which might be linked against older versions of the system libraries, as there was no way to load a matching version of the required modules.
OpenPAM, on the other hand, looks for modules that have the same version number as the PAM library (currently 2), and only falls back to an unversioned module if no versioned module could be loaded. Thus legacy modules can be provided for legacy applications, while allowing new (or newly built) applications to take advantage of the most recent modules.
OpenPAM, on the other hand, looks for modules that have the same version number as the PAM library (currently 2), and only falls back to an unversioned module if no versioned module could be loaded.
Thus legacy modules can be provided for legacy applications, while allowing new (or newly built) applications to take advantage of the most recent modules.
Although Solaris(TM) PAM modules commonly have a version number, they are not truly versioned, because the number is a part of the module name and must be included in the configuration.
[[pam-chains-policies]]
=== Chains and Policies
When a server initiates a PAM transaction, the PAM library tries to load a policy for the service specified in the man:pam_start[3] call. The policy specifies how authentication requests should be processed, and is defined in a configuration file. This is the other central concept in PAM: the possibility for the admin to tune the system security policy (in the wider sense of the word) simply by editing a text file.
When a server initiates a PAM transaction, the PAM library tries to load a policy for the service specified in the man:pam_start[3] call.
The policy specifies how authentication requests should be processed, and is defined in a configuration file.
This is the other central concept in PAM: the possibility for the admin to tune the system security policy (in the wider sense of the word) simply by editing a text file.
A policy consists of four chains, one for each of the four PAM facilities. Each chain is a sequence of configuration statements, each specifying a module to invoke, some (optional) parameters to pass to the module, and a control flag that describes how to interpret the return code from the module.
A policy consists of four chains, one for each of the four PAM facilities.
Each chain is a sequence of configuration statements, each specifying a module to invoke, some (optional) parameters to pass to the module, and a control flag that describes how to interpret the return code from the module.
Understanding the control flags is essential to understanding PAM configuration files. There are four different control flags:
Understanding the control flags is essential to understanding PAM configuration files.
There are four different control flags:
`binding`::
If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the rest of the chain is executed, but the request is ultimately denied.
If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted.
If the module fails, the rest of the chain is executed, but the request is ultimately denied.
+
This control flag was introduced by Sun in Solaris(TM) 9 (SunOS(TM) 5.9), and is also supported by OpenPAM.
`required`::
If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the rest of the chain is also executed, but the request is ultimately denied.
If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails.
If the module fails, the rest of the chain is also executed, but the request is ultimately denied.
`requisite`::
If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the chain is immediately terminated and the request is denied.
If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails.
If the module fails, the chain is immediately terminated and the request is denied.
`sufficient`::
If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the module is ignored and the rest of the chain is executed.
If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted.
If the module fails, the module is ignored and the rest of the chain is executed.
+
As the semantics of this flag may be somewhat confusing, especially when it is used for the last module in a chain, it is recommended that the `binding` control flag be used instead if the implementation supports it.
`optional`::
The module is executed, but its result is ignored. If all modules in a chain are marked `optional`, all requests will always be granted.
The module is executed, but its result is ignored.
If all modules in a chain are marked `optional`, all requests will always be granted.
When a server invokes one of the six PAM primitives, PAM retrieves the chain for the facility the primitive belongs to, and invokes each of the modules listed in the chain, in the order they are listed, until it reaches the end, or determines that no further processing is necessary (either because a `binding` or `sufficient` module succeeded, or because a `requisite` module failed.) The request is granted if and only if at least one module was invoked, and all non-optional modules succeeded.
When a server invokes one of the six PAM primitives, PAM retrieves the chain for the facility the primitive belongs to, and invokes each of the modules listed in the chain, in the order they are listed, until it reaches the end, or determines that no further processing is necessary (either because a `binding` or `sufficient` module succeeded, or because a `requisite` module failed.)
The request is granted if and only if at least one module was invoked, and all non-optional modules succeeded.
Note that it is possible, though not very common, to have the same module listed several times in the same chain. For instance, a module that looks up user names and passwords in a directory server could be invoked multiple times with different parameters specifying different directory servers to contact. PAM treat different occurrences of the same module in the same chain as different, unrelated modules.
Note that it is possible, though not very common, to have the same module listed several times in the same chain.
For instance, a module that looks up user names and passwords in a directory server could be invoked multiple times with different parameters specifying different directory servers to contact.
PAM treat different occurrences of the same module in the same chain as different, unrelated modules.
[[pam-transactions]]
=== Transactions
The lifecycle of a typical PAM transaction is described below. Note that if any of these steps fails, the server should report a suitable error message to the client and abort the transaction.
The lifecycle of a typical PAM transaction is described below.
Note that if any of these steps fails, the server should report a suitable error message to the client and abort the transaction.
. If necessary, the server obtains arbitrator credentials through a mechanism independent of PAM-most commonly by virtue of having been started by `root`, or of being setuid `root`.
. The server calls man:pam_start[3] to initialize the PAM library and specify its service name and the target account, and register a suitable conversation function.
@ -264,30 +297,39 @@ The lifecycle of a typical PAM transaction is described below. Note that if any
[[pam-config-pam.conf]]
==== The [.filename]#/etc/pam.conf#
The traditional PAM policy file is [.filename]#/etc/pam.conf#. This file contains all the PAM policies for your system. Each line of the file describes one step in a chain, as shown below:
The traditional PAM policy file is [.filename]#/etc/pam.conf#.
This file contains all the PAM policies for your system.
Each line of the file describes one step in a chain, as shown below:
[.programlisting]
....
login auth required pam_nologin.so no_warn
....
The fields are, in order: service name, facility name, control flag, module name, and module arguments. Any additional fields are interpreted as additional module arguments.
The fields are, in order: service name, facility name, control flag, module name, and module arguments.
Any additional fields are interpreted as additional module arguments.
A separate chain is constructed for each service / facility pair, so while the order in which lines for the same service and facility appear is significant, the order in which the individual services and facilities are listed is not. The examples in the original PAM paper grouped configuration lines by facility, and the Solaris(TM) stock [.filename]#pam.conf# still does that, but FreeBSD's stock configuration groups configuration lines by service. Either way is fine; either way makes equal sense.
A separate chain is constructed for each service / facility pair, so while the order in which lines for the same service and facility appear is significant, the order in which the individual services and facilities are listed is not.
The examples in the original PAM paper grouped configuration lines by facility, and the Solaris(TM) stock [.filename]#pam.conf# still does that, but FreeBSD's stock configuration groups configuration lines by service.
Either way is fine; either way makes equal sense.
[[pam-config-pam.d]]
==== The [.filename]#/etc/pam.d#
OpenPAM and Linux-PAM support an alternate configuration mechanism, which is the preferred mechanism in FreeBSD. In this scheme, each policy is contained in a separate file bearing the name of the service it applies to. These files are stored in [.filename]#/etc/pam.d/#.
OpenPAM and Linux-PAM support an alternate configuration mechanism, which is the preferred mechanism in FreeBSD.
In this scheme, each policy is contained in a separate file bearing the name of the service it applies to.
These files are stored in [.filename]#/etc/pam.d/#.
These per-service policy files have only four fields instead of [.filename]#pam.conf#'s five: the service name field is omitted. Thus, instead of the sample [.filename]#pam.conf# line from the previous section, one would have the following line in [.filename]#/etc/pam.d/login#:
These per-service policy files have only four fields instead of [.filename]#pam.conf#'s five: the service name field is omitted.
Thus, instead of the sample [.filename]#pam.conf# line from the previous section, one would have the following line in [.filename]#/etc/pam.d/login#:
[.programlisting]
....
auth required pam_nologin.so no_warn
....
As a consequence of this simplified syntax, it is possible to use the same policy for multiple services by linking each service name to a same policy file. For instance, to use the same policy for the `su` and `sudo` services, one could do as follows:
As a consequence of this simplified syntax, it is possible to use the same policy for multiple services by linking each service name to a same policy file.
For instance, to use the same policy for the `su` and `sudo` services, one could do as follows:
[source,shell]
....
@ -302,7 +344,8 @@ Since each service's policy is stored in a separate file, the [.filename]#pam.d#
[[pam-config-file-order]]
==== The Policy Search Order
As we have seen above, PAM policies can be found in a number of places. What happens if policies for the same service exist in multiple places?
As we have seen above, PAM policies can be found in a number of places.
What happens if policies for the same service exist in multiple places?
It is essential to understand that PAM's configuration system is centered on chains.
@ -311,22 +354,28 @@ It is essential to understand that PAM's configuration system is centered on cha
As explained in <<pam-config-file>>, each line in [.filename]#/etc/pam.conf# consists of four or more fields: the service name, the facility name, the control flag, the module name, and zero or more module arguments.
The service name is generally (though not always) the name of the application the statement applies to. If you are unsure, refer to the individual application's documentation to determine what service name it uses.
The service name is generally (though not always) the name of the application the statement applies to.
If you are unsure, refer to the individual application's documentation to determine what service name it uses.
Note that if you use [.filename]#/etc/pam.d/# instead of [.filename]#/etc/pam.conf#, the service name is specified by the name of the policy file, and omitted from the actual configuration lines, which then start with the facility name.
The facility is one of the four facility keywords described in <<pam-facilities-primitives>>.
Likewise, the control flag is one of the four keywords described in <<pam-chains-policies>>, describing how to interpret the return code from the module. Linux-PAM supports an alternate syntax that lets you specify the action to associate with each possible return code, but this should be avoided as it is non-standard and closely tied in with the way Linux-PAM dispatches service calls (which differs greatly from the way Solaris(TM) and OpenPAM do it.) Unsurprisingly, OpenPAM does not support this syntax.
Likewise, the control flag is one of the four keywords described in <<pam-chains-policies>>, describing how to interpret the return code from the module.
Linux-PAM supports an alternate syntax that lets you specify the action to associate with each possible return code, but this should be avoided as it is non-standard and closely tied in with the way Linux-PAM dispatches service calls (which differs greatly from the way Solaris(TM) and OpenPAM do it.)
Unsurprisingly, OpenPAM does not support this syntax.
[[pam-policies]]
=== Policies
To configure PAM correctly, it is essential to understand how policies are interpreted.
When an application calls man:pam_start[3], the PAM library loads the policy for the specified service and constructs four module chains (one for each facility.) If one or more of these chains are empty, the corresponding chains from the policy for the `other` service are substituted.
When an application calls man:pam_start[3], the PAM library loads the policy for the specified service and constructs four module chains (one for each facility.)
If one or more of these chains are empty, the corresponding chains from the policy for the `other` service are substituted.
When the application later calls one of the six PAM primitives, the PAM library retrieves the chain for the corresponding facility and calls the appropriate service function in each module listed in the chain, in the order in which they were listed in the configuration. After each call to a service function, the module type and the error code returned by the service function are used to determine what happens next. With a few exceptions, which we discuss below, the following table applies:
When the application later calls one of the six PAM primitives, the PAM library retrieves the chain for the corresponding facility and calls the appropriate service function in each module listed in the chain, in the order in which they were listed in the configuration.
After each call to a service function, the module type and the error code returned by the service function are used to determine what happens next.
With a few exceptions, which we discuss below, the following table applies:
.PAM Chain Execution Summary
[cols="1,1,1,1", options="header"]
@ -362,7 +411,8 @@ When the application later calls one of the six PAM primitives, the PAM library
|-
|===
If `fail` is true at the end of a chain, or when a "break" is reached, the dispatcher returns the error code returned by the first module that failed. Otherwise, it returns `PAM_SUCCESS`.
If `fail` is true at the end of a chain, or when a "break" is reached, the dispatcher returns the error code returned by the first module that failed.
Otherwise, it returns `PAM_SUCCESS`.
The first exception of note is that the error code `PAM_NEW_AUTHTOK_REQD` is treated like a success, except that if no module failed, and at least one module returned `PAM_NEW_AUTHTOK_REQD`, the dispatcher will return `PAM_NEW_AUTHTOK_REQD`.
@ -376,17 +426,20 @@ The third and final exception is that man:pam_chauthtok[3] runs the entire chain
[[pam-modules-deny]]
=== man:pam_deny[8]
The man:pam_deny[8] module is one of the simplest modules available; it responds to any request with `PAM_AUTH_ERR`. It is useful for quickly disabling a service (add it to the top of every chain), or for terminating chains of `sufficient` modules.
The man:pam_deny[8] module is one of the simplest modules available; it responds to any request with `PAM_AUTH_ERR`.
It is useful for quickly disabling a service (add it to the top of every chain), or for terminating chains of `sufficient` modules.
[[pam-modules-echo]]
=== man:pam_echo[8]
The man:pam_echo[8] module simply passes its arguments to the conversation function as a `PAM_TEXT_INFO` message. It is mostly useful for debugging, but can also serve to display messages such as "Unauthorized access will be prosecuted" before starting the authentication procedure.
The man:pam_echo[8] module simply passes its arguments to the conversation function as a `PAM_TEXT_INFO` message.
It is mostly useful for debugging, but can also serve to display messages such as "Unauthorized access will be prosecuted" before starting the authentication procedure.
[[pam-modules-exec]]
=== man:pam_exec[8]
The man:pam_exec[8] module takes its first argument to be the name of a program to execute, and the remaining arguments are passed to that program as command-line arguments. One possible application is to use it to run a program at login time which mounts the user's home directory.
The man:pam_exec[8] module takes its first argument to be the name of a program to execute, and the remaining arguments are passed to that program as command-line arguments.
One possible application is to use it to run a program at login time which mounts the user's home directory.
[[pam-modules-ftpusers]]
=== man:pam_ftpusers[8]
@ -396,12 +449,15 @@ The man:pam_ftpusers[8] module
[[pam-modules-group]]
=== man:pam_group[8]
The man:pam_group[8] module accepts or rejects applicants on the basis of their membership in a particular file group (normally `wheel` for man:su[1]). It is primarily intended for maintaining the traditional behavior of BSD man:su[1], but has many other uses, such as excluding certain groups of users from a particular service.
The man:pam_group[8] module accepts or rejects applicants on the basis of their membership in a particular file group (normally `wheel` for man:su[1]).
It is primarily intended for maintaining the traditional behavior of BSD man:su[1], but has many other uses, such as excluding certain groups of users from a particular service.
[[pam-modules-guest]]
=== man:pam_guest[8]
The man:pam_guest[8] module allows guest logins using fixed login names. Various requirements can be placed on the password, but the default behavior is to allow any password as long as the login name is that of a guest account. The man:pam_guest[8] module can easily be used to implement anonymous FTP logins.
The man:pam_guest[8] module allows guest logins using fixed login names.
Various requirements can be placed on the password, but the default behavior is to allow any password as long as the login name is that of a guest account.
The man:pam_guest[8] module can easily be used to implement anonymous FTP logins.
[[pam-modules-krb5]]
=== man:pam_krb5[8]
@ -426,17 +482,22 @@ The man:pam_login_access[8] module provides an implementation of the account man
[[pam-modules-nologin]]
=== man:pam_nologin[8]
The man:pam_nologin[8] module refuses non-root logins when [.filename]#/var/run/nologin# exists. This file is normally created by man:shutdown[8] when less than five minutes remain until the scheduled shutdown time.
The man:pam_nologin[8] module refuses non-root logins when [.filename]#/var/run/nologin# exists.
This file is normally created by man:shutdown[8] when less than five minutes remain until the scheduled shutdown time.
[[pam-modules-opie]]
=== man:pam_opie[8]
The man:pam_opie[8] module implements the man:opie[4] authentication method. The man:opie[4] system is a challenge-response mechanism where the response to each challenge is a direct function of the challenge and a passphrase, so the response can be easily computed "just in time" by anyone possessing the passphrase, eliminating the need for password lists. Moreover, since man:opie[4] never reuses a challenge that has been correctly answered, it is not vulnerable to replay attacks.
The man:pam_opie[8] module implements the man:opie[4] authentication method.
The man:opie[4] system is a challenge-response mechanism where the response to each challenge is a direct function of the challenge and a passphrase, so the response can be easily computed "just in time" by anyone possessing the passphrase, eliminating the need for password lists.
Moreover, since man:opie[4] never reuses a challenge that has been correctly answered, it is not vulnerable to replay attacks.
[[pam-modules-opieaccess]]
=== man:pam_opieaccess[8]
The man:pam_opieaccess[8] module is a companion module to man:pam_opie[8]. Its purpose is to enforce the restrictions codified in man:opieaccess[5], which regulate the conditions under which a user who would normally authenticate herself using man:opie[4] is allowed to use alternate methods. This is most often used to prohibit the use of password authentication from untrusted hosts.
The man:pam_opieaccess[8] module is a companion module to man:pam_opie[8].
Its purpose is to enforce the restrictions codified in man:opieaccess[5], which regulate the conditions under which a user who would normally authenticate herself using man:opie[4] is allowed to use alternate methods.
This is most often used to prohibit the use of password authentication from untrusted hosts.
In order to be effective, the man:pam_opieaccess[8] module must be listed as `requisite` immediately after a `sufficient` entry for man:pam_opie[8], and before any other modules, in the `auth` chain.
@ -448,7 +509,8 @@ The man:pam_passwdqc[8] module
[[pam-modules-permit]]
=== man:pam_permit[8]
The man:pam_permit[8] module is one of the simplest modules available; it responds to any request with `PAM_SUCCESS`. It is useful as a placeholder for services where one or more chains would otherwise be empty.
The man:pam_permit[8] module is one of the simplest modules available; it responds to any request with `PAM_SUCCESS`.
It is useful as a placeholder for services where one or more chains would otherwise be empty.
[[pam-modules-radius]]
=== man:pam_radius[8]
@ -463,7 +525,8 @@ The man:pam_rhosts[8] module
[[pam-modules-rootok]]
=== man:pam_rootok[8]
The man:pam_rootok[8] module reports success if and only if the real user id of the process calling it (which is assumed to be run by the applicant) is 0. This is useful for non-networked services such as man:su[1] or man:passwd[1], to which the `root` should have automatic access.
The man:pam_rootok[8] module reports success if and only if the real user id of the process calling it (which is assumed to be run by the applicant) is 0.
This is useful for non-networked services such as man:su[1] or man:passwd[1], to which the `root` should have automatic access.
[[pam-modules-securetty]]
=== man:pam_securetty[8]
@ -473,12 +536,16 @@ The man:pam_securetty[8] module
[[pam-modules-self]]
=== man:pam_self[8]
The man:pam_self[8] module reports success if and only if the names of the applicant matches that of the target account. It is most useful for non-networked services such as man:su[1], where the identity of the applicant can be easily verified.
The man:pam_self[8] module reports success if and only if the names of the applicant matches that of the target account.
It is most useful for non-networked services such as man:su[1], where the identity of the applicant can be easily verified.
[[pam-modules-ssh]]
=== man:pam_ssh[8]
The man:pam_ssh[8] module provides both authentication and session services. The authentication service allows users who have passphrase-protected SSH secret keys in their [.filename]#~/.ssh# directory to authenticate themselves by typing their passphrase. The session service starts man:ssh-agent[1] and preloads it with the keys that were decrypted in the authentication phase. This feature is particularly useful for local logins, whether in X (using man:xdm[1] or another PAM-aware X login manager) or at the console.
The man:pam_ssh[8] module provides both authentication and session services.
The authentication service allows users who have passphrase-protected SSH secret keys in their [.filename]#~/.ssh# directory to authenticate themselves by typing their passphrase.
The session service starts man:ssh-agent[1] and preloads it with the keys that were decrypted in the authentication phase.
This feature is particularly useful for local logins, whether in X (using man:xdm[1] or another PAM-aware X login manager) or at the console.
[[pam-modules-tacplus]]
=== man:pam_tacplus[8]
@ -488,7 +555,9 @@ The man:pam_tacplus[8] module
[[pam-modules-unix]]
=== man:pam_unix[8]
The man:pam_unix[8] module implements traditional UNIX(R) password authentication, using man:getpwnam[3] to obtain the target account's password and compare it with the one provided by the applicant. It also provides account management services (enforcing account and password expiration times) and password-changing services. This is probably the single most useful module, as the great majority of admins will want to maintain historical behavior for at least some services.
The man:pam_unix[8] module implements traditional UNIX(R) password authentication, using man:getpwnam[3] to obtain the target account's password and compare it with the one provided by the applicant.
It also provides account management services (enforcing account and password expiration times) and password-changing services.
This is probably the single most useful module, as the great majority of admins will want to maintain historical behavior for at least some services.
[[pam-appl-prog]]
== PAM Application Programming
@ -506,7 +575,11 @@ This section has not yet been written.
[[pam-sample-appl]]
== Sample PAM Application
The following is a minimal implementation of man:su[1] using PAM. Note that it uses the OpenPAM-specific man:openpam_ttyconv[3] conversation function, which is prototyped in [.filename]#security/openpam.h#. If you wish build this application on a system with a different PAM library, you will have to provide your own conversation function. A robust conversation function is surprisingly difficult to implement; the one presented in <<pam-sample-conv>> is a good starting point, but should not be used in real-world applications.
The following is a minimal implementation of man:su[1] using PAM.
Note that it uses the OpenPAM-specific man:openpam_ttyconv[3] conversation function, which is prototyped in [.filename]#security/openpam.h#.
If you wish build this application on a system with a different PAM library, you will have to provide your own conversation function.
A robust conversation function is surprisingly difficult to implement;
the one presented in <<pam-sample-conv>> is a good starting point, but should not be used in real-world applications.
[.programlisting]
....
@ -519,7 +592,8 @@ include::static/source/articles/pam/su.c[]
[[pam-sample-module]]
== Sample PAM Module
The following is a minimal implementation of man:pam_unix[8], offering only authentication services. It should build and run with most PAM implementations, but takes advantage of OpenPAM extensions if available: note the use of man:pam_get_authtok[3], which enormously simplifies prompting the user for a password.
The following is a minimal implementation of man:pam_unix[8], offering only authentication services.
It should build and run with most PAM implementations, but takes advantage of OpenPAM extensions if available: note the use of man:pam_get_authtok[3], which enormously simplifies prompting the user for a password.
[.programlisting]
....
@ -532,7 +606,9 @@ include::static/source/articles/pam/pam_unix.c[]
[[pam-sample-conv]]
== Sample PAM Conversation Function
The conversation function presented below is a greatly simplified version of OpenPAM's man:openpam_ttyconv[3]. It is fully functional, and should give the reader a good idea of how a conversation function should behave, but it is far too simple for real-world use. Even if you are not using OpenPAM, feel free to download the source code and adapt man:openpam_ttyconv[3] to your uses; we believe it to be as robust as a tty-oriented conversation function can reasonably get.
The conversation function presented below is a greatly simplified version of OpenPAM's man:openpam_ttyconv[3].
It is fully functional, and should give the reader a good idea of how a conversation function should behave, but it is far too simple for real-world use.
Even if you are not using OpenPAM, feel free to download the source code and adapt man:openpam_ttyconv[3] to your uses; we believe it to be as robust as a tty-oriented conversation function can reasonably get.
[.programlisting]
....

3
documentation/content/en/articles/pgpkeys/_index.adoc
View file

@ -20,7 +20,8 @@ include::shared/en/teams.adoc[lines=16..-1]
toc::[]
These OpenPGP keys can be used to verify a signature or send encrypted email to `FreeBSD.org` officers or developers. The complete keyring can be downloaded at link:https://www.FreeBSD.org/doc/pgpkeyring.txt[https://www.FreeBSD.org/doc/pgpkeyring.txt].
These OpenPGP keys can be used to verify a signature or send encrypted email to `FreeBSD.org` officers or developers.
The complete keyring can be downloaded at link:https://www.FreeBSD.org/doc/pgpkeyring.txt[https://www.FreeBSD.org/doc/pgpkeyring.txt].
////
Do not edit this file except as instructed by the addkey.sh script.

48
documentation/content/en/articles/port-mentor-guidelines/_index.adoc
View file

@ -25,7 +25,8 @@ toc::[]
[[port-mentor.guidelines]]
== Guideline for Mentor/Mentee Relationships
This section is intended to help demystify the mentoring process, as well as a way to openly promote a constructive discussion to adapt and grow the guidelines. In our lives we have too many rules; we are not a government organization that inflicts regulation, but rather a collective of like minded individuals working toward a common goal, maintaining the quality assurance of the product we call the Ports Tree.
This section is intended to help demystify the mentoring process, as well as a way to openly promote a constructive discussion to adapt and grow the guidelines.
In our lives we have too many rules; we are not a government organization that inflicts regulation, but rather a collective of like minded individuals working toward a common goal, maintaining the quality assurance of the product we call the Ports Tree.
[[why.mentor]]
=== Why Mentor?
@ -56,9 +57,11 @@ Reasons for sole mentorship:
We expect mentors to review and test-build all proposed patches, at least for an initial period lasting more than a week or two.
We expect that mentors should take responsibility for the actions of their mentee. A mentor should follow up with all commits the mentee makes, both approved and implicit.
We expect that mentors should take responsibility for the actions of their mentee.
A mentor should follow up with all commits the mentee makes, both approved and implicit.
We expect mentors to make sure their mentees read the link:{porters-handbook}[Porter's Handbook], the link:{pr-guidelines}[PR handling guide], and the link:{committers-guide}[Committer's Guide]. While it is not necessary to memorize all the details, every committer needs to have an overview of these things to be an effective part of the community (and avoid as many rookie mistakes as possible).
We expect mentors to make sure their mentees read the link:{porters-handbook}[Porter's Handbook], the link:{pr-guidelines}[PR handling guide], and the link:{committers-guide}[Committer's Guide].
While it is not necessary to memorize all the details, every committer needs to have an overview of these things to be an effective part of the community (and avoid as many rookie mistakes as possible).
[[mentees]]
=== Selecting a Mentee
@ -67,31 +70,52 @@ There is no defined rule for what makes a candidate ready; it can be a combinati
A candidate should have almost no timeouts, be responsive to requests, and generally helpful in supporting their ports.
There must be a history of commitment, as it is widely understood that training a committer requires time and effort. If somebody has been around longer, and spent the time observing how things are done, there is some anticipation of accumulated knowledge. All too often we have seen a maintainer submit a few PRs, show up in IRC and ask when they will be given a commit bit.
There must be a history of commitment, as it is widely understood that training a committer requires time and effort.
If somebody has been around longer, and spent the time observing how things are done, there is some anticipation of accumulated knowledge.
All too often we have seen a maintainer submit a few PRs, show up in IRC and ask when they will be given a commit bit.
Being subscribed to, and following the mailing lists is very beneficial. There is no real expectation that submitting posts on the lists will make somebody a committer, but it demonstrates a commitment. Some mails offer insights into the knowledge of a candidate as well how they interact with others. Similarly participating in IRC can give somebody a higher profile.
Being subscribed to, and following the mailing lists is very beneficial.
There is no real expectation that submitting posts on the lists will make somebody a committer, but it demonstrates a commitment.
Some mails offer insights into the knowledge of a candidate as well how they interact with others.
Similarly participating in IRC can give somebody a higher profile.
Ask six different committers how many PRs a maintainer should submit prior to being nominated, and you will get six different answers. Ask those same individuals how long somebody should have been participating, same dilemma. How many ports should they have at a minimum? Now we have a bikeshed! Some things are just hard to quantify, a mentor will just have to use their best judgement, and hope that portmgr agrees.
Ask six different committers how many PRs a maintainer should submit prior to being nominated, and you will get six different answers.
Ask those same individuals how long somebody should have been participating, same dilemma.
How many ports should they have at a minimum? Now we have a bikeshed! Some things are just hard to quantify, a mentor will just have to use their best judgement, and hope that portmgr agrees.
[[mentorship.duration]]
=== Mentorship Duration
As the trust level develops and grows, the mentee may be granted "implicit" commit rights. This can include trivial changes to a [.filename]#Makefile#, [.filename]#pkg-descr# etc. Similarly, it may include `PORTVERSION` updates that do not include `plist` changes. Other circumstances may be formulated at the discretion of the Mentor. However, during the period of mentorship, a port version bump that affects dependent ports should be checked by a mentor.
As the trust level develops and grows, the mentee may be granted "implicit" commit rights.
This can include trivial changes to a [.filename]#Makefile#, [.filename]#pkg-descr# etc.
Similarly, it may include `PORTVERSION` updates that do not include `plist` changes.
Other circumstances may be formulated at the discretion of the Mentor.
However, during the period of mentorship, a port version bump that affects dependent ports should be checked by a mentor.
Just as we are all varied individuals, each mentee has different learning curves, time commitments, and other influencing factors that will contribute to the time required before they can "fly solo". Empirically, a mentee should be observed for at least 3 months. 90-100 commits is another target that a mentor could use before releasing a mentee. Other factors to consider prior releasing a mentee are the number of mistakes they may have made, QATs received etc. If they are still making rookie mistakes, they still require mentor guidance.
Just as we are all varied individuals, each mentee has different learning curves, time commitments, and other influencing factors that will contribute to the time required before they can "fly solo".
Empirically, a mentee should be observed for at least 3 months.
90-100 commits is another target that a mentor could use before releasing a mentee.
Other factors to consider prior releasing a mentee are the number of mistakes they may have made, QATs received etc.
If they are still making rookie mistakes, they still require mentor guidance.
[[mentor.comentor.debate]]
=== Mentor/Co-Mentor Debate
When a request gets to portmgr, it usually reads as, "I propose 'foo' for a ports commit bit, I will co-mentor with 'bar'". Proposal received, voted, and carried.
When a request gets to portmgr, it usually reads as, "I propose 'foo' for a ports commit bit, I will co-mentor with 'bar'".
Proposal received, voted, and carried.
The mentor is the primary point of contact or the "first among equals", the co-mentor is the backup.
Some reprobate, whose name shall be withheld, made the https://lists.freebsd.org/pipermail/cvs-ports/2007-September/134614.html[first recorded co-mentor commit]. Similar co-mentor commits have also been spotted in the src tree. Does this make it right? Does this make it wrong? It seems to be part of the evolution of how things are done.
Some reprobate, whose name shall be withheld, made the https://lists.freebsd.org/pipermail/cvs-ports/2007-September/134614.html[first recorded co-mentor commit].
Similar co-mentor commits have also been spotted in the src tree.
Does this make it right? Does this make it wrong? It seems to be part of the evolution of how things are done.
[[mentee.expectations]]
=== Expectations
We expect mentees to be prepared for constructive criticism from the community. There's still a lot of "lore" that is not written down. Responding well to constructive criticism is what we hope we are selecting for by first reviewing their existing contributions on IRC and mailing lists.
We expect mentees to be prepared for constructive criticism from the community.
There's still a lot of "lore" that is not written down.
Responding well to constructive criticism is what we hope we are selecting for by first reviewing their existing contributions on IRC and mailing lists.
We warn mentees that some of the criticism they receive may be less "constructive" than others, (whether through language communication problems, or excessive nit-picking), and that dealing with this gracefully is just part of being in a large community. In case of specific problems with specific people, or any questions, we hope that they will approach a portmgr member on IRC or by email.
We warn mentees that some of the criticism they receive may be less "constructive" than others, (whether through language communication problems, or excessive nit-picking), and that dealing with this gracefully is just part of being in a large community.
In case of specific problems with specific people, or any questions, we hope that they will approach a portmgr member on IRC or by email.

70
documentation/content/en/articles/pr-guidelines/_index.adoc
View file

@ -23,7 +23,8 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
These guidelines describe recommended handling practices for FreeBSD Problem Reports (PRs). Whilst developed for the FreeBSD PR Database Maintenance Team mailto:freebsd-bugbusters@FreeBSD.org[freebsd-bugbusters@FreeBSD.org], these guidelines should be followed by anyone working with FreeBSD PRs.
These guidelines describe recommended handling practices for FreeBSD Problem Reports (PRs).
Whilst developed for the FreeBSD PR Database Maintenance Team mailto:freebsd-bugbusters@FreeBSD.org[freebsd-bugbusters@FreeBSD.org], these guidelines should be followed by anyone working with FreeBSD PRs.
'''
@ -32,9 +33,11 @@ toc::[]
[[intro]]
== Introduction
Bugzilla is an issue management system used by the FreeBSD Project. As accurate tracking of outstanding software defects is important to FreeBSD's quality, the correct use of the software is essential to the forward progress of the Project.
Bugzilla is an issue management system used by the FreeBSD Project.
As accurate tracking of outstanding software defects is important to FreeBSD's quality, the correct use of the software is essential to the forward progress of the Project.
Access to Bugzilla is available to the entire FreeBSD community. In order to maintain consistency within the database and provide a consistent user experience, guidelines have been established covering common aspects of bug management such as presenting followup, handling close requests, and so forth.
Access to Bugzilla is available to the entire FreeBSD community.
In order to maintain consistency within the database and provide a consistent user experience, guidelines have been established covering common aspects of bug management such as presenting followup, handling close requests, and so forth.
[[pr-lifecycle]]
== Problem Report Life-cycle
@ -49,41 +52,45 @@ Access to Bugzilla is available to the entire FreeBSD community. In order to mai
[NOTE]
====
Many PRs are submitted with very little information about the problem, and some are either very complex to solve, or just scratch the surface of a larger problem; in these cases, it is very important to obtain all the necessary information needed to solve the problem. If the problem contained within cannot be solved, or has occurred again, it is necessary to re-open the PR.
Many PRs are submitted with very little information about the problem, and some are either very complex to solve, or just scratch the surface of a larger problem; in these cases, it is very important to obtain all the necessary information needed to solve the problem.
If the problem contained within cannot be solved, or has occurred again, it is necessary to re-open the PR.
====
[[pr-states]]
== Problem Report State
It is important to update the state of a PR when certain actions are taken. The state should accurately reflect the current state of work on the PR.
It is important to update the state of a PR when certain actions are taken.
The state should accurately reflect the current state of work on the PR.
.A small example on when to change PR state
[example]
====
When a PR has been worked on and the developer(s) responsible feel comfortable about the fix, they will submit a followup to the PR and change its state to "feedback". At this point, the originator should evaluate the fix in their context and respond indicating whether the defect has indeed been remedied.
When a PR has been worked on and the developer(s) responsible feel comfortable about the fix, they will submit a followup to the PR and change its state to "feedback".
At this point, the originator should evaluate the fix in their context and respond indicating whether the defect has indeed been remedied.
====
A Problem Report may be in one of the following states:
[.glosslist]
open::
Initial state; the problem has been pointed out and it needs reviewing.
Initial state; the problem has been pointed out and it needs reviewing.
analyzed::
The problem has been reviewed and a solution is being sought.
The problem has been reviewed and a solution is being sought.
feedback::
Further work requires additional information from the originator or the community; possibly information regarding the proposed solution.
Further work requires additional information from the originator or the community; possibly information regarding the proposed solution.
patched::
A patch has been committed, but something (MFC, or maybe confirmation from originator) is still pending.
A patch has been committed, but something (MFC, or maybe confirmation from originator) is still pending.
suspended::
The problem is not being worked on, due to lack of information or resources. This is a prime candidate for somebody who is looking for a project to take on. If the problem cannot be solved at all, it will be closed, rather than suspended. The documentation project uses suspended for wish-list items that entail a significant amount of work which no one currently has time for.
The problem is not being worked on, due to lack of information or resources.
This is a prime candidate for somebody who is looking for a project to take on.
If the problem cannot be solved at all, it will be closed, rather than suspended.
The documentation project uses suspended for wish-list items that entail a significant amount of work which no one currently has time for.
closed::
A problem report is closed when any changes have been integrated, documented, and tested, or when fixing the problem is abandoned.
A problem report is closed when any changes have been integrated, documented, and tested, or when fixing the problem is abandoned.
[NOTE]
====
@ -106,7 +113,11 @@ The following sections describe what each different type of PRs is used for, whe
[[pr-unassigned]]
== Unassigned PRs
When PRs arrive, they are initially assigned to a generic (placeholder) assignee. These are always prepended with `freebsd-`. The exact value for this default depends on the category; in most cases, it corresponds to a specific FreeBSD mailing list. Here is the current list, with the most common ones listed first:
When PRs arrive, they are initially assigned to a generic (placeholder) assignee.
These are always prepended with `freebsd-`.
The exact value for this default depends on the category; in most cases, it corresponds to a specific FreeBSD mailing list.
Here is the current list, with the most common ones listed first:
[[default-assignees-common]]
.Default Assignees - most common
[cols="1,1,1", options="header"]
@ -165,11 +176,16 @@ When PRs arrive, they are initially assigned to a generic (placeholder) assignee
|freebsd-usb
|===
Do not be surprised to find that the submitter of the PR has assigned it to the wrong category. If you fix the category, do not forget to fix the assignment as well. (In particular, our submitters seem to have a hard time understanding that just because their problem manifested on an i386 system, that it might be generic to all of FreeBSD, and thus be more appropriate for `kern`. The converse is also true, of course.)
Do not be surprised to find that the submitter of the PR has assigned it to the wrong category.
If you fix the category, do not forget to fix the assignment as well.
(In particular, our submitters seem to have a hard time understanding that just because their problem manifested on an i386 system, that it might be generic to all of FreeBSD, and thus be more appropriate for `kern`.
The converse is also true, of course.)
Certain PRs may be reassigned away from these generic assignees by anyone. There are several types of assignees: specialized mailing lists; mail aliases (used for certain limited-interest items); and individuals.
Certain PRs may be reassigned away from these generic assignees by anyone.
There are several types of assignees: specialized mailing lists; mail aliases (used for certain limited-interest items); and individuals.
For assignees which are mailing lists, please use the long form when making the assignment (e.g., `freebsd-foo` instead of `foo`); this will avoid duplicate emails sent to the mailing list.
For assignees which are mailing lists, please use the long form when making the assignment (e.g., `freebsd-foo` instead of `foo`);
this will avoid duplicate emails sent to the mailing list.
[NOTE]
====
@ -405,7 +421,9 @@ Here is a sample list of such entities; it is probably not complete.
Ports PRs which have a maintainer who is a ports committer may be reassigned by anyone (but note that not every FreeBSD committer is necessarily a ports committer, so you cannot simply go by the email address alone.)
For other PRs, please do not reassign them to individuals (other than yourself) unless you are certain that the assignee really wants to track the PR. This will help to avoid the case where no one looks at fixing a particular problem because everyone assumes that the assignee is already working on it.
For other PRs, please do not reassign them to individuals (other than yourself) unless you are certain that the assignee really wants to track the PR.
This will help to avoid the case where no one looks at fixing a particular problem because everyone assumes that the assignee is already working on it.
[[common-assignees-other]]
.Common Assignees - Other
[cols="1,1,1,1", options="header"]
@ -431,12 +449,16 @@ For other PRs, please do not reassign them to individuals (other than yourself)
If a PR has the `responsible` field set to the username of a FreeBSD developer, it means that the PR has been handed over to that particular person for further work.
Assigned PRs should not be touched by anyone but the assignee or bugmeister. If you have comments, submit a followup. If for some reason you think the PR should change state or be reassigned, send a message to the assignee. If the assignee does not respond within two weeks, unassign the PR and do as you please.
Assigned PRs should not be touched by anyone but the assignee or bugmeister.
If you have comments, submit a followup.
If for some reason you think the PR should change state or be reassigned, send a message to the assignee.
If the assignee does not respond within two weeks, unassign the PR and do as you please.
[[pr-dups]]
== Duplicate PRs
If you find more than one PR that describe the same issue, choose the one that contains the largest amount of useful information and close the others, stating clearly the number of the superseding PR. If several PRs contain non-overlapping useful information, submit all the missing information to one in a followup, including references to the others; then close the other PRs (which are now completely superseded).
If you find more than one PR that describe the same issue, choose the one that contains the largest amount of useful information and close the others, stating clearly the number of the superseding PR.
If several PRs contain non-overlapping useful information, submit all the missing information to one in a followup, including references to the others; then close the other PRs (which are now completely superseded).
[[pr-stale]]
== Stale PRs
@ -454,7 +476,8 @@ A PR is considered stale if it has not been modified in more than six months. Ap
Developers that come across PRs that look like they should have been posted to {freebsd-bugs} or some other list should close the PR, informing the submitter in a comment why this is not really a PR and where the message should be posted.
The email addresses that Bugzilla listens to for incoming PRs have been published as part of the FreeBSD documentation, have been announced and listed on the web-site. This means that spammers found them.
The email addresses that Bugzilla listens to for incoming PRs have been published as part of the FreeBSD documentation, have been announced and listed on the web-site.
This means that spammers found them.
Whenever you close one of these PRs, please do the following:
@ -467,6 +490,7 @@ Setting the category to `junk` makes it obvious that there is no useful content
[[references]]
== Further Reading
This is a list of resources relevant to the proper writing and processing of problem reports. It is by no means complete.
This is a list of resources relevant to the proper writing and processing of problem reports.
It is by no means complete.
* link:{problem-reports}[How to Write FreeBSD Problem Reports]-guidelines for PR originators.

96
documentation/content/en/articles/problem-reports/_index.adoc
View file

@ -32,20 +32,28 @@ toc::[]
[[pr-intro]]
== Introduction
One of the most frustrating experiences one can have as a software user is to submit a problem report only to have it summarily closed with a terse and unhelpful explanation like "not a bug" or "bogus PR". Similarly, one of the most frustrating experiences as a software developer is to be flooded with problem reports that are not really problem reports but requests for support, or that contain little or no information about what the problem is and how to reproduce it.
One of the most frustrating experiences one can have as a software user is to submit a problem report only to have it summarily closed with a terse and unhelpful explanation like "not a bug" or "bogus PR".
Similarly, one of the most frustrating experiences as a software developer is to be flooded with problem reports that are not really problem reports but requests for support, or that contain little or no information about what the problem is and how to reproduce it.
This document attempts to describe how to write good problem reports. What, one asks, is a good problem report? Well, to go straight to the bottom line, a good problem report is one that can be analyzed and dealt with swiftly, to the mutual satisfaction of both user and developer.
This document attempts to describe how to write good problem reports.
What, one asks, is a good problem report? Well, to go straight to the bottom line, a good problem report is one that can be analyzed and dealt with swiftly, to the mutual satisfaction of both user and developer.
Although the primary focus of this article is on FreeBSD problem reports, most of it should apply quite well to other software projects.
Note that this article is organized thematically, not chronologically. Read the entire document before submitting a problem report, rather than treating it as a step-by-step tutorial.
Note that this article is organized thematically, not chronologically.
Read the entire document before submitting a problem report, rather than treating it as a step-by-step tutorial.
[[pr-when]]
== When to Submit a Problem Report
There are many types of problems, and not all of them should engender a problem report. Of course, nobody is perfect, and there will be times when what seems to be a bug in a program is, in fact, a misunderstanding of the syntax for a command or a typographical error in a configuration file (though that in itself may sometimes be indicative of poor documentation or poor error handling in the application). There are still many cases where submitting a problem report is clearly _not_ the right course of action, and will only serve to frustrate both the submitter and the developers. Conversely, there are cases where it might be appropriate to submit a problem report about something else than a bug-an enhancement or a new feature, for instance.
There are many types of problems, and not all of them should engender a problem report.
Of course, nobody is perfect, and there will be times when what seems to be a bug in a program is, in fact, a misunderstanding of the syntax for a command or a typographical error in a configuration file (though that in itself may sometimes be indicative of poor documentation or poor error handling in the application).
There are still many cases where submitting a problem report is clearly _not_ the right course of action, and will only serve to frustrate both the submitter and the developers.
Conversely, there are cases where it might be appropriate to submit a problem report about something else than a bug-an enhancement or a new feature, for instance.
So how does one determine what is a bug and what is not? As a simple rule of thumb, the problem is _not_ a bug if it can be expressed as a question (usually of the form "How do I do X?" or "Where can I find Y?"). It is not always quite so black and white, but the question rule covers a large majority of cases. When looking for an answer, consider posing the question to the {freebsd-questions}.
So how does one determine what is a bug and what is not? As a simple rule of thumb, the problem is _not_ a bug if it can be expressed as a question (usually of the form "How do I do X?" or "Where can I find Y?").
It is not always quite so black and white, but the question rule covers a large majority of cases.
When looking for an answer, consider posing the question to the {freebsd-questions}.
Consider these factors when submitting PRs about ports or other software that is not part of FreeBSD itself:
@ -53,7 +61,10 @@ Consider these factors when submitting PRs about ports or other software that is
* For unmaintained ports (`MAINTAINER` is `ports@FreeBSD.org`), a PR without an included patch is unlikely to get picked up by a committer. To become the maintainer of an unmaintained port, submit a PR with the request (patch preferred but not required).
* In either case, following the process described in link:{porters-handbook}#port-upgrading[Porter's Handbook] will yield the best results. (You might also wish to read link:{contributing}#ports-contributing[Contributing to the FreeBSD Ports Collection].)
A bug that cannot be reproduced can rarely be fixed. If the bug only occurred once and you cannot reproduce it, and it does not seem to happen to anybody else, chances are none of the developers will be able to reproduce it or figure out what is wrong. That does not mean it did not happen, but it does mean that the chances of your problem report ever leading to a bug fix are very slim. To make matters worse, often these kinds of bugs are actually caused by failing hard drives or overheating processors - you should always try to rule out these causes, whenever possible, before submitting a PR.
A bug that cannot be reproduced can rarely be fixed.
If the bug only occurred once and you cannot reproduce it, and it does not seem to happen to anybody else, chances are none of the developers will be able to reproduce it or figure out what is wrong.
That does not mean it did not happen, but it does mean that the chances of your problem report ever leading to a bug fix are very slim.
To make matters worse, often these kinds of bugs are actually caused by failing hard drives or overheating processors - you should always try to rule out these causes, whenever possible, before submitting a PR.
Next, to decide to whom you should file your problem report, you need to understand that the software that makes up FreeBSD is composed of several different elements:
@ -61,16 +72,23 @@ Next, to decide to whom you should file your problem report, you need to underst
* Code in the base system that is written and maintained by others, and imported into FreeBSD and adapted. Examples include man:clang[1], and man:sendmail[8]. Most bugs in these areas should be reported to the FreeBSD developers; but in some cases they may need to be reported to the original authors instead if the problems are not FreeBSD-specific.
* Individual applications that are not in the base system but are instead part of the FreeBSD Ports Collection (category `ports`). Most of these applications are not written by FreeBSD developers; what FreeBSD provides is merely a framework for installing the application. Therefore, only report a problem to the FreeBSD developers when the problem is believed to be FreeBSD-specific; otherwise, report it to the authors of the software.
Then, ascertain whether the problem is timely. There are few things that will annoy a developer more than receiving a problem report about a bug she has already fixed.
Then, ascertain whether the problem is timely.
There are few things that will annoy a developer more than receiving a problem report about a bug she has already fixed.
If the problem is in the base system, first read the FAQ section on link:{faq}#LATEST-VERSION[FreeBSD versions], if you are not already familiar with the topic. It is not possible for FreeBSD to fix problems in anything other than certain recent branches of the base system, so filing a bug report about an older version will probably only result in a developer advising you to upgrade to a supported version to see if the problem still recurs. The Security Officer team maintains the link:https://www.FreeBSD.org/security/[list of supported versions].
If the problem is in the base system, first read the FAQ section on link:{faq}#LATEST-VERSION[FreeBSD versions], if you are not already familiar with the topic.
It is not possible for FreeBSD to fix problems in anything other than certain recent branches of the base system, so filing a bug report about an older version will probably only result in a developer advising you to upgrade to a supported version to see if the problem still recurs.
The Security Officer team maintains the link:https://www.FreeBSD.org/security/[list of supported versions].
If the problem is in a port, consider filing a bug with the upstream. The FreeBSD Project can not fix all bugs in all software.
If the problem is in a port, consider filing a bug with the upstream.
The FreeBSD Project can not fix all bugs in all software.
[[pr-prep]]
== Preparations
A good rule to follow is to always do a background search before submitting a problem report. Maybe the problem has already been reported; maybe it is being discussed on the mailing lists, or recently was; it may even already be fixed in a newer version than what you are running. You should therefore check all the obvious places before submitting your problem report. For FreeBSD, this means:
A good rule to follow is to always do a background search before submitting a problem report.
Maybe the problem has already been reported; maybe it is being discussed on the mailing lists, or recently was; it may even already be fixed in a newer version than what you are running.
You should therefore check all the obvious places before submitting your problem report.
For FreeBSD, this means:
* The FreeBSD link:{faq}[Frequently Asked Questions] (FAQ) list. The FAQ attempts to provide answers for a wide range of questions, such as those concerning link:{faq}#hardware[hardware compatibility], link:{faq}#applications[user applications], and link:{faq}#kernelconfig[kernel configuration].
* The link:{handbook}#eresources-mail[mailing lists]-if you are not subscribed, use https://www.FreeBSD.org/search/#mailinglists[the searchable archives] on the FreeBSD web site. If the problem has not been discussed on the lists, you might try posting a message about it and waiting a few days to see if someone can spot something that has been overlooked.
@ -78,14 +96,17 @@ A good rule to follow is to always do a background search before submitting a pr
* Next, the searchable https://bugs.freebsd.org/bugzilla/query.cgi[FreeBSD PR database] (Bugzilla). Unless the problem is recent or obscure, there is a fair chance it has already been reported.
* Most importantly, attempt to see if existing documentation in the source base addresses your problem.
+
For the base FreeBSD code, you should carefully study the contents of [.filename]#/usr/src/UPDATING# on your system or the latest version at https://svnweb.freebsd.org/base/head/UPDATING?view=log[https://svnweb.freebsd.org/base/head/UPDATING?view=log]. (This is vital information if you are upgrading from one version to another-especially if you are upgrading to the FreeBSD-CURRENT branch).
For the base FreeBSD code, you should carefully study the contents of [.filename]#/usr/src/UPDATING# on your system or the latest version at https://svnweb.freebsd.org/base/head/UPDATING?view=log[https://svnweb.freebsd.org/base/head/UPDATING?view=log].
(This is vital information if you are upgrading from one version to another-especially if you are upgrading to the FreeBSD-CURRENT branch).
+
However, if the problem is in something that was installed as a part of the FreeBSD Ports Collection, you should refer to [.filename]#/usr/ports/UPDATING# (for individual ports) or [.filename]#/usr/ports/CHANGES# (for changes that affect the entire Ports Collection). https://svnweb.freebsd.org/ports/head/UPDATING?view=log[https://svnweb.freebsd.org/ports/head/UPDATING?view=log] and https://svnweb.freebsd.org/ports/head/CHANGES?view=log[https://svnweb.freebsd.org/ports/head/CHANGES?view=log] are also available via svnweb.
However, if the problem is in something that was installed as a part of the FreeBSD Ports Collection, you should refer to [.filename]#/usr/ports/UPDATING# (for individual ports) or [.filename]#/usr/ports/CHANGES# (for changes that affect the entire Ports Collection).
https://svnweb.freebsd.org/ports/head/UPDATING?view=log[https://svnweb.freebsd.org/ports/head/UPDATING?view=log] and https://svnweb.freebsd.org/ports/head/CHANGES?view=log[https://svnweb.freebsd.org/ports/head/CHANGES?view=log] are also available via svnweb.
[[pr-writing]]
== Writing the Problem Report
Now that you have decided that your issue merits a problem report, and that it is a FreeBSD problem, it is time to write the actual problem report. Before we get into the mechanics of the program used to generate and submit PRs, here are some tips and tricks to help make sure that your PR will be most effective.
Now that you have decided that your issue merits a problem report, and that it is a FreeBSD problem, it is time to write the actual problem report.
Before we get into the mechanics of the program used to generate and submit PRs, here are some tips and tricks to help make sure that your PR will be most effective.
[[pr-writing-tips]]
== Tips and Tricks for Writing a Good Problem Report
@ -123,20 +144,27 @@ Now that you have decided that your issue merits a problem report, and that it i
[[pr-writing-before-beginning]]
== Before Beginning
Similar considerations apply to use of the https://bugs.freebsd.org/bugzilla/enter_bug.cgi[web-based PR submission form]. Be careful of cut-and-paste operations that might change whitespace or other text formatting.
Similar considerations apply to use of the https://bugs.freebsd.org/bugzilla/enter_bug.cgi[web-based PR submission form].
Be careful of cut-and-paste operations that might change whitespace or other text formatting.
Finally, if the submission is lengthy, prepare the work offline so that nothing will be lost if there is a problem submitting it.
[[pr-writing-attaching-patches]]
== Attaching Patches or Files
When attaching a patch, be sure to use either `svn diff` or man:diff[1] with the `-u` option to create a unified diff and make sure to specify the SVN revision number of the repository against which you modified files, so the developers who read your report will be able to apply them easily. For problems with the kernel or the base utilities, a patch against FreeBSD-CURRENT (the HEAD Subversion branch) is preferred since all new code should be applied and tested there first. After appropriate or substantial testing has been done, the code will be merged/migrated to the FreeBSD-STABLE branch.
When attaching a patch, be sure to use either `svn diff` or man:diff[1] with the `-u` option to create a unified diff and make sure to specify the SVN revision number of the repository against which you modified files, so the developers who read your report will be able to apply them easily.
For problems with the kernel or the base utilities, a patch against FreeBSD-CURRENT (the HEAD Subversion branch) is preferred since all new code should be applied and tested there first.
After appropriate or substantial testing has been done, the code will be merged/migrated to the FreeBSD-STABLE branch.
If you attach a patch inline, instead of as an attachment, note that the most common problem by far is the tendency of some email programs to render tabs as spaces, which will completely ruin anything intended to be part of a Makefile.
Do not send patches as attachments using `Content-Transfer-Encoding: quoted-printable`. These will perform character escaping and the entire patch will be useless.
Do not send patches as attachments using `Content-Transfer-Encoding: quoted-printable`.
These will perform character escaping and the entire patch will be useless.
Also note that while including small patches in a PR is generally all right-particularly when they fix the problem described in the PR-large patches and especially new code which may require substantial review before committing should be placed on a web or ftp server, and the URL should be included in the PR instead of the patch. Patches in email tend to get mangled, and the larger the patch, the harder it will be for interested parties to unmangle it. Also, posting a patch on the web allows you to modify it without having to resubmit the entire patch in a followup to the original PR. Finally, large patches simply increase the size of the database, since closed PRs are not actually deleted but instead kept and simply marked as complete.
Also note that while including small patches in a PR is generally all right-particularly when they fix the problem described in the PR-large patches and especially new code which may require substantial review before committing should be placed on a web or ftp server, and the URL should be included in the PR instead of the patch.
Patches in email tend to get mangled, and the larger the patch, the harder it will be for interested parties to unmangle it.
Also, posting a patch on the web allows you to modify it without having to resubmit the entire patch in a followup to the original PR.
Finally, large patches simply increase the size of the database, since closed PRs are not actually deleted but instead kept and simply marked as complete.
You should also take note that unless you explicitly specify otherwise in your PR or in the patch itself, any patches you submit will be assumed to be licensed under the same terms as the original file you modified.
@ -145,7 +173,9 @@ You should also take note that unless you explicitly specify otherwise in your P
[NOTE]
====
The email address you use will become public information and may become available to spammers. You should either have spam handling procedures in place, or use a temporary email account. However, please note that if you do not use a valid email account at all, we will not be able to ask you questions about your PR.
The email address you use will become public information and may become available to spammers.
You should either have spam handling procedures in place, or use a temporary email account.
However, please note that if you do not use a valid email account at all, we will not be able to ask you questions about your PR.
====
When you file a bug, you will find the following fields:
@ -154,7 +184,10 @@ When you file a bug, you will find the following fields:
* _Severity:_ One of `Affects only me`, `Affects some people` or `Affects many people`. Do not overreact; refrain from labeling your problem `Affects many people` unless it really does. FreeBSD developers will not necessarily work on your problem faster if you inflate its importance since there are so many other people who have done exactly that.
* _Category:_ Choose an appropriate category.
+
The first thing you need to do is to decide what part of the system your problem lies in. Remember, FreeBSD is a complete operating system, which installs both a kernel, the standard libraries, many peripheral drivers, and a large number of utilities (the "base system"). However, there are thousands of additional applications in the Ports Collection. You'll first need to decide if the problem is in the base system or something installed via the Ports Collection.
The first thing you need to do is to decide what part of the system your problem lies in.
Remember, FreeBSD is a complete operating system, which installs both a kernel, the standard libraries, many peripheral drivers, and a large number of utilities (the "base system").
However, there are thousands of additional applications in the Ports Collection.
You'll first need to decide if the problem is in the base system or something installed via the Ports Collection.
+
Here is a description of the major categories:
@ -167,7 +200,6 @@ Here is a description of the major categories:
====
if you are having a problem with something from a port named `www/_someportname_`, this nevertheless goes in the `ports` category.
====
+
There are a few more specialized categories.
@ -184,14 +216,12 @@ These categories are quite often misused for "I do not know" problems. Rather th
.Correct Use of Arch-Specific Category
[example]
====
You have a common PC-based machine, and think you have encountered a problem specific to a particular chipset or a particular motherboard: `i386` is the right category.
====
+
.Incorrect Use of Arch-Specific Category
[example]
====
You are having a problem with an add-in peripheral card on a commonly seen bus, or a problem with a particular type of hard disk drive: in this case, it probably applies to more than one architecture, and `kern` is the right category.
====
** If you really do not know where the problem lies (or the explanation does not seem to fit into the ones above), use the `misc` category. Before you do so, you may wish to ask for help on the {freebsd-questions} first. You may be advised that one of the existing categories really is a better choice.
@ -201,13 +231,19 @@ You are having a problem with an add-in peripheral card on a commonly seen bus,
[[pr-followup]]
== Follow-up
Once the problem report has been filed, you will receive a confirmation by email which will include the tracking number that was assigned to your problem report and a URL you can use to check its status. With a little luck, someone will take an interest in your problem and try to address it, or, as the case may be, explain why it is not a problem. You will be automatically notified of any change of status, and you will receive copies of any comments or patches someone may attach to your problem report's audit trail.
Once the problem report has been filed, you will receive a confirmation by email which will include the tracking number that was assigned to your problem report and a URL you can use to check its status.
With a little luck, someone will take an interest in your problem and try to address it, or, as the case may be, explain why it is not a problem.
You will be automatically notified of any change of status, and you will receive copies of any comments or patches someone may attach to your problem report's audit trail.
If someone requests additional information from you, or you remember or discover something you did not mention in the initial report, please submit a follow up. The number one reason for a bug not getting fixed is lack of communication with the originator. The easiest way is to use the comment option on the individual PR's web page, which you can reach from the https://bugs.freebsd.org/bugzilla/query.cgi[PR search page].
If someone requests additional information from you, or you remember or discover something you did not mention in the initial report, please submit a follow up.
The number one reason for a bug not getting fixed is lack of communication with the originator.
The easiest way is to use the comment option on the individual PR's web page, which you can reach from the https://bugs.freebsd.org/bugzilla/query.cgi[PR search page].
If the problem report remains open after the problem has gone away, just add a comment saying that the problem report can be closed, and, if possible, explaining how or when the problem was fixed.
Sometimes there is a delay of a week or two where the problem report remains untouched, not assigned or commented on by anyone. This can happen when there is an increased problem report backlog or during a holiday season. When a problem report has not received attention after several weeks, it is worth finding a committer particularly interested in working on it.
Sometimes there is a delay of a week or two where the problem report remains untouched, not assigned or commented on by anyone.
This can happen when there is an increased problem report backlog or during a holiday season.
When a problem report has not received attention after several weeks, it is worth finding a committer particularly interested in working on it.
There are a few ways to do so, ideally in the following order, with a few days between attempting each communication channel:
@ -215,17 +251,21 @@ There are a few ways to do so, ideally in the following order, with a few days b
* Join the relevant IRC channels. A partial listing is here: https://wiki.freebsd.org/IrcChannels[]. Inform the people in that channel about the problem report and ask for assistance. Be patient and stay in the channel after posting, so that the people from different time zones around the world have a chance to catch up.
* Find committers interested in the problem that was reported. If the problem was in a particular tool, binary, port, document, or source file, check the http://svnweb.FreeBSD.org[SVN Repository]. Locate the last few committers who made substantive changes to the file, and try to reach them via IRC or email. A list of committers and their emails can be found in the link:{contributors}[Contributors to FreeBSD] article.
Remember that these people are volunteers, just like maintainers and users, so they might not be immediately available to assist with the problem report. Patience and consistency in the follow-ups is highly advised and appreciated. With enough care and effort dedicated to that follow-up process, finding a committer to take care of the problem report is just a matter of time.
Remember that these people are volunteers, just like maintainers and users, so they might not be immediately available to assist with the problem report.
Patience and consistency in the follow-ups is highly advised and appreciated.
With enough care and effort dedicated to that follow-up process, finding a committer to take care of the problem report is just a matter of time.
[[pr-problems]]
== If There Are Problems
If you found an issue with the bug system, file a bug! There is a category for exactly this purpose. If you are unable to do so, contact the bug wranglers at mailto:bugmeister@FreeBSD.org[bugmeister@FreeBSD.org].
If you found an issue with the bug system, file a bug! There is a category for exactly this purpose.
If you are unable to do so, contact the bug wranglers at mailto:bugmeister@FreeBSD.org[bugmeister@FreeBSD.org].
[[pr-further]]
== Further Reading
This is a list of resources relevant to the proper writing and processing of problem reports. It is by no means complete.
This is a list of resources relevant to the proper writing and processing of problem reports.
It is by no means complete.
* https://github.com/smileytechguy/reporting-bugs-effectively/blob/master/ENGLISH.md[How to Report Bugs Effectively]-an excellent essay by Simon G. Tatham on composing useful (non-FreeBSD-specific) problem reports.
* link:{pr-guidelines}[Problem Report Handling Guidelines]-valuable insight into how problem reports are handled by the FreeBSD developers.

385
documentation/content/en/articles/rc-scripting/_index.adoc
View file

@ -23,7 +23,9 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
Beginners may find it difficult to relate the facts from the formal documentation on the BSD [.filename]#rc.d# framework with the practical tasks of [.filename]#rc.d# scripting. In this article, we consider a few typical cases of increasing complexity, show [.filename]#rc.d# features suited for each case, and discuss how they work. Such an examination should provide reference points for further study of the design and efficient application of [.filename]#rc.d#.
Beginners may find it difficult to relate the facts from the formal documentation on the BSD [.filename]#rc.d# framework with the practical tasks of [.filename]#rc.d# scripting.
In this article, we consider a few typical cases of increasing complexity, show [.filename]#rc.d# features suited for each case, and discuss how they work.
Such an examination should provide reference points for further study of the design and efficient application of [.filename]#rc.d#.
'''
@ -32,26 +34,63 @@ toc::[]
[[rcng-intro]]
== Introduction
The historical BSD had a monolithic startup script, [.filename]#/etc/rc#. It was invoked by man:init[8] at system boot time and performed all userland tasks required for multi-user operation: checking and mounting file systems, setting up the network, starting daemons, and so on. The precise list of tasks was not the same in every system; admins needed to customize it. With few exceptions, [.filename]#/etc/rc# had to be modified, and true hackers liked it.
The historical BSD had a monolithic startup script, [.filename]#/etc/rc#.
It was invoked by man:init[8] at system boot time and performed all userland tasks required for multi-user operation: checking and mounting file systems, setting up the network, starting daemons, and so on.
The precise list of tasks was not the same in every system; admins needed to customize it.
With few exceptions, [.filename]#/etc/rc# had to be modified, and true hackers liked it.
The real problem with the monolithic approach was that it provided no control over the individual components started from [.filename]#/etc/rc#. For instance, [.filename]#/etc/rc# could not restart a single daemon. The system admin had to find the daemon process by hand, kill it, wait until it actually exited, then browse through [.filename]#/etc/rc# for the flags, and finally type the full command line to start the daemon again. The task would become even more difficult and prone to errors if the service to restart consisted of more than one daemon or demanded additional actions. In a few words, the single script failed to fulfil what scripts are for: to make the system admin's life easier.
The real problem with the monolithic approach was that it provided no control over the individual components started from [.filename]#/etc/rc#.
For instance, [.filename]#/etc/rc# could not restart a single daemon.
The system admin had to find the daemon process by hand, kill it, wait until it actually exited, then browse through [.filename]#/etc/rc# for the flags, and finally type the full command line to start the daemon again.
The task would become even more difficult and prone to errors if the service to restart consisted of more than one daemon or demanded additional actions.
In a few words, the single script failed to fulfil what scripts are for: to make the system admin's life easier.
Later there was an attempt to split out some parts of [.filename]#/etc/rc# for the sake of starting the most important subsystems separately. The notorious example was [.filename]#/etc/netstart# to bring up networking. It did allow for accessing the network from single-user mode, but it did not integrate well into the automatic startup process because parts of its code needed to interleave with actions essentially unrelated to networking. That was why [.filename]#/etc/netstart# mutated into [.filename]#/etc/rc.network#. The latter was no longer an ordinary script; it comprised of large, tangled man:sh[1] functions called from [.filename]#/etc/rc# at different stages of system startup. However, as the startup tasks grew diverse and sophisticated, the "quasi-modular" approach became even more of a drag than the monolithic [.filename]#/etc/rc# had been.
Later there was an attempt to split out some parts of [.filename]#/etc/rc# for the sake of starting the most important subsystems separately.
The notorious example was [.filename]#/etc/netstart# to bring up networking.
It did allow for accessing the network from single-user mode, but it did not integrate well into the automatic startup process because parts of its code needed to interleave with actions essentially unrelated to networking.
That was why [.filename]#/etc/netstart# mutated into [.filename]#/etc/rc.network#.
The latter was no longer an ordinary script; it comprised of large, tangled man:sh[1] functions called from [.filename]#/etc/rc# at different stages of system startup.
However, as the startup tasks grew diverse and sophisticated, the "quasi-modular" approach became even more of a drag than the monolithic [.filename]#/etc/rc# had been.
Without a clean and well-designed framework, the startup scripts had to bend over backwards to satisfy the needs of rapidly developing BSD-based operating systems. It became obvious at last that more steps are necessary on the way to a fine-grained and extensible [.filename]#rc# system. Thus BSD [.filename]#rc.d# was born. Its acknowledged fathers were Luke Mewburn and the NetBSD community. Later it was imported into FreeBSD. Its name refers to the location of system scripts for individual services, which is in [.filename]#/etc/rc.d#. Soon we will learn about more components of the [.filename]#rc.d# system and see how the individual scripts are invoked.
Without a clean and well-designed framework, the startup scripts had to bend over backwards to satisfy the needs of rapidly developing BSD-based operating systems.
It became obvious at last that more steps are necessary on the way to a fine-grained and extensible [.filename]#rc# system.
Thus BSD [.filename]#rc.d# was born.
Its acknowledged fathers were Luke Mewburn and the NetBSD community.
Later it was imported into FreeBSD.
Its name refers to the location of system scripts for individual services, which is in [.filename]#/etc/rc.d#.
Soon we will learn about more components of the [.filename]#rc.d# system and see how the individual scripts are invoked.
The basic ideas behind BSD [.filename]#rc.d# are _fine modularity_ and __code reuse__. _Fine modularity_ means that each basic "service" such as a system daemon or primitive startup task gets its own man:sh[1] script able to start the service, stop it, reload it, check its status. A particular action is chosen by the command-line argument to the script. The [.filename]#/etc/rc# script still drives system startup, but now it merely invokes the smaller scripts one by one with the `start` argument. It is easy to perform shutdown tasks as well by running the same set of scripts with the `stop` argument, which is done by [.filename]#/etc/rc.shutdown#. Note how closely this follows the Unix way of having a set of small specialized tools, each fulfilling its task as well as possible. _Code reuse_ means that common operations are implemented as man:sh[1] functions and collected in [.filename]#/etc/rc.subr#. Now a typical script can be just a few lines' worth of man:sh[1] code. Finally, an important part of the [.filename]#rc.d# framework is man:rcorder[8], which helps [.filename]#/etc/rc# to run the small scripts orderly with respect to dependencies between them. It can help [.filename]#/etc/rc.shutdown#, too, because the proper order for the shutdown sequence is opposite to that of startup.
The basic ideas behind BSD [.filename]#rc.d# are _fine modularity_ and __code reuse__.
_Fine modularity_ means that each basic "service" such as a system daemon or primitive startup task gets its own man:sh[1] script able to start the service, stop it, reload it, check its status.
A particular action is chosen by the command-line argument to the script.
The [.filename]#/etc/rc# script still drives system startup, but now it merely invokes the smaller scripts one by one with the `start` argument.
It is easy to perform shutdown tasks as well by running the same set of scripts with the `stop` argument, which is done by [.filename]#/etc/rc.shutdown#.
Note how closely this follows the Unix way of having a set of small specialized tools, each fulfilling its task as well as possible.
_Code reuse_ means that common operations are implemented as man:sh[1] functions and collected in [.filename]#/etc/rc.subr#.
Now a typical script can be just a few lines' worth of man:sh[1] code.
Finally, an important part of the [.filename]#rc.d# framework is man:rcorder[8], which helps [.filename]#/etc/rc# to run the small scripts orderly with respect to dependencies between them.
It can help [.filename]#/etc/rc.shutdown#, too, because the proper order for the shutdown sequence is opposite to that of startup.
The BSD [.filename]#rc.d# design is described in <<lukem, the original article by Luke Mewburn>>, and the [.filename]#rc.d# components are documented in great detail in <<manpages, the respective manual pages>>. However, it might not appear obvious to an [.filename]#rc.d# newbie how to tie the numerous bits and pieces together in order to create a well-styled script for a particular task. Therefore this article will try a different approach to describe [.filename]#rc.d#. It will show which features should be used in a number of typical cases, and why. Note that this is not a how-to document because our aim is not at giving ready-made recipes, but at showing a few easy entrances into the [.filename]#rc.d# realm. Neither is this article a replacement for the relevant manual pages. Do not hesitate to refer to them for more formal and complete documentation while reading this article.
The BSD [.filename]#rc.d# design is described in <<lukem, the original article by Luke Mewburn>>, and the [.filename]#rc.d# components are documented in great detail in <<manpages, the respective manual pages>>.
However, it might not appear obvious to an [.filename]#rc.d# newbie how to tie the numerous bits and pieces together in order to create a well-styled script for a particular task.
Therefore this article will try a different approach to describe [.filename]#rc.d#.
It will show which features should be used in a number of typical cases, and why.
Note that this is not a how-to document because our aim is not at giving ready-made recipes, but at showing a few easy entrances into the [.filename]#rc.d# realm.
Neither is this article a replacement for the relevant manual pages.
Do not hesitate to refer to them for more formal and complete documentation while reading this article.
There are prerequisites to understanding this article. First of all, you should be familiar with the man:sh[1] scripting language in order to master [.filename]#rc.d#. In addition, you should know how the system performs userland startup and shutdown tasks, which is described in man:rc[8].
There are prerequisites to understanding this article.
First of all, you should be familiar with the man:sh[1] scripting language in order to master [.filename]#rc.d#.
In addition, you should know how the system performs userland startup and shutdown tasks, which is described in man:rc[8].
This article focuses on the FreeBSD branch of [.filename]#rc.d#. Nevertheless, it may be useful to NetBSD developers, too, because the two branches of BSD [.filename]#rc.d# not only share the same design but also stay similar in their aspects visible to script authors.
This article focuses on the FreeBSD branch of [.filename]#rc.d#.
Nevertheless, it may be useful to NetBSD developers, too, because the two branches of BSD [.filename]#rc.d# not only share the same design but also stay similar in their aspects visible to script authors.
[[rcng-task]]
== Outlining the task
A little consideration before starting `$EDITOR` will not hurt. In order to write a well-tempered [.filename]#rc.d# script for a system service, we should be able to answer the following questions first:
A little consideration before starting `$EDITOR` will not hurt.
In order to write a well-tempered [.filename]#rc.d# script for a system service, we should be able to answer the following questions first:
* Is the service mandatory or optional?
* Will the script serve a single program, e.g., a daemon, or perform more complex actions?
@ -85,7 +124,11 @@ run_rc_command "$1" <.>
Things to note are:
&#10122; An interpreted script should begin with the magic "shebang" line. That line specifies the interpreter program for the script. Due to the shebang line, the script can be invoked exactly like a binary program provided that it has the execute bit set. (See man:chmod[1].) For example, a system admin can run our script manually, from the command line:
&#10122; An interpreted script should begin with the magic "shebang" line.
That line specifies the interpreter program for the script.
Due to the shebang line, the script can be invoked exactly like a binary program provided that it has the execute bit set.
(See man:chmod[1].)
For example, a system admin can run our script manually, from the command line:
[source,shell]
....
@ -94,57 +137,83 @@ Things to note are:
[NOTE]
====
In order to be properly managed by the [.filename]#rc.d# framework, its scripts need to be written in the man:sh[1] language. If you have a service or port that uses a binary control utility or a startup routine written in another language, install that element in [.filename]#/usr/sbin# (for the system) or [.filename]#/usr/local/sbin# (for ports) and call it from a man:sh[1] script in the appropriate [.filename]#rc.d# directory.
In order to be properly managed by the [.filename]#rc.d# framework, its scripts need to be written in the man:sh[1] language.
If you have a service or port that uses a binary control utility or a startup routine written in another language, install that element in [.filename]#/usr/sbin# (for the system) or [.filename]#/usr/local/sbin# (for ports) and call it from a man:sh[1] script in the appropriate [.filename]#rc.d# directory.
====
[TIP]
====
If you would like to learn the details of why [.filename]#rc.d# scripts must be written in the man:sh[1] language, see how [.filename]#/etc/rc# invokes them by means of `run_rc_script`, then study the implementation of `run_rc_script` in [.filename]#/etc/rc.subr#.
====
&#10123; In [.filename]#/etc/rc.subr#, a number of man:sh[1] functions are defined for an [.filename]#rc.d# script to use. The functions are documented in man:rc.subr[8]. While it is theoretically possible to write an [.filename]#rc.d# script without ever using man:rc.subr[8], its functions prove extremely handy and make the job an order of magnitude easier. So it is no surprise that everybody resorts to man:rc.subr[8] in [.filename]#rc.d# scripts. We are not going to be an exception.
&#10123; In [.filename]#/etc/rc.subr#, a number of man:sh[1] functions are defined for an [.filename]#rc.d# script to use.
The functions are documented in man:rc.subr[8].
While it is theoretically possible to write an [.filename]#rc.d# script without ever using man:rc.subr[8], its functions prove extremely handy and make the job an order of magnitude easier. So it is no surprise that everybody resorts to man:rc.subr[8] in [.filename]#rc.d# scripts.
We are not going to be an exception.
An [.filename]#rc.d# script must "source"[.filename]#/etc/rc.subr# (include it using "`.`") _before_ it calls man:rc.subr[8] functions so that man:sh[1] has an opportunity to learn the functions. The preferred style is to source [.filename]#/etc/rc.subr# first of all.
An [.filename]#rc.d# script must "source"[.filename]#/etc/rc.subr# (include it using "`.`") _before_ it calls man:rc.subr[8] functions so that man:sh[1] has an opportunity to learn the functions.
The preferred style is to source [.filename]#/etc/rc.subr# first of all.
[NOTE]
====
Some useful functions related to networking are provided by another include file, [.filename]#/etc/network.subr#.
====
&#10124; [[name-var]]The mandatory variable `name` specifies the name of our script. It is required by man:rc.subr[8]. That is, each [.filename]#rc.d# script _must_ set `name` before it calls man:rc.subr[8] functions.
&#10124; [[name-var]]The mandatory variable `name` specifies the name of our script.
It is required by man:rc.subr[8].
That is, each [.filename]#rc.d# script _must_ set `name` before it calls man:rc.subr[8] functions.
Now it is the right time to choose a unique name for our script once and for all. We will use it in a number of places while developing the script. For a start, let us give the same name to the script file, too.
Now it is the right time to choose a unique name for our script once and for all.
We will use it in a number of places while developing the script.
For a start, let us give the same name to the script file, too.
[NOTE]
====
The current style of [.filename]#rc.d# scripting is to enclose values assigned to variables in double quotes. Keep in mind that it is just a style issue that may not always be applicable. You can safely omit quotes from around simple words without man:sh[1] metacharacters in them, while in certain cases you will need single quotes to prevent any interpretation of the value by man:sh[1]. A programmer should be able to tell the language syntax from style conventions and use both of them wisely.
The current style of [.filename]#rc.d# scripting is to enclose values assigned to variables in double quotes.
Keep in mind that it is just a style issue that may not always be applicable.
You can safely omit quotes from around simple words without man:sh[1] metacharacters in them, while in certain cases you will need single quotes to prevent any interpretation of the value by man:sh[1].
A programmer should be able to tell the language syntax from style conventions and use both of them wisely.
====
&#10125; The main idea behind man:rc.subr[8] is that an [.filename]#rc.d# script provides handlers, or methods, for man:rc.subr[8] to invoke. In particular, `start`, `stop`, and other arguments to an [.filename]#rc.d# script are handled this way. A method is a man:sh[1] expression stored in a variable named `argument_cmd`, where _argument_ corresponds to what can be specified on the script's command line. We will see later how man:rc.subr[8] provides default methods for the standard arguments.
&#10125; The main idea behind man:rc.subr[8] is that an [.filename]#rc.d# script provides handlers, or methods, for man:rc.subr[8] to invoke.
In particular, `start`, `stop`, and other arguments to an [.filename]#rc.d# script are handled this way.
A method is a man:sh[1] expression stored in a variable named `argument_cmd`, where _argument_ corresponds to what can be specified on the script's command line.
We will see later how man:rc.subr[8] provides default methods for the standard arguments.
[NOTE]
====
To make the code in [.filename]#rc.d# more uniform, it is common to use `${name}` wherever appropriate. Thus a number of lines can be just copied from one script to another.
To make the code in [.filename]#rc.d# more uniform, it is common to use `${name}` wherever appropriate.
Thus a number of lines can be just copied from one script to another.
====
&#10126; We should keep in mind that man:rc.subr[8] provides default methods for the standard arguments. Consequently, we must override a standard method with a no-op man:sh[1] expression if we want it to do nothing.
&#10126; We should keep in mind that man:rc.subr[8] provides default methods for the standard arguments.
Consequently, we must override a standard method with a no-op man:sh[1] expression if we want it to do nothing.
&#10127; The body of a sophisticated method can be implemented as a function. It is a good idea to make the function name meaningful.
&#10127; The body of a sophisticated method can be implemented as a function.
It is a good idea to make the function name meaningful.
[IMPORTANT]
====
It is strongly recommended to add the prefix `${name}` to the names of all functions defined in our script so they never clash with the functions from man:rc.subr[8] or another common include file.
====
&#10128; This call to man:rc.subr[8] loads man:rc.conf[5] variables. Our script makes no use of them yet, but it still is recommended to load man:rc.conf[5] because there can be man:rc.conf[5] variables controlling man:rc.subr[8] itself.
&#10128; This call to man:rc.subr[8] loads man:rc.conf[5] variables.
Our script makes no use of them yet, but it still is recommended to load man:rc.conf[5] because there can be man:rc.conf[5] variables controlling man:rc.subr[8] itself.
&#10129; Usually this is the last command in an [.filename]#rc.d# script. It invokes the man:rc.subr[8] machinery to perform the requested action using the variables and methods our script has provided.
&#10129; Usually this is the last command in an [.filename]#rc.d# script.
It invokes the man:rc.subr[8] machinery to perform the requested action using the variables and methods our script has provided.
[[rcng-confdummy]]
== A configurable dummy script
Now let us add some controls to our dummy script. As you may know, [.filename]#rc.d# scripts are controlled with man:rc.conf[5]. Fortunately, man:rc.subr[8] hides all the complications from us. The following script uses man:rc.conf[5] via man:rc.subr[8] to see whether it is enabled in the first place, and to fetch a message to show at boot time. These two tasks in fact are independent. On the one hand, an [.filename]#rc.d# script can just support enabling and disabling its service. On the other hand, a mandatory [.filename]#rc.d# script can have configuration variables. We will do both things in the same script though:
Now let us add some controls to our dummy script.
As you may know, [.filename]#rc.d# scripts are controlled with man:rc.conf[5].
Fortunately, man:rc.subr[8] hides all the complications from us.
The following script uses man:rc.conf[5] via man:rc.subr[8] to see whether it is enabled in the first place, and to fetch a message to show at boot time.
These two tasks in fact are independent.
On the one hand, an [.filename]#rc.d# script can just support enabling and disabling its service.
On the other hand, a mandatory [.filename]#rc.d# script can have configuration variables.
We will do both things in the same script though:
[.programlisting]
....
@ -178,14 +247,22 @@ What changed in this example?
[NOTE]
====
While examining [.filename]#rc.d# scripts, keep in mind that man:sh[1] defers the evaluation of expressions in a function until the latter is called. Therefore it is not an error to invoke `load_rc_config` as late as just before `run_rc_command` and still access man:rc.conf[5] variables from the method functions exported to `run_rc_command`. This is because the method functions are to be called by `run_rc_command`, which is invoked _after_ `load_rc_config`.
While examining [.filename]#rc.d# scripts, keep in mind that man:sh[1] defers the evaluation of expressions in a function until the latter is called.
Therefore it is not an error to invoke `load_rc_config` as late as just before `run_rc_command` and still access man:rc.conf[5] variables from the method functions exported to `run_rc_command`.
This is because the method functions are to be called by `run_rc_command`, which is invoked _after_ `load_rc_config`.
====
&#10124; A warning will be emitted by `run_rc_command` if `rcvar` itself is set, but the indicated knob variable is unset. If your [.filename]#rc.d# script is for the base system, you should add a default setting for the knob to [.filename]#/etc/defaults/rc.conf# and document it in man:rc.conf[5]. Otherwise it is your script that should provide a default setting for the knob. The canonical approach to the latter case is shown in the example.
&#10124; A warning will be emitted by `run_rc_command` if `rcvar` itself is set, but the indicated knob variable is unset.
If your [.filename]#rc.d# script is for the base system, you should add a default setting for the knob to [.filename]#/etc/defaults/rc.conf# and document it in man:rc.conf[5].
Otherwise it is your script that should provide a default setting for the knob.
The canonical approach to the latter case is shown in the example.
[NOTE]
====
You can make man:rc.subr[8] act as though the knob is set to `ON`, irrespective of its current setting, by prefixing the argument to the script with `one` or `force`, as in `onestart` or `forcestop`. Keep in mind though that `force` has other dangerous effects we will touch upon below, while `one` just overrides the ON/OFF knob. E.g., assume that `dummy_enable` is `OFF`. The following command will run the `start` method in spite of the setting:
You can make man:rc.subr[8] act as though the knob is set to `ON`, irrespective of its current setting, by prefixing the argument to the script with `one` or `force`, as in `onestart` or `forcestop`.
Keep in mind though that `force` has other dangerous effects we will touch upon below, while `one` just overrides the ON/OFF knob.
E.g., assume that `dummy_enable` is `OFF`.
The following command will run the `start` method in spite of the setting:
[source,shell]
....
@ -194,21 +271,26 @@ You can make man:rc.subr[8] act as though the knob is set to `ON`, irrespective
====
&#10125; Now the message to be shown at boot time is no longer hard-coded in the script. It is specified by an man:rc.conf[5] variable named `dummy_msg`. This is a trivial example of how man:rc.conf[5] variables can control an [.filename]#rc.d# script.
&#10125; Now the message to be shown at boot time is no longer hard-coded in the script.
It is specified by an man:rc.conf[5] variable named `dummy_msg`.
This is a trivial example of how man:rc.conf[5] variables can control an [.filename]#rc.d# script.
[IMPORTANT]
====
The names of all man:rc.conf[5] variables used exclusively by our script _must_ have the same prefix: `${name}_`. For example: `dummy_mode`, `dummy_state_file`, and so on.
The names of all man:rc.conf[5] variables used exclusively by our script _must_ have the same prefix: `${name}_`.
For example: `dummy_mode`, `dummy_state_file`, and so on.
====
[NOTE]
====
While it is possible to use a shorter name internally, e.g., just `msg`, adding the unique prefix `${name}_` to all global names introduced by our script will save us from possible collisions with the man:rc.subr[8] namespace.
As a rule, [.filename]#rc.d# scripts of the base system need not provide defaults for their man:rc.conf[5] variables because the defaults should be set in [.filename]#/etc/defaults/rc.conf# instead. On the other hand, [.filename]#rc.d# scripts for ports should provide the defaults as shown in the example.
As a rule, [.filename]#rc.d# scripts of the base system need not provide defaults for their man:rc.conf[5] variables because the defaults should be set in [.filename]#/etc/defaults/rc.conf# instead.
On the other hand, [.filename]#rc.d# scripts for ports should provide the defaults as shown in the example.
====
&#10126; Here we use `dummy_msg` to actually control our script, i.e., to emit a variable message. Use of a shell function is overkill here, since it only runs a single command; an equally valid alternative is:
&#10126; Here we use `dummy_msg` to actually control our script, i.e., to emit a variable message.
Use of a shell function is overkill here, since it only runs a single command; an equally valid alternative is:
[.programlisting]
....
@ -218,7 +300,11 @@ start_cmd="echo \"$dummy_msg\""
[[rcng-daemon]]
== Startup and shutdown of a simple daemon
We said earlier that man:rc.subr[8] could provide default methods. Obviously, such defaults cannot be too general. They are suited for the common case of starting and shutting down a simple daemon program. Let us assume now that we need to write an [.filename]#rc.d# script for such a daemon called `mumbled`. Here it is:
We said earlier that man:rc.subr[8] could provide default methods.
Obviously, such defaults cannot be too general.
They are suited for the common case of starting and shutting down a simple daemon program.
Let us assume now that we need to write an [.filename]#rc.d# script for such a daemon called `mumbled`.
Here it is:
[.programlisting]
....
@ -235,19 +321,37 @@ load_rc_config $name
run_rc_command "$1"
....
Pleasingly simple, isn't it? Let us examine our little script. The only new thing to note is as follows:
Pleasingly simple, isn't it? Let us examine our little script.
The only new thing to note is as follows:
&#10122; The `command` variable is meaningful to man:rc.subr[8]. If it is set, man:rc.subr[8] will act according to the scenario of serving a conventional daemon. In particular, the default methods will be provided for such arguments: `start`, `stop`, `restart`, `poll`, and `status`.
&#10122; The `command` variable is meaningful to man:rc.subr[8].
If it is set, man:rc.subr[8] will act according to the scenario of serving a conventional daemon.
In particular, the default methods will be provided for such arguments: `start`, `stop`, `restart`, `poll`, and `status`.
The daemon will be started by running `$command` with command-line flags specified by `$mumbled_flags`. Thus all the input data for the default `start` method are available in the variables set by our script. Unlike `start`, other methods may require additional information about the process started. For instance, `stop` must know the PID of the process to terminate it. In the present case, man:rc.subr[8] will scan through the list of all processes, looking for a process with its name equal to `procname`. The latter is another variable of meaning to man:rc.subr[8], and its value defaults to that of `command`. In other words, when we set `command`, `procname` is effectively set to the same value. This enables our script to kill the daemon and to check if it is running in the first place.
The daemon will be started by running `$command` with command-line flags specified by `$mumbled_flags`.
Thus all the input data for the default `start` method are available in the variables set by our script.
Unlike `start`, other methods may require additional information about the process started.
For instance, `stop` must know the PID of the process to terminate it.
In the present case, man:rc.subr[8] will scan through the list of all processes, looking for a process with its name equal to `procname`.
The latter is another variable of meaning to man:rc.subr[8], and its value defaults to that of `command`.
In other words, when we set `command`, `procname` is effectively set to the same value.
This enables our script to kill the daemon and to check if it is running in the first place.
[NOTE]
====
Some programs are in fact executable scripts. The system runs such a script by starting its interpreter and passing the name of the script to it as a command-line argument. This is reflected in the list of processes, which can confuse man:rc.subr[8]. You should additionally set `command_interpreter` to let man:rc.subr[8] know the actual name of the process if `$command` is a script.
Some programs are in fact executable scripts.
The system runs such a script by starting its interpreter and passing the name of the script to it as a command-line argument.
This is reflected in the list of processes, which can confuse man:rc.subr[8].
You should additionally set `command_interpreter` to let man:rc.subr[8] know the actual name of the process if `$command` is a script.
For each [.filename]#rc.d# script, there is an optional man:rc.conf[5] variable that takes precedence over `command`. Its name is constructed as follows: `${name}_program`, where `name` is the mandatory variable we discussed <<name-var, earlier>>. E.g., in this case it will be `mumbled_program`. It is man:rc.subr[8] that arranges `${name}_program` to override `command`.
For each [.filename]#rc.d# script, there is an optional man:rc.conf[5] variable that takes precedence over `command`.
Its name is constructed as follows: `${name}_program`, where `name` is the mandatory variable we discussed <<name-var, earlier>>.
E.g., in this case it will be `mumbled_program`.
It is man:rc.subr[8] that arranges `${name}_program` to override `command`.
Of course, man:sh[1] will permit you to set `${name}_program` from man:rc.conf[5] or the script itself even if `command` is unset. In that case, the special properties of `${name}_program` are lost, and it becomes an ordinary variable your script can use for its own purposes. However, the sole use of `${name}_program` is discouraged because using it together with `command` became an idiom of [.filename]#rc.d# scripting.
Of course, man:sh[1] will permit you to set `${name}_program` from man:rc.conf[5] or the script itself even if `command` is unset.
In that case, the special properties of `${name}_program` are lost, and it becomes an ordinary variable your script can use for its own purposes.
However, the sole use of `${name}_program` is discouraged because using it together with `command` became an idiom of [.filename]#rc.d# scripting.
====
For more detailed information on default methods, refer to man:rc.subr[8].
@ -255,7 +359,9 @@ For more detailed information on default methods, refer to man:rc.subr[8].
[[rcng-daemon-adv]]
== Startup and shutdown of an advanced daemon
Let us add some meat onto the bones of the previous script and make it more complex and featureful. The default methods can do a good job for us, but we may need some of their aspects tweaked. Now we will learn how to tune the default methods to our needs.
Let us add some meat onto the bones of the previous script and make it more complex and featureful.
The default methods can do a good job for us, but we may need some of their aspects tweaked.
Now we will learn how to tune the default methods to our needs.
[.programlisting]
....
@ -313,57 +419,86 @@ load_rc_config $name
run_rc_command "$1"
....
&#10122; Additional arguments to `$command` can be passed in `command_args`. They will be added to the command line after `$mumbled_flags`. Since the final command line is passed to `eval` for its actual execution, input and output redirections can be specified in `command_args`.
&#10122; Additional arguments to `$command` can be passed in `command_args`.
They will be added to the command line after `$mumbled_flags`.
Since the final command line is passed to `eval` for its actual execution, input and output redirections can be specified in `command_args`.
[NOTE]
====
_Never_ include dashed options, like `-X` or `--foo`, in `command_args`. The contents of `command_args` will appear at the end of the final command line, hence they are likely to follow arguments present in `${name}_flags`; but most commands will not recognize dashed options after ordinary arguments. A better way of passing additional options to `$command` is to add them to the beginning of `${name}_flags`. Another way is to modify `rc_flags` <<rc-flags, as shown later>>.
_Never_ include dashed options, like `-X` or `--foo`, in `command_args`.
The contents of `command_args` will appear at the end of the final command line, hence they are likely to follow arguments present in `${name}_flags`; but most commands will not recognize dashed options after ordinary arguments.
A better way of passing additional options to `$command` is to add them to the beginning of `${name}_flags`.
Another way is to modify `rc_flags` <<rc-flags, as shown later>>.
====
&#10123; A good-mannered daemon should create a _pidfile_ so that its process can be found more easily and reliably. The variable `pidfile`, if set, tells man:rc.subr[8] where it can find the pidfile for its default methods to use.
&#10123; A good-mannered daemon should create a _pidfile_ so that its process can be found more easily and reliably.
The variable `pidfile`, if set, tells man:rc.subr[8] where it can find the pidfile for its default methods to use.
[NOTE]
====
In fact, man:rc.subr[8] will also use the pidfile to see if the daemon is already running before starting it. This check can be skipped by using the `faststart` argument.
In fact, man:rc.subr[8] will also use the pidfile to see if the daemon is already running before starting it.
This check can be skipped by using the `faststart` argument.
====
&#10124; If the daemon cannot run unless certain files exist, just list them in `required_files`, and man:rc.subr[8] will check that those files do exist before starting the daemon. There also are `required_dirs` and `required_vars` for directories and environment variables, respectively. They all are described in detail in man:rc.subr[8].
&#10124; If the daemon cannot run unless certain files exist, just list them in `required_files`, and man:rc.subr[8] will check that those files do exist before starting the daemon.
There also are `required_dirs` and `required_vars` for directories and environment variables, respectively.
They all are described in detail in man:rc.subr[8].
[NOTE]
====
The default method from man:rc.subr[8] can be forced to skip the prerequisite checks by using `forcestart` as the argument to the script.
====
&#10125; We can customize signals to send to the daemon in case they differ from the well-known ones. In particular, `sig_reload` specifies the signal that makes the daemon reload its configuration; it is SIGHUP by default. Another signal is sent to stop the daemon process; the default is SIGTERM, but this can be changed by setting `sig_stop` appropriately.
&#10125; We can customize signals to send to the daemon in case they differ from the well-known ones.
In particular, `sig_reload` specifies the signal that makes the daemon reload its configuration; it is SIGHUP by default.
Another signal is sent to stop the daemon process;
the default is SIGTERM, but this can be changed by setting `sig_stop` appropriately.
[NOTE]
====
The signal names should be specified to man:rc.subr[8] without the `SIG` prefix, as it is shown in the example. The FreeBSD version of man:kill[1] can recognize the `SIG` prefix, but the versions from other OS types may not.
The signal names should be specified to man:rc.subr[8] without the `SIG` prefix, as it is shown in the example.
The FreeBSD version of man:kill[1] can recognize the `SIG` prefix, but the versions from other OS types may not.
====
&#10126;&#10127; Performing additional tasks before or after the default methods is easy. For each command-argument supported by our script, we can define `argument_precmd` and `argument_postcmd`. These man:sh[1] commands are invoked before and after the respective method, as it is evident from their names.
&#10126;&#10127; Performing additional tasks before or after the default methods is easy.
For each command-argument supported by our script, we can define `argument_precmd` and `argument_postcmd`.
These man:sh[1] commands are invoked before and after the respective method, as it is evident from their names.
[NOTE]
====
Overriding a default method with a custom `argument_cmd` still does not prevent us from making use of `argument_precmd` or `argument_postcmd` if we need to. In particular, the former is good for checking custom, sophisticated conditions that should be met before performing the command itself. Using `argument_precmd` along with `argument_cmd` lets us logically separate the checks from the action.
Overriding a default method with a custom `argument_cmd` still does not prevent us from making use of `argument_precmd` or `argument_postcmd` if we need to.
In particular, the former is good for checking custom, sophisticated conditions that should be met before performing the command itself.
Using `argument_precmd` along with `argument_cmd` lets us logically separate the checks from the action.
Do not forget that you can cram any valid man:sh[1] expressions into the methods, pre-, and post-commands you define. Just invoking a function that makes the real job is a good style in most cases, but never let style limit your understanding of what is going on behind the curtain.
Do not forget that you can cram any valid man:sh[1] expressions into the methods, pre-, and post-commands you define.
Just invoking a function that makes the real job is a good style in most cases, but never let style limit your understanding of what is going on behind the curtain.
====
&#10128; If we would like to implement custom arguments, which can also be thought of as _commands_ to our script, we need to list them in `extra_commands` and provide methods to handle them.
[NOTE]
====
The `reload` command is special. On the one hand, it has a preset method in man:rc.subr[8]. On the other hand, `reload` is not offered by default. The reason is that not all daemons use the same reload mechanism and some have nothing to reload at all. So we need to ask explicitly that the builtin functionality be provided. We can do so via `extra_commands`.
The `reload` command is special. On the one hand, it has a preset method in man:rc.subr[8].
On the other hand, `reload` is not offered by default.
The reason is that not all daemons use the same reload mechanism and some have nothing to reload at all.
So we need to ask explicitly that the builtin functionality be provided.
We can do so via `extra_commands`.
What do we get from the default method for `reload`? Quite often daemons reload their configuration upon reception of a signal - typically, SIGHUP. Therefore man:rc.subr[8] attempts to reload the daemon by sending a signal to it. The signal is preset to SIGHUP but can be customized via `sig_reload` if necessary.
What do we get from the default method for `reload`? Quite often daemons reload their configuration upon reception of a signal - typically, SIGHUP.
Therefore man:rc.subr[8] attempts to reload the daemon by sending a signal to it.
The signal is preset to SIGHUP but can be customized via `sig_reload` if necessary.
====
&#10129;&#9454; Our script supports two non-standard commands, `plugh` and `xyzzy`. We saw them listed in `extra_commands`, and now it is time to provide methods for them. The method for `xyzzy` is just inlined while that for `plugh` is implemented as the `mumbled_plugh` function.
&#10129;&#9454; Our script supports two non-standard commands, `plugh` and `xyzzy`.
We saw them listed in `extra_commands`, and now it is time to provide methods for them.
The method for `xyzzy` is just inlined while that for `plugh` is implemented as the `mumbled_plugh` function.
Non-standard commands are not invoked during startup or shutdown. Usually they are for the system admin's convenience. They can also be used from other subsystems, e.g., man:devd[8] if specified in man:devd.conf[5].
Non-standard commands are not invoked during startup or shutdown.
Usually they are for the system admin's convenience.
They can also be used from other subsystems, e.g., man:devd[8] if specified in man:devd.conf[5].
The full list of available commands can be found in the usage line printed by man:rc.subr[8] when the script is invoked without arguments. For example, here is the usage line from the script under study:
The full list of available commands can be found in the usage line printed by man:rc.subr[8] when the script is invoked without arguments.
For example, here is the usage line from the script under study:
[source,shell]
....
@ -371,15 +506,24 @@ The full list of available commands can be found in the usage line printed by ma
Usage: /etc/rc.d/mumbled [fast|force|one](start|stop|restart|rcvar|reload|plugh|xyzzy|status|poll)
....
&#9453; A script can invoke its own standard or non-standard commands if needed. This may look similar to calling functions, but we know that commands and shell functions are not always the same thing. For instance, `xyzzy` is not implemented as a function here. In addition, there can be a pre-command and post-command, which should be invoked orderly. So the proper way for a script to run its own command is by means of man:rc.subr[8], as shown in the example.
&#9453; A script can invoke its own standard or non-standard commands if needed.
This may look similar to calling functions, but we know that commands and shell functions are not always the same thing.
For instance, `xyzzy` is not implemented as a function here.
In addition, there can be a pre-command and post-command, which should be invoked orderly.
So the proper way for a script to run its own command is by means of man:rc.subr[8], as shown in the example.
&#10130; A handy function named `checkyesno` is provided by man:rc.subr[8]. It takes a variable name as its argument and returns a zero exit code if and only if the variable is set to `YES`, or `TRUE`, or `ON`, or `1`, case insensitive; a non-zero exit code is returned otherwise. In the latter case, the function tests the variable for being set to `NO`, `FALSE`, `OFF`, or `0`, case insensitive; it prints a warning message if the variable contains anything else, i.e., junk.
&#10130; A handy function named `checkyesno` is provided by man:rc.subr[8].
It takes a variable name as its argument and returns a zero exit code if and only if the variable is set to `YES`, or `TRUE`, or `ON`, or `1`, case insensitive;
a non-zero exit code is returned otherwise.
In the latter case, the function tests the variable for being set to `NO`, `FALSE`, `OFF`, or `0`, case insensitive;
it prints a warning message if the variable contains anything else, i.e., junk.
Keep in mind that for man:sh[1] a zero exit code means true and a non-zero exit code means false.
[IMPORTANT]
====
The `checkyesno` function takes a __variable name__. Do not pass the expanded _value_ of a variable to it; it will not work as expected.
The `checkyesno` function takes a __variable name__.
Do not pass the expanded _value_ of a variable to it; it will not work as expected.
The following is the correct usage of `checkyesno`:
@ -403,9 +547,13 @@ fi
&#10131; [[rc-flags]]We can affect the flags to be passed to `$command` by modifying `rc_flags` in `$start_precmd`.
&#9451; In certain cases we may need to emit an important message that should go to `syslog` as well. This can be done easily with the following man:rc.subr[8] functions: `debug`, `info`, `warn`, and `err`. The latter function then exits the script with the code specified.
&#9451; In certain cases we may need to emit an important message that should go to `syslog` as well.
This can be done easily with the following man:rc.subr[8] functions: `debug`, `info`, `warn`, and `err`.
The latter function then exits the script with the code specified.
&#9452; The exit codes from methods and their pre-commands are not just ignored by default. If `argument_precmd` returns a non-zero exit code, the main method will not be performed. In turn, `argument_postcmd` will not be invoked unless the main method returns a zero exit code.
&#9452; The exit codes from methods and their pre-commands are not just ignored by default.
If `argument_precmd` returns a non-zero exit code, the main method will not be performed.
In turn, `argument_postcmd` will not be invoked unless the main method returns a zero exit code.
[NOTE]
====
@ -415,20 +563,35 @@ However, man:rc.subr[8] can be instructed from the command line to ignore those
[[rcng-hookup]]
== Connecting a script to the rc.d framework
After a script has been written, it needs to be integrated into [.filename]#rc.d#. The crucial step is to install the script in [.filename]#/etc/rc.d# (for the base system) or [.filename]#/usr/local/etc/rc.d# (for ports). Both [.filename]#bsd.prog.mk# and [.filename]#bsd.port.mk# provide convenient hooks for that, and usually you do not have to worry about the proper ownership and mode. System scripts should be installed from [.filename]#src/etc/rc.d# through the [.filename]#Makefile# found there. Port scripts can be installed using `USE_RC_SUBR` as described link:{porters-handbook}#rc-scripts[in the Porter's Handbook].
After a script has been written, it needs to be integrated into [.filename]#rc.d#.
The crucial step is to install the script in [.filename]#/etc/rc.d# (for the base system) or [.filename]#/usr/local/etc/rc.d# (for ports).
Both [.filename]#bsd.prog.mk# and [.filename]#bsd.port.mk# provide convenient hooks for that, and usually you do not have to worry about the proper ownership and mode.
System scripts should be installed from [.filename]#src/etc/rc.d# through the [.filename]#Makefile# found there.
Port scripts can be installed using `USE_RC_SUBR` as described link:{porters-handbook}#rc-scripts[in the Porter's Handbook].
However, we should consider beforehand the place of our script in the system startup sequence. The service handled by our script is likely to depend on other services. For instance, a network daemon cannot function without the network interfaces and routing up and running. Even if a service seems to demand nothing, it can hardly start before the basic filesystems have been checked and mounted.
However, we should consider beforehand the place of our script in the system startup sequence.
The service handled by our script is likely to depend on other services.
For instance, a network daemon cannot function without the network interfaces and routing up and running.
Even if a service seems to demand nothing, it can hardly start before the basic filesystems have been checked and mounted.
We mentioned man:rcorder[8] already. Now it is time to have a close look at it. In a nutshell, man:rcorder[8] takes a set of files, examines their contents, and prints a dependency-ordered list of files from the set to `stdout`. The point is to keep dependency information _inside_ the files so that each file can speak for itself only. A file can specify the following information:
We mentioned man:rcorder[8] already.
Now it is time to have a close look at it.
In a nutshell, man:rcorder[8] takes a set of files, examines their contents, and prints a dependency-ordered list of files from the set to `stdout`.
The point is to keep dependency information _inside_ the files so that each file can speak for itself only.
A file can specify the following information:
* the names of the "conditions" (which means services to us) it __provides__;
* the names of the "conditions" it __requires__;
* the names of the "conditions" this file should run __before__;
* additional _keywords_ that can be used to select a subset from the whole set of files (man:rcorder[8] can be instructed via options to include or omit the files having particular keywords listed.)
It is no surprise that man:rcorder[8] can handle only text files with a syntax close to that of man:sh[1]. That is, special lines understood by man:rcorder[8] look like man:sh[1] comments. The syntax of such special lines is rather rigid to simplify their processing. See man:rcorder[8] for details.
It is no surprise that man:rcorder[8] can handle only text files with a syntax close to that of man:sh[1].
That is, special lines understood by man:rcorder[8] look like man:sh[1] comments.
The syntax of such special lines is rather rigid to simplify their processing.
See man:rcorder[8] for details.
Besides using man:rcorder[8] special lines, a script can insist on its dependency upon another service by just starting it forcibly. This can be needed when the other service is optional and will not start by itself because the system admin has disabled it mistakenly in man:rc.conf[5].
Besides using man:rcorder[8] special lines, a script can insist on its dependency upon another service by just starting it forcibly.
This can be needed when the other service is optional and will not start by itself because the system admin has disabled it mistakenly in man:rc.conf[5].
With this general knowledge in mind, let us consider the simple daemon script enhanced with dependency stuff:
@ -464,46 +627,81 @@ run_rc_command "$1"
As before, detailed analysis follows:
&#10122; That line declares the names of "conditions" our script provides. Now other scripts can record a dependency on our script by those names.
&#10122; That line declares the names of "conditions" our script provides.
Now other scripts can record a dependency on our script by those names.
[NOTE]
====
Usually a script specifies a single condition provided. However, nothing prevents us from listing several conditions there, e.g., for compatibility reasons.
Usually a script specifies a single condition provided.
However, nothing prevents us from listing several conditions there, e.g., for compatibility reasons.
In any case, the name of the main, or the only, `PROVIDE:` condition should be the same as `${name}`.
====
&#10123;&#10124; So our script indicates which "conditions" provided by other scripts it depends on. According to the lines, our script asks man:rcorder[8] to put it after the script(s) providing [.filename]#DAEMON# and [.filename]#cleanvar#, but before that providing [.filename]#LOGIN#.
&#10123;&#10124; So our script indicates which "conditions" provided by other scripts it depends on.
According to the lines, our script asks man:rcorder[8] to put it after the script(s) providing [.filename]#DAEMON# and [.filename]#cleanvar#, but before that providing [.filename]#LOGIN#.
[NOTE]
====
The `BEFORE:` line should not be abused to work around an incomplete dependency list in the other script. The appropriate case for using `BEFORE:` is when the other script does not care about ours, but our script can do its task better if run before the other one. A typical real-life example is the network interfaces vs. the firewall: While the interfaces do not depend on the firewall in doing their job, the system security will benefit from the firewall being ready before there is any network traffic.
The `BEFORE:` line should not be abused to work around an incomplete dependency list in the other script.
The appropriate case for using `BEFORE:` is when the other script does not care about ours, but our script can do its task better if run before the other one.
A typical real-life example is the network interfaces vs. the firewall: While the interfaces do not depend on the firewall in doing their job, the system security will benefit from the firewall being ready before there is any network traffic.
Besides conditions corresponding to a single service each, there are meta-conditions and their "placeholder" scripts used to ensure that certain groups of operations are performed before others. These are denoted by [.filename]#UPPERCASE# names. Their list and purposes can be found in man:rc[8].
Besides conditions corresponding to a single service each, there are meta-conditions and their "placeholder" scripts used to ensure that certain groups of operations are performed before others.
These are denoted by [.filename]#UPPERCASE# names.
Their list and purposes can be found in man:rc[8].
Keep in mind that putting a service name in the `REQUIRE:` line does not guarantee that the service will actually be running by the time our script starts. The required service may fail to start or just be disabled in man:rc.conf[5]. Obviously, man:rcorder[8] cannot track such details, and man:rc[8] will not do that either. Consequently, the application started by our script should be able to cope with any required services being unavailable. In certain cases, we can help it as discussed <<forcedep, below>>
Keep in mind that putting a service name in the `REQUIRE:` line does not guarantee that the service will actually be running by the time our script starts.
The required service may fail to start or just be disabled in man:rc.conf[5].
Obviously, man:rcorder[8] cannot track such details, and man:rc[8] will not do that either.
Consequently, the application started by our script should be able to cope with any required services being unavailable.
In certain cases, we can help it as discussed <<forcedep, below>>
====
[[keywords]]&#10125; As we remember from the above text, man:rcorder[8] keywords can be used to select or leave out some scripts. Namely any man:rcorder[8] consumer can specify through `-k` and `-s` options which keywords are on the "keep list" and "skip list", respectively. From all the files to be dependency sorted, man:rcorder[8] will pick only those having a keyword from the keep list (unless empty) and not having a keyword from the skip list.
[[keywords]]&#10125; As we remember from the above text, man:rcorder[8] keywords can be used to select or leave out some scripts.
Namely any man:rcorder[8] consumer can specify through `-k` and `-s` options which keywords are on the "keep list" and "skip list", respectively.
From all the files to be dependency sorted, man:rcorder[8] will pick only those having a keyword from the keep list (unless empty) and not having a keyword from the skip list.
In FreeBSD, man:rcorder[8] is used by [.filename]#/etc/rc# and [.filename]#/etc/rc.shutdown#. These two scripts define the standard list of FreeBSD [.filename]#rc.d# keywords and their meanings as follows:
In FreeBSD, man:rcorder[8] is used by [.filename]#/etc/rc# and [.filename]#/etc/rc.shutdown#.
These two scripts define the standard list of FreeBSD [.filename]#rc.d# keywords and their meanings as follows:
[[forcedep]]&#10126; To begin with, `force_depend` should be used with much care. It is generally better to revise the hierarchy of configuration variables for your [.filename]#rc.d# scripts if they are interdependent.
[[forcedep]]&#10126; To begin with, `force_depend` should be used with much care.
It is generally better to revise the hierarchy of configuration variables for your [.filename]#rc.d# scripts if they are interdependent.
If you still cannot do without `force_depend`, the example offers an idiom of how to invoke it conditionally. In the example, our `mumbled` daemon requires that another one, `frotz`, be started in advance. However, `frotz` is optional, too; and man:rcorder[8] knows nothing about such details. Fortunately, our script has access to all man:rc.conf[5] variables. If `frotz_enable` is true, we hope for the best and rely on [.filename]#rc.d# to have started `frotz`. Otherwise we forcibly check the status of `frotz`. Finally, we enforce our dependency on `frotz` if it is found to be not running. A warning message will be emitted by `force_depend` because it should be invoked only if a misconfiguration has been detected.
If you still cannot do without `force_depend`, the example offers an idiom of how to invoke it conditionally.
In the example, our `mumbled` daemon requires that another one, `frotz`, be started in advance.
However, `frotz` is optional, too; and man:rcorder[8] knows nothing about such details.
Fortunately, our script has access to all man:rc.conf[5] variables.
If `frotz_enable` is true, we hope for the best and rely on [.filename]#rc.d# to have started `frotz`.
Otherwise we forcibly check the status of `frotz`.
Finally, we enforce our dependency on `frotz` if it is found to be not running.
A warning message will be emitted by `force_depend` because it should be invoked only if a misconfiguration has been detected.
[[rcng-args]]
== Giving more flexibility to an rc.d script
When invoked during startup or shutdown, an [.filename]#rc.d# script is supposed to act on the entire subsystem it is responsible for. E.g., [.filename]#/etc/rc.d/netif# should start or stop all network interfaces described by man:rc.conf[5]. Either task can be uniquely indicated by a single command argument such as `start` or `stop`. Between startup and shutdown, [.filename]#rc.d# scripts help the admin to control the running system, and it is when the need for more flexibility and precision arises. For instance, the admin may want to add the settings of a new network interface to man:rc.conf[5] and then to start it without interfering with the operation of the existing interfaces. Next time the admin may need to shut down a single network interface. In the spirit of the command line, the respective [.filename]#rc.d# script calls for an extra argument, the interface name.
When invoked during startup or shutdown, an [.filename]#rc.d# script is supposed to act on the entire subsystem it is responsible for.
E.g., [.filename]#/etc/rc.d/netif# should start or stop all network interfaces described by man:rc.conf[5].
Either task can be uniquely indicated by a single command argument such as `start` or `stop`.
Between startup and shutdown, [.filename]#rc.d# scripts help the admin to control the running system, and it is when the need for more flexibility and precision arises.
For instance, the admin may want to add the settings of a new network interface to man:rc.conf[5] and then to start it without interfering with the operation of the existing interfaces.
Next time the admin may need to shut down a single network interface.
In the spirit of the command line, the respective [.filename]#rc.d# script calls for an extra argument, the interface name.
Fortunately, man:rc.subr[8] allows for passing any number of arguments to script's methods (within the system limits). Due to that, the changes in the script itself can be minimal.
Fortunately, man:rc.subr[8] allows for passing any number of arguments to script's methods (within the system limits).
Due to that, the changes in the script itself can be minimal.
How can man:rc.subr[8] gain access to the extra command-line arguments. Should it just grab them directly? Not by any means. Firstly, an man:sh[1] function has no access to the positional parameters of its caller, but man:rc.subr[8] is just a sack of such functions. Secondly, the good manner of [.filename]#rc.d# dictates that it is for the main script to decide which arguments are to be passed to its methods.
How can man:rc.subr[8] gain access to the extra command-line arguments.
Should it just grab them directly? Not by any means.
Firstly, an man:sh[1] function has no access to the positional parameters of its caller, but man:rc.subr[8] is just a sack of such functions.
Secondly, the good manner of [.filename]#rc.d# dictates that it is for the main script to decide which arguments are to be passed to its methods.
So the approach adopted by man:rc.subr[8] is as follows: `run_rc_command` passes on all its arguments but the first one to the respective method verbatim. The first, omitted, argument is the name of the method itself: `start`, `stop`, etc. It will be shifted out by `run_rc_command`, so what is `$2` in the original command line will be presented as `$1` to the method, and so on.
So the approach adopted by man:rc.subr[8] is as follows: `run_rc_command` passes on all its arguments but the first one to the respective method verbatim.
The first, omitted, argument is the name of the method itself: `start`, `stop`, etc.
It will be shifted out by `run_rc_command`, so what is `$2` in the original command line will be presented as `$1` to the method, and so on.
To illustrate this opportunity, let us modify the primitive dummy script so that its messages depend on the additional arguments supplied. Here we go:
To illustrate this opportunity, let us modify the primitive dummy script so that its messages depend on the additional arguments supplied.
Here we go:
[.programlisting]
....
@ -548,7 +746,10 @@ run_rc_command "$@" <.>
What essential changes can we notice in the script?
&#10122; All arguments you type after `start` can end up as positional parameters to the respective method. We can use them in any way according to our task, skills, and fancy. In the current example, we just pass all of them to man:echo[1] as one string in the next line - note `$*` within the double quotes. Here is how the script can be invoked now:
&#10122; All arguments you type after `start` can end up as positional parameters to the respective method.
We can use them in any way according to our task, skills, and fancy.
In the current example, we just pass all of them to man:echo[1] as one string in the next line - note `$*` within the double quotes.
Here is how the script can be invoked now:
[source,shell]
....
@ -559,7 +760,8 @@ Nothing started.
Greeting message: Hello world!
....
&#10123; The same applies to any method our script provides, not only to a standard one. We have added a custom method named `kiss`, and it can take advantage of the extra arguments not less than `start` does. E.g.:
&#10123; The same applies to any method our script provides, not only to a standard one.
We have added a custom method named `kiss`, and it can take advantage of the extra arguments not less than `start` does. E.g.:
[source,shell]
....
@ -574,19 +776,28 @@ A ghost gives you a kiss and whispers: Once I was Etaoin Shrdlu...
[IMPORTANT]
====
An man:sh[1] programmer ought to understand the subtle difference between `$*` and `$@` as the ways to designate all positional parameters. For its in-depth discussion, refer to a good handbook on man:sh[1] scripting. _Do not_ use the expressions until you fully understand them because their misuse will result in buggy and insecure scripts.
An man:sh[1] programmer ought to understand the subtle difference between `$*` and `$@` as the ways to designate all positional parameters.
For its in-depth discussion, refer to a good handbook on man:sh[1] scripting.
_Do not_ use the expressions until you fully understand them because their misuse will result in buggy and insecure scripts.
====
[NOTE]
====
Currently `run_rc_command` may have a bug that prevents it from keeping the original boundaries between arguments. That is, arguments with embedded whitespace may not be processed correctly. The bug stems from `$*` misuse.
Currently `run_rc_command` may have a bug that prevents it from keeping the original boundaries between arguments.
That is, arguments with embedded whitespace may not be processed correctly.
The bug stems from `$*` misuse.
====
[[rcng-furthur]]
== Further reading
[[lukem]]http://www.mewburn.net/luke/papers/rc.d.pdf[The original article by Luke Mewburn] offers a general overview of [.filename]#rc.d# and detailed rationale for its design decisions. It provides insight on the whole [.filename]#rc.d# framework and its place in a modern BSD operating system.
[[lukem]]http://www.mewburn.net/luke/papers/rc.d.pdf[The original article by Luke Mewburn] offers a general overview of [.filename]#rc.d# and detailed rationale for its design decisions.
It provides insight on the whole [.filename]#rc.d# framework and its place in a modern BSD operating system.
[[manpages]]The manual pages man:rc[8], man:rc.subr[8], and man:rcorder[8] document the [.filename]#rc.d# components in great detail. You cannot fully use the [.filename]#rc.d# power without studying the manual pages and referring to them while writing your own scripts.
[[manpages]]The manual pages man:rc[8], man:rc.subr[8], and man:rcorder[8] document the [.filename]#rc.d# components in great detail.
You cannot fully use the [.filename]#rc.d# power without studying the manual pages and referring to them while writing your own scripts.
The major source of working, real-life examples is [.filename]#/etc/rc.d# in a live system. Its contents are easy and pleasant to read because most rough corners are hidden deep in man:rc.subr[8]. Keep in mind though that the [.filename]#/etc/rc.d# scripts were not written by angels, so they might suffer from bugs and suboptimal design decisions. Now you can improve them!
The major source of working, real-life examples is [.filename]#/etc/rc.d# in a live system.
Its contents are easy and pleasant to read because most rough corners are hidden deep in man:rc.subr[8].
Keep in mind though that the [.filename]#/etc/rc.d# scripts were not written by angels, so they might suffer from bugs and suboptimal design decisions.
Now you can improve them!

166
documentation/content/en/articles/releng/_index.adoc
View file

@ -25,7 +25,7 @@ include::shared/en/mailing-lists.adoc[]
include::shared/en/urls.adoc[]
ifeval::["{backend}" == "html5"]
:imagesdir: ../../images/articles/releng/
:imagesdir: ../../../images/articles/releng/
endif::[]
ifeval::["{backend}" == "pdf"]
@ -41,10 +41,13 @@ Abstract
[NOTE]
====
This document is outdated and does not accurately describe the current release procedures of the FreeBSD Release Engineering team. It is retained for historical purposes. The current procedures used by the FreeBSD Release Engineering team are available in the link:{freebsd-releng}[FreeBSD Release Engineering] article.
This document is outdated and does not accurately describe the current release procedures of the FreeBSD Release Engineering team.
It is retained for historical purposes.
The current procedures used by the FreeBSD Release Engineering team are available in the link:{freebsd-releng}[FreeBSD Release Engineering] article.
====
This paper describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It details the methodology used for the official FreeBSD releases and describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization.
This paper describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System.
It details the methodology used for the official FreeBSD releases and describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization.
'''
@ -53,23 +56,46 @@ toc::[]
[[introduction]]
== Introduction
The development of FreeBSD is a very open process. FreeBSD is comprised of contributions from thousands of people around the world. The FreeBSD Project provides Subversion footnote:[Subversion, http://subversion.apache.org] access to the general public so that others can have access to log messages, diffs (patches) between development branches, and other productivity enhancements that formal source code management provides. This has been a huge help in attracting more talented developers to FreeBSD. However, I think everyone would agree that chaos would soon manifest if write access to the main repository was opened up to everyone on the Internet. Therefore only a "select" group of nearly 300 people are given write access to the Subversion repository. These link:{contributors}#staff-committers[FreeBSD committers]footnote:[link:{contributors}#staff-committers[FreeBSD committers]] are usually the people who do the bulk of FreeBSD development. An elected link:https://www.FreeBSD.org/administration/#t-core[Core Team]footnote:[link:https://www.FreeBSD.org/administration/#t-core[FreeBSD Core Team]] of developers provide some level of direction over the project.
The development of FreeBSD is a very open process.
FreeBSD is comprised of contributions from thousands of people around the world.
The FreeBSD Project provides Subversion footnote:[Subversion, http://subversion.apache.org] access to the general public so that others can have access to log messages, diffs (patches) between development branches, and other productivity enhancements that formal source code management provides.
This has been a huge help in attracting more talented developers to FreeBSD.
However, I think everyone would agree that chaos would soon manifest if write access to the main repository was opened up to everyone on the Internet.
Therefore only a "select" group of nearly 300 people are given write access to the Subversion repository.
These link:{contributors}#staff-committers[FreeBSD committers]footnote:[link:{contributors}#staff-committers[FreeBSD committers]] are usually the people who do the bulk of FreeBSD development.
An elected link:https://www.FreeBSD.org/administration/#t-core[Core Team]footnote:[link:https://www.FreeBSD.org/administration/#t-core[FreeBSD Core Team]] of developers provide some level of direction over the project.
The rapid pace of `FreeBSD` development makes the main development branch unsuitable for the everyday use by the general public. In particular, stabilizing efforts are required for polishing the development system into a production quality release. To solve this conflict, development continues on several parallel tracks. The main development branch is the _HEAD_ or _trunk_ of our Subversion tree, known as "FreeBSD-CURRENT" or "-CURRENT" for short.
The rapid pace of `FreeBSD` development makes the main development branch unsuitable for the everyday use by the general public.
In particular, stabilizing efforts are required for polishing the development system into a production quality release.
To solve this conflict, development continues on several parallel tracks.
The main development branch is the _HEAD_ or _trunk_ of our Subversion tree, known as "FreeBSD-CURRENT" or "-CURRENT" for short.
A set of more stable branches are maintained, known as "FreeBSD-STABLE" or "-STABLE" for short. All branches live in a master Subversion repository maintained by the FreeBSD Project. FreeBSD-CURRENT is the "bleeding-edge" of FreeBSD development where all new changes first enter the system. FreeBSD-STABLE is the development branch from which major releases are made. Changes go into this branch at a different pace, and with the general assumption that they have first gone into FreeBSD-CURRENT and have been thoroughly tested by our user community.
A set of more stable branches are maintained, known as "FreeBSD-STABLE" or "-STABLE" for short.
All branches live in a master Subversion repository maintained by the FreeBSD Project.
FreeBSD-CURRENT is the "bleeding-edge" of FreeBSD development where all new changes first enter the system.
FreeBSD-STABLE is the development branch from which major releases are made.
Changes go into this branch at a different pace, and with the general assumption that they have first gone into FreeBSD-CURRENT and have been thoroughly tested by our user community.
The term _stable_ in the name of the branch refers to the presumed Application Binary Interface stability, which is promised by the project. This means that a user application compiled on an older version of the system from the same branch works on a newer system from the same branch. The ABI stability has improved greatly from the compared to previous releases. In most cases, binaries from the older _STABLE_ systems run unmodified on newer systems, including __HEAD__, assuming that the system management interfaces are not used.
The term _stable_ in the name of the branch refers to the presumed Application Binary Interface stability, which is promised by the project.
This means that a user application compiled on an older version of the system from the same branch works on a newer system from the same branch.
The ABI stability has improved greatly from the compared to previous releases.
In most cases, binaries from the older _STABLE_ systems run unmodified on newer systems, including __HEAD__, assuming that the system management interfaces are not used.
In the interim period between releases, weekly snapshots are built automatically by the FreeBSD Project build machines and made available for download from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/`. The widespread availability of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with Subversion and "`make buildworld`" footnote:[link:{handbook}#makeworld[Rebuilding world]] helps to keep FreeBSD-STABLE in a very reliable condition even before the quality assurance activities ramp up pending a major release.
In the interim period between releases, weekly snapshots are built automatically by the FreeBSD Project build machines and made available for download from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/`.
The widespread availability of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with Subversion and "`make buildworld`" footnote:[link:{handbook}#makeworld[Rebuilding world]] helps to keep FreeBSD-STABLE in a very reliable condition even before the quality assurance activities ramp up pending a major release.
In addition to installation ISO snapshots, weekly virtual machine images are also provided for use with VirtualBox, qemu, or other popular emulation software. The virtual machine images can be downloaded from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/VM-IMAGES/`.
In addition to installation ISO snapshots, weekly virtual machine images are also provided for use with VirtualBox, qemu, or other popular emulation software.
The virtual machine images can be downloaded from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/VM-IMAGES/`.
The virtual machine images are approximately 150MB man:xz[1] compressed, and contain a 10GB sparse filesystem when attached to a virtual machine.
Bug reports and feature requests are continuously submitted by users throughout the release cycle. Problems reports are entered into our Bugzilla database through the web interface provided at https://www.freebsd.org/support/bugreports/[https://www.freebsd.org/support/bugreports/].
Bug reports and feature requests are continuously submitted by users throughout the release cycle.
Problems reports are entered into our Bugzilla database through the web interface provided at https://www.freebsd.org/support/bugreports/[https://www.freebsd.org/support/bugreports/].
To service our most conservative users, individual release branches were introduced with FreeBSD 4.3. These release branches are created shortly before a final release is made. After the release goes out, only the most critical security fixes and additions are merged onto the release branch. In addition to source updates via Subversion, binary patchkits are available to keep systems on the _releng/X.Y_ branches updated.
To service our most conservative users, individual release branches were introduced with FreeBSD 4.3.
These release branches are created shortly before a final release is made.
After the release goes out, only the most critical security fixes and additions are merged onto the release branch.
In addition to source updates via Subversion, binary patchkits are available to keep systems on the _releng/X.Y_ branches updated.
=== What This Article Describes
@ -93,13 +119,20 @@ Future directions of development.
[[release-proc]]
== Release Process
New releases of FreeBSD are released from the -STABLE branch at approximately four month intervals. The FreeBSD release process begins to ramp up 70-80 days before the anticipated release date when the release engineer sends an email to the development mailing lists to remind developers that they only have 15 days to integrate new changes before the code freeze. During this time, many developers perform what have become known as "MFC sweeps".
New releases of FreeBSD are released from the -STABLE branch at approximately four month intervals.
The FreeBSD release process begins to ramp up 70-80 days before the anticipated release date when the release engineer sends an email to the development mailing lists to remind developers that they only have 15 days to integrate new changes before the code freeze.
During this time, many developers perform what have become known as "MFC sweeps".
MFC stands for "Merge From CURRENT" and it describes the process of merging a tested change from our -CURRENT development branch to our -STABLE branch. Project policy requires any change to be first applied to trunk, and merged to the -STABLE branches after sufficient external testing was done by -CURRENT users (developers are expected to extensively test the change before committing to -CURRENT, but it is impossible for a person to exercise all usages of the general-purpose operating system). Minimal MFC period is 3 days, which is typically used only for trivial or critical bugfixes.
MFC stands for "Merge From CURRENT" and it describes the process of merging a tested change from our -CURRENT development branch to our -STABLE branch.
Project policy requires any change to be first applied to trunk, and merged to the -STABLE branches after sufficient external testing was done by -CURRENT users (developers are expected to extensively test the change before committing to -CURRENT, but it is impossible for a person to exercise all usages of the general-purpose operating system).
Minimal MFC period is 3 days, which is typically used only for trivial or critical bugfixes.
=== Code Review
Sixty days before the anticipated release, the source repository enters a "code freeze". During this time, all commits to the -STABLE branch must be approved by `{re}`. The approval process is technically enforced by a pre-commit hook. The kinds of changes that are allowed during this period include:
Sixty days before the anticipated release, the source repository enters a "code freeze".
During this time, all commits to the -STABLE branch must be approved by `{re}`.
The approval process is technically enforced by a pre-commit hook.
The kinds of changes that are allowed during this period include:
* Bug fixes.
* Documentation updates.
@ -108,9 +141,12 @@ Sixty days before the anticipated release, the source repository enters a "code
* Driver updates from the vendors.
* Any additional change that the release engineering team feels is justified, given the potential risk.
Shortly after the code freeze is started, a _BETA1_ image is built and released for widespread testing. During the code freeze, at least one beta image or release candidate is released every two weeks until the final release is ready. During the days preceding the final release, the release engineering team is in constant communication with the security-officer team, the documentation maintainers, and the port maintainers to ensure that all of the different components required for a successful release are available.
Shortly after the code freeze is started, a _BETA1_ image is built and released for widespread testing.
During the code freeze, at least one beta image or release candidate is released every two weeks until the final release is ready.
During the days preceding the final release, the release engineering team is in constant communication with the security-officer team, the documentation maintainers, and the port maintainers to ensure that all of the different components required for a successful release are available.
After the quality of the BETA images is satisfying enough, and no large and potentially risky changes are planned, the release branch is created and _Release Candidate_ (RC) images are built from the release branch, instead of the BETA images from the STABLE branch. Also, the freeze on the STABLE branch is lifted and release branch enters a "hard code freeze" where it becomes much harder to justify new changes to the system unless a serious bug-fix or security issue is involved.
After the quality of the BETA images is satisfying enough, and no large and potentially risky changes are planned, the release branch is created and _Release Candidate_ (RC) images are built from the release branch, instead of the BETA images from the STABLE branch.
Also, the freeze on the STABLE branch is lifted and release branch enters a "hard code freeze" where it becomes much harder to justify new changes to the system unless a serious bug-fix or security issue is involved.
=== Final Release Checklist
@ -124,7 +160,8 @@ When several BETA images have been made available for widespread testing and all
In all examples below, `$FSVN` refers to the location of the FreeBSD Subversion repository, `svn+ssh://svn.FreeBSD.org/base/`.
====
The layout of FreeBSD branches in Subversion is described in the link:{committers-guide}#subversion-primer-base-layout[Committer's Guide]. The first step in creating a branch is to identify the revision of the `stable/_X_` sources that you want to branch _from_.
The layout of FreeBSD branches in Subversion is described in the link:{committers-guide}#subversion-primer-base-layout[Committer's Guide].
The first step in creating a branch is to identify the revision of the `stable/_X_` sources that you want to branch _from_.
[source,shell]
....
@ -193,9 +230,12 @@ The release notes and errata files also need to be adjusted for the new release
* [.filename]#src/release/doc/en_US.ISO8859-1/relnotes/common/new.xml#
* [.filename]#src/release/doc/en_US.ISO8859-1/errata/article.xml#
Sysinstall should be updated to note the number of available ports and the amount of disk space required for the Ports Collection. footnote:[FreeBSD Ports Collection https://www.FreeBSD.org/ports] This information is currently kept in [.filename]#src/usr.sbin/bsdinstall/dist.c#.
Sysinstall should be updated to note the number of available ports and the amount of disk space required for the Ports Collection.
footnote:[FreeBSD Ports Collection https://www.FreeBSD.org/ports]
This information is currently kept in [.filename]#src/usr.sbin/bsdinstall/dist.c#.
After the release has been built, a number of files should be updated to announce the release to the world. These files are relative to `head/` within the `doc/` subversion tree.
After the release has been built, a number of files should be updated to announce the release to the world.
These files are relative to `head/` within the `doc/` subversion tree.
* [.filename]#share/images/articles/releng/branches-relengX.pic#
* [.filename]#head/shared/xml/release.ent#
@ -218,16 +258,26 @@ When the final release is ready, the following command will create the `release/
The Documentation and Ports managers are responsible for tagging their respective trees with the `tags/RELEASE_9_2_0` tag.
When the Subversion `svn cp` command is used to create a __release tag__, this identifies the source at a specific point in time. By creating tags, we ensure that future release builders will always be able to use the exact same source we used to create the official FreeBSD Project releases.
When the Subversion `svn cp` command is used to create a __release tag__, this identifies the source at a specific point in time.
By creating tags, we ensure that future release builders will always be able to use the exact same source we used to create the official FreeBSD Project releases.
[[release-build]]
== Release Building
FreeBSD "releases" can be built by anyone with a fast machine and access to a source repository. (That should be everyone, since we offer Subversion access! See the link:{handbook}#svn[Subversion section in the Handbook] for details.) The _only_ special requirement is that the man:md[4] device must be available. If the device is not loaded into your kernel, then the kernel module should be automatically loaded when man:mdconfig[8] is executed during the boot media creation phase. All of the tools necessary to build a release are available from the Subversion repository in [.filename]#src/release#. These tools aim to provide a consistent way to build FreeBSD releases. A complete release can actually be built with only a single command, including the creation of ISO images suitable for burning to CDROM or DVD, and an FTP install directory. man:release[7] fully documents the `src/release/generate-release.sh` script which is used to build a release. `generate-release.sh` is a wrapper around the Makefile target: `make release`.
FreeBSD "releases" can be built by anyone with a fast machine and access to a source repository.
(That should be everyone, since we offer Subversion access! See the link:{handbook}#svn[Subversion section in the Handbook] for details.)
The _only_ special requirement is that the man:md[4] device must be available.
If the device is not loaded into your kernel, then the kernel module should be automatically loaded when man:mdconfig[8] is executed during the boot media creation phase.
All of the tools necessary to build a release are available from the Subversion repository in [.filename]#src/release#.
These tools aim to provide a consistent way to build FreeBSD releases.
A complete release can actually be built with only a single command, including the creation of ISO images suitable for burning to CDROM or DVD, and an FTP install directory.
man:release[7] fully documents the `src/release/generate-release.sh` script which is used to build a release.
`generate-release.sh` is a wrapper around the Makefile target: `make release`.
=== Building a Release
man:release[7] documents the exact commands required to build a FreeBSD release. The following sequences of commands can build an 9.2.0 release:
man:release[7] documents the exact commands required to build a FreeBSD release.
The following sequences of commands can build an 9.2.0 release:
[source,shell]
....
@ -257,16 +307,20 @@ For more information about the release build infrastructure, please see man:rele
[NOTE]
====
It is important to remove any site-specific settings from [.filename]#/etc/make.conf#. For example, it would be unwise to distribute binaries that were built on a system with `CPUTYPE` set to a specific processor.
It is important to remove any site-specific settings from [.filename]#/etc/make.conf#.
For example, it would be unwise to distribute binaries that were built on a system with `CPUTYPE` set to a specific processor.
====
=== Contributed Software ("ports")
The https://www.FreeBSD.org/ports[FreeBSD Ports collection] is a collection of over {numports} third-party software packages available for FreeBSD. The `{portmgr}` is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany official FreeBSD releases.
The https://www.FreeBSD.org/ports[FreeBSD Ports collection] is a collection of over {numports} third-party software packages available for FreeBSD.
The `{portmgr}` is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany official FreeBSD releases.
=== Release ISOs
Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the _BSDi/Wind River Systems/FreeBSD Mall_ "official" CDROM distributions. Each of the four discs must contain a [.filename]#README.TXT# file that explains the contents of the disc, a [.filename]#CDROM.INF# file that provides meta-data for the disc so that man:bsdinstall[8] can validate and use the contents, and a [.filename]#filename.txt# file that provides a manifest for the disc. This _manifest_ can be created with a simple command:
Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the _BSDi/Wind River Systems/FreeBSD Mall_ "official" CDROM distributions.
Each of the four discs must contain a [.filename]#README.TXT# file that explains the contents of the disc, a [.filename]#CDROM.INF# file that provides meta-data for the disc so that man:bsdinstall[8] can validate and use the contents, and a [.filename]#filename.txt# file that provides a manifest for the disc.
This _manifest_ can be created with a simple command:
[source,shell]
....
@ -277,17 +331,28 @@ The specific requirements of each CD are outlined below.
==== Disc 1
The first disc is almost completely created by `make release`. The only changes that should be made to the [.filename]#disc1# directory are the addition of a [.filename]#tools# directory, and as many popular third party software packages as will fit on the disc. The [.filename]#tools# directory contains software that allow users to create installation floppies from other operating systems. This disc should be made bootable so that users of modern PCs do not need to create installation floppy disks.
The first disc is almost completely created by `make release`.
The only changes that should be made to the [.filename]#disc1# directory are the addition of a [.filename]#tools# directory, and as many popular third party software packages as will fit on the disc.
The [.filename]#tools# directory contains software that allow users to create installation floppies from other operating systems.
This disc should be made bootable so that users of modern PCs do not need to create installation floppy disks.
If a custom kernel of FreeBSD is to be included, then man:bsdinstall[8] and man:release[7] must be updated to include installation instructions. The relevant code is contained in [.filename]#src/release# and [.filename]#src/usr.sbin/bsdinstall#. Specifically, the file [.filename]#src/release/Makefile#, and [.filename]#dist.c#, [.filename]#dist.h#, [.filename]#menus.c#, [.filename]#install.c#, and [.filename]#Makefile# will need to be updated under [.filename]#src/usr.sbin/bsdinstall#. Optionally, you may choose to update [.filename]#bsdinstall.8#.
If a custom kernel of FreeBSD is to be included, then man:bsdinstall[8] and man:release[7] must be updated to include installation instructions.
The relevant code is contained in [.filename]#src/release# and [.filename]#src/usr.sbin/bsdinstall#.
Specifically, the file [.filename]#src/release/Makefile#, and [.filename]#dist.c#, [.filename]#dist.h#, [.filename]#menus.c#, [.filename]#install.c#, and [.filename]#Makefile# will need to be updated under [.filename]#src/usr.sbin/bsdinstall#.
Optionally, you may choose to update [.filename]#bsdinstall.8#.
==== Disc 2
The second disc is also largely created by `make release`. This disc contains a "live filesystem" that can be used from man:bsdinstall[8] to troubleshoot a FreeBSD installation. This disc should be bootable and should also contain a compressed copy of the CVS repository in the [.filename]#CVSROOT# directory and commercial software demos in the [.filename]#commerce# directory.
The second disc is also largely created by `make release`.
This disc contains a "live filesystem" that can be used from man:bsdinstall[8] to troubleshoot a FreeBSD installation.
This disc should be bootable and should also contain a compressed copy of the CVS repository in the [.filename]#CVSROOT# directory and commercial software demos in the [.filename]#commerce# directory.
==== Multi-volume Support
Sysinstall supports multiple volume package installations. This requires that each disc have an [.filename]#INDEX# file containing all of the packages on all volumes of a set, along with an extra field that indicates which volume that particular package is on. Each volume in the set must also have the `CD_VOLUME` variable set in the [.filename]#cdrom.inf# file so that bsdinstall can tell which volume is which. When a user attempts to install a package that is not on the current disc, bsdinstall will prompt the user to insert the appropriate one.
Sysinstall supports multiple volume package installations.
This requires that each disc have an [.filename]#INDEX# file containing all of the packages on all volumes of a set, along with an extra field that indicates which volume that particular package is on.
Each volume in the set must also have the `CD_VOLUME` variable set in the [.filename]#cdrom.inf# file so that bsdinstall can tell which volume is which.
When a user attempts to install a package that is not on the current disc, bsdinstall will prompt the user to insert the appropriate one.
[[distribution]]
== Distribution
@ -295,7 +360,10 @@ Sysinstall supports multiple volume package installations. This requires that ea
[[dist-ftp]]
=== FTP Sites
When the release has been thoroughly tested and packaged for distribution, the master FTP site must be updated. The official FreeBSD public FTP sites are all mirrors of a master server that is open only to other FTP sites. This site is known as `ftp-master`. When the release is ready, the following files must be modified on `ftp-master`:
When the release has been thoroughly tested and packaged for distribution, the master FTP site must be updated.
The official FreeBSD public FTP sites are all mirrors of a master server that is open only to other FTP sites.
This site is known as `ftp-master`.
When the release is ready, the following files must be modified on `ftp-master`:
[.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/#::
The installable FTP directory as output from `make release`.
@ -310,11 +378,18 @@ A symlink to [.filename]#../../../tools#.
A symlink to [.filename]#../../../ports/arch/packages-X.Y-release#.
[.filename]#/pub/FreeBSD/releases/arch/ISO-IMAGES/X.Y/X.Y-RELEASE-arch-*.iso#::
The ISO images. The "*" is [.filename]#disc1#, [.filename]#disc2#, etc. Only if there is a [.filename]#disc1# and there is an alternative first installation CD (for example a stripped-down install with no windowing system) there may be a [.filename]#mini# as well.
The ISO images. The "*" is [.filename]#disc1#, [.filename]#disc2#, etc.
Only if there is a [.filename]#disc1# and there is an alternative first installation CD (for example a stripped-down install with no windowing system) there may be a [.filename]#mini# as well.
For more information about the distribution mirror architecture of the FreeBSD FTP sites, please see the link:{hubs}[Mirroring FreeBSD] article.
It may take many hours to two days after updating `ftp-master` before a majority of the Tier-1 FTP sites have the new software depending on whether or not a package set got loaded at the same time. It is imperative that the release engineers coordinate with the {mirror-announce} before announcing the general availability of new software on the FTP sites. Ideally the release package set should be loaded at least four days prior to release day. The release bits should be loaded between 24 and 48 hours before the planned release time with "other" file permissions turned off. This will allow the mirror sites to download it but the general public will not be able to download it from the mirror sites. Mail should be sent to {mirror-announce} at the time the release bits get posted saying the release has been staged and giving the time that the mirror sites should begin allowing access. Be sure to include a time zone with the time, for example make it relative to GMT.
It may take many hours to two days after updating `ftp-master` before a majority of the Tier-1 FTP sites have the new software depending on whether or not a package set got loaded at the same time.
It is imperative that the release engineers coordinate with the {mirror-announce} before announcing the general availability of new software on the FTP sites.
Ideally the release package set should be loaded at least four days prior to release day.
The release bits should be loaded between 24 and 48 hours before the planned release time with "other" file permissions turned off.
This will allow the mirror sites to download it but the general public will not be able to download it from the mirror sites.
Mail should be sent to {mirror-announce} at the time the release bits get posted saying the release has been staged and giving the time that the mirror sites should begin allowing access.
Be sure to include a time zone with the time, for example make it relative to GMT.
[[dist-cdrom]]
=== CD-ROM Replication
@ -324,23 +399,35 @@ Coming soon: Tips for sending FreeBSD ISOs to a replicator and quality assurance
[[extensibility]]
== Extensibility
Although FreeBSD forms a complete operating system, there is nothing that forces you to use the system exactly as we have packaged it up for distribution. We have tried to design the system to be as extensible as possible so that it can serve as a platform that other commercial products can be built on top of. The only "rule" we have about this is that if you are going to distribute FreeBSD with non-trivial changes, we encourage you to document your enhancements! The FreeBSD community can only help support users of the software we provide. We certainly encourage innovation in the form of advanced installation and administration tools, for example, but we cannot be expected to answer questions about it.
Although FreeBSD forms a complete operating system, there is nothing that forces you to use the system exactly as we have packaged it up for distribution.
We have tried to design the system to be as extensible as possible so that it can serve as a platform that other commercial products can be built on top of.
The only "rule" we have about this is that if you are going to distribute FreeBSD with non-trivial changes, we encourage you to document your enhancements!
The FreeBSD community can only help support users of the software we provide.
We certainly encourage innovation in the form of advanced installation and administration tools, for example, but we cannot be expected to answer questions about it.
=== Scripting `bsdinstall`
The FreeBSD system installation and configuration tool, man:bsdinstall[8], can be scripted to provide automated installs for large sites. This functionality can be used in conjunction with Intel(R) PXE footnote:[link:{handbook}#network-diskless[Diskless Operation with PXE]] to bootstrap systems from the network.
The FreeBSD system installation and configuration tool, man:bsdinstall[8], can be scripted to provide automated installs for large sites.
This functionality can be used in conjunction with Intel(R) PXE footnote:[link:{handbook}#network-diskless[Diskless Operation with PXE]] to bootstrap systems from the network.
[[lessons-learned]]
== Lessons Learned from FreeBSD 4.4
The release engineering process for 4.4 formally began on August 1st, 2001. After that date all commits to the `RELENG_4` branch of FreeBSD had to be explicitly approved by the `{re}`. The first release candidate for the x86 architecture was released on August 16, followed by 4 more release candidates leading up to the final release on September 18th. The security officer was very involved in the last week of the process as several security issues were found in the earlier release candidates. A total of over _500_ emails were sent to the `{re}` in little over a month.
The release engineering process for 4.4 formally began on August 1st, 2001.
After that date all commits to the `RELENG_4` branch of FreeBSD had to be explicitly approved by the `{re}`.
The first release candidate for the x86 architecture was released on August 16, followed by 4 more release candidates leading up to the final release on September 18th.
The security officer was very involved in the last week of the process as several security issues were found in the earlier release candidates.
A total of over _500_ emails were sent to the `{re}` in little over a month.
Our user community has made it very clear that the security and stability of a FreeBSD release should not be sacrificed for any self-imposed deadlines or target release dates. The FreeBSD Project has grown tremendously over its lifetime and the need for standardized release engineering procedures has never been more apparent. This will become even more important as FreeBSD is ported to new platforms.
Our user community has made it very clear that the security and stability of a FreeBSD release should not be sacrificed for any self-imposed deadlines or target release dates.
The FreeBSD Project has grown tremendously over its lifetime and the need for standardized release engineering procedures has never been more apparent.
This will become even more important as FreeBSD is ported to new platforms.
[[future]]
== Future Directions
It is imperative for our release engineering activities to scale with our growing userbase. Along these lines we are working very hard to document the procedures involved in producing FreeBSD releases.
It is imperative for our release engineering activities to scale with our growing userbase.
Along these lines we are working very hard to document the procedures involved in producing FreeBSD releases.
* _Parallelism_ - Certain portions of the release build are actually "embarrassingly parallel". Most of the tasks are very I/O intensive, so having multiple high-speed disk drives is actually more important than using multiple processors in speeding up the `make release` process. If multiple disks are used for different hierarchies in the man:chroot[2] environment, then the CVS checkout of the [.filename]#ports# and [.filename]#doc# trees can be happening simultaneously as the `make world` on another disk. Using a RAID solution (hardware or software) can significantly decrease the overall build time.
* _Cross-building releases_ - Building IA-64 or Alpha release on x86 hardware? `make TARGET=ia64 release`.
@ -350,4 +437,7 @@ It is imperative for our release engineering activities to scale with our growin
[[ackno]]
== Acknowledgements
I would like to thank Jordan Hubbard for giving me the opportunity to take on some of the release engineering responsibilities for FreeBSD 4.4 and also for all of his work throughout the years making FreeBSD what it is today. Of course the release would not have been possible without all of the release-related work done by `{asami}`, `{steve}`, `{bmah}`, `{nik}`, `{obrien}`, `{kris}`, `{jhb}` and the rest of the FreeBSD development community. I would also like to thank `{rgrimes}`, `{phk}`, and others who worked on the release engineering tools in the very early days of FreeBSD. This article was influenced by release engineering documents from the CSRG footnote:[Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: link:http://docs.FreeBSD.org/44doc/papers/releng.html[The Release Engineering of 4.3BSD]] , the NetBSD Project, footnote:[NetBSD Developer Documentation: Release Engineering http://www.NetBSD.org/developers/releng/index.html] , and John Baldwin's proposed release engineering process notes. footnote:[John Baldwin's FreeBSD Release Engineering Proposal https://people.FreeBSD.org/~jhb/docs/releng.txt]
I would like to thank Jordan Hubbard for giving me the opportunity to take on some of the release engineering responsibilities for FreeBSD 4.4 and also for all of his work throughout the years making FreeBSD what it is today.
Of course the release would not have been possible without all of the release-related work done by `{asami}`, `{steve}`, `{bmah}`, `{nik}`, `{obrien}`, `{kris}`, `{jhb}` and the rest of the FreeBSD development community.
I would also like to thank `{rgrimes}`, `{phk}`, and others who worked on the release engineering tools in the very early days of FreeBSD.
This article was influenced by release engineering documents from the CSRG footnote:[Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: link:http://docs.FreeBSD.org/44doc/papers/releng.html[The Release Engineering of 4.3BSD]] , the NetBSD Project, footnote:[NetBSD Developer Documentation: Release Engineering http://www.NetBSD.org/developers/releng/index.html] , and John Baldwin's proposed release engineering process notes. footnote:[John Baldwin's FreeBSD Release Engineering Proposal https://people.FreeBSD.org/~jhb/docs/releng.txt]

110
documentation/content/en/articles/remote-install/_index.adoc
View file

@ -24,7 +24,8 @@ include::shared/en/urls.adoc[]
[.abstract-title]
Abstract
This article documents the remote installation of the FreeBSD operating system when the console of the remote system is unavailable. The main idea behind this article is the result of a collaboration with `{mm}` with valuable input provided by `{pjd}`.
This article documents the remote installation of the FreeBSD operating system when the console of the remote system is unavailable.
The main idea behind this article is the result of a collaboration with `{mm}` with valuable input provided by `{pjd}`.
'''
@ -33,16 +34,20 @@ toc::[]
[[background]]
== Background
There are many server hosting providers in the world, but very few of them are officially supporting FreeBSD. They usually provide support for a Linux(R) distribution to be installed on the servers they offer.
There are many server hosting providers in the world, but very few of them are officially supporting FreeBSD.
They usually provide support for a Linux(R) distribution to be installed on the servers they offer.
In some cases, these companies will install your preferred Linux(R) distribution if you request it. Using this option, we will attempt to install FreeBSD. In other cases, they may offer a rescue system which would be used in an emergency. It is possible to use this for our purposes as well.
In some cases, these companies will install your preferred Linux(R) distribution if you request it.
Using this option, we will attempt to install FreeBSD. In other cases, they may offer a rescue system which would be used in an emergency.
It is possible to use this for our purposes as well.
This article covers the basic installation and configuration steps required to bootstrap a remote installation of FreeBSD with RAID-1 and ZFS capabilities.
[[intro]]
== Introduction
This section will summarize the purpose of this article and better explain what is covered herein. The instructions included in this article will benefit those using services provided by colocation facilities not supporting FreeBSD.
This section will summarize the purpose of this article and better explain what is covered herein.
The instructions included in this article will benefit those using services provided by colocation facilities not supporting FreeBSD.
[.procedure]
====
@ -65,11 +70,16 @@ To continue successfully, you must:
[[preparation]]
== Preparation - mfsBSD
Before FreeBSD may be installed on the target system, it is necessary to build the minimal FreeBSD operating system image which will boot from the hard drive. This way the new system can be accessed from the network, and the rest of the installation can be done without remote access to the system console.
Before FreeBSD may be installed on the target system, it is necessary to build the minimal FreeBSD operating system image which will boot from the hard drive.
This way the new system can be accessed from the network, and the rest of the installation can be done without remote access to the system console.
The mfsBSD tool-set can be used to build a tiny FreeBSD image. As the name of mfsBSD suggests ("mfs" means "memory file system"), the resulting image runs entirely from a ramdisk. Thanks to this feature, the manipulation of hard drives will not be limited, therefore it will be possible to install a complete FreeBSD operating system. The mfsBSD http://mfsbsd.vx.sk/[home page] includes pointers to the latest release of the toolset.
The mfsBSD tool-set can be used to build a tiny FreeBSD image.
As the name of mfsBSD suggests ("mfs" means "memory file system"), the resulting image runs entirely from a ramdisk.
Thanks to this feature, the manipulation of hard drives will not be limited, therefore it will be possible to install a complete FreeBSD operating system.
The mfsBSD http://mfsbsd.vx.sk/[home page] includes pointers to the latest release of the toolset.
Please note that the internals of mfsBSD and how it all fits together is beyond the scope of this article. The interested reader should consult the original documentation of mfsBSD for more details.
Please note that the internals of mfsBSD and how it all fits together is beyond the scope of this article.
The interested reader should consult the original documentation of mfsBSD for more details.
Download and extract the latest mfsBSD release and change your working directory to the directory where the mfsBSD scripts will reside:
@ -83,13 +93,19 @@ Download and extract the latest mfsBSD release and change your working directory
[[mfsbsd-config]]
=== Configuration of mfsBSD
Before booting mfsBSD, a few important configuration options have to be set. The most important that we have to get right is, naturally, the network setup. The most suitable method to configure networking options depends on whether we know beforehand the type of the network interface we will use, and the network interface driver to be loaded for our hardware. We will see how mfsBSD can be configured in either case.
Before booting mfsBSD, a few important configuration options have to be set.
The most important that we have to get right is, naturally, the network setup.
The most suitable method to configure networking options depends on whether we know beforehand the type of the network interface we will use, and the network interface driver to be loaded for our hardware.
We will see how mfsBSD can be configured in either case.
Another important thing to set is the `root` password. This can be done by editing [.filename]#conf/loader.conf#. Please see the included comments.
Another important thing to set is the `root` password.
This can be done by editing [.filename]#conf/loader.conf#.
Please see the included comments.
==== The [.filename]#conf/interfaces.conf# method
When the installed network interface card is unknown, it is possible to use the auto-detection features of mfsBSD. The startup scripts of mfsBSD can detect the correct driver to use, based on the MAC address of the interface, if we set the following options in [.filename]#conf/interfaces.conf#:
When the installed network interface card is unknown, it is possible to use the auto-detection features of mfsBSD.
The startup scripts of mfsBSD can detect the correct driver to use, based on the MAC address of the interface, if we set the following options in [.filename]#conf/interfaces.conf#:
[.programlisting]
....
@ -107,7 +123,8 @@ defaultrouter="192.168.0.1"
==== The [.filename]#conf/rc.conf# Method
When the network interface driver is known, it is more convenient to use [.filename]#conf/rc.conf# for networking options. The syntax of this file is the same as the one used in the standard man:rc.conf[5] file of FreeBSD.
When the network interface driver is known, it is more convenient to use [.filename]#conf/rc.conf# for networking options.
The syntax of this file is the same as the one used in the standard man:rc.conf[5] file of FreeBSD.
For example, if you know that a man:re[4] network interface is going to be available, you can set the following options in [.filename]#conf/rc.conf#:
@ -122,7 +139,9 @@ ifconfig_re0="inet 192.168.0.2/24"
The process of building an mfsBSD image is pretty straightforward.
The first step is to mount the FreeBSD installation CD, or the installation ISO image to [.filename]#/cdrom#. For the sake of example, in this article we will assume that you have downloaded the FreeBSD 10.1-RELEASE ISO. Mounting this ISO image to the [.filename]#/cdrom# directory is easy with the man:mdconfig[8] utility:
The first step is to mount the FreeBSD installation CD, or the installation ISO image to [.filename]#/cdrom#.
For the sake of example, in this article we will assume that you have downloaded the FreeBSD 10.1-RELEASE ISO.
Mounting this ISO image to the [.filename]#/cdrom# directory is easy with the man:mdconfig[8] utility:
[source,shell]
....
@ -153,39 +172,52 @@ The above `make` has to be run from the top level of the mfsBSD directory tree,
=== Booting mfsBSD
Now that the mfsBSD image is ready, it must be uploaded to the remote system running a live rescue system or pre-installed Linux(R) distribution. The most suitable tool for this task is scp:
Now that the mfsBSD image is ready, it must be uploaded to the remote system running a live rescue system or pre-installed Linux(R) distribution.
The most suitable tool for this task is scp:
[source,shell]
....
# scp disk.img root@192.168.0.2:.
....
To boot mfsBSD image properly, it must be placed on the first (bootable) device of the given machine. This may be accomplished using this example providing that [.filename]#sda# is the first bootable disk device:
To boot mfsBSD image properly, it must be placed on the first (bootable) device of the given machine.
This may be accomplished using this example providing that [.filename]#sda# is the first bootable disk device:
[source,shell]
....
# dd if=/root/disk.img of=/dev/sda bs=1m
....
If all went well, the image should now be in the MBR of the first device and the machine can be rebooted. Watch for the machine to boot up properly with the man:ping[8] tool. Once it has came back on-line, it should be possible to access it over man:ssh[1] as user `root` with the configured password.
If all went well, the image should now be in the MBR of the first device and the machine can be rebooted.
Watch for the machine to boot up properly with the man:ping[8] tool.
Once it has came back on-line, it should be possible to access it over man:ssh[1] as user `root` with the configured password.
[[installation]]
== Installation of the FreeBSD Operating System
The mfsBSD has been successfully booted and it should be possible to log in through man:ssh[1]. This section will describe how to create and label slices, set up `gmirror` for RAID-1, and how to use `sysinstall` to install a minimal distribution of the FreeBSD operating system.
The mfsBSD has been successfully booted and it should be possible to log in through man:ssh[1].
This section will describe how to create and label slices, set up `gmirror` for RAID-1, and how to use `sysinstall` to install a minimal distribution of the FreeBSD operating system.
=== Preparation of Hard Drives
The first task is to allocate disk space for FreeBSD, i.e.: to create slices and partitions. Obviously, the currently running system is fully loaded in system memory and therefore there will be no problems with manipulating hard drives. To complete this task, it is possible to use either `sysinstall` or man:fdisk[8] in conjunction to man:bsdlabel[8].
The first task is to allocate disk space for FreeBSD, i.e.: to create slices and partitions.
Obviously, the currently running system is fully loaded in system memory and therefore there will be no problems with manipulating hard drives.
To complete this task, it is possible to use either `sysinstall` or man:fdisk[8] in conjunction to man:bsdlabel[8].
At the start, mark all system disks as empty. Repeat the following command for each hard drive:
At the start, mark all system disks as empty.
Repeat the following command for each hard drive:
[source,shell]
....
# dd if=/dev/zero of=/dev/ad0 count=2
....
Next, create slices and label them with your preferred tool. While it is considered easier to use `sysinstall`, a powerful and also probably less buggy method will be to use standard text-based UNIX(R) tools, such as man:fdisk[8] and man:bsdlabel[8], which will also be covered in this section. The former option is well documented in the link:{handbook}#install-steps[Installing FreeBSD] chapter of the FreeBSD Handbook. As it was mentioned in the introduction, this article will present how to set up a system with RAID-1 and ZFS capabilities. Our set up will consist of a small man:gmirror[8] mirrored [.filename]#/# (root), [.filename]#/usr# and [.filename]#/var# dataset, and the rest of the disk space will be allocated for a man:zpool[8] mirrored ZFS file system. Please note, that the ZFS file system will be configured after the FreeBSD operating system is successfully installed and booted.
Next, create slices and label them with your preferred tool.
While it is considered easier to use `sysinstall`, a powerful and also probably less buggy method will be to use standard text-based UNIX(R) tools, such as man:fdisk[8] and man:bsdlabel[8], which will also be covered in this section.
The former option is well documented in the link:{handbook}#install-steps[Installing FreeBSD] chapter of the FreeBSD Handbook.
As it was mentioned in the introduction, this article will present how to set up a system with RAID-1 and ZFS capabilities.
Our set up will consist of a small man:gmirror[8] mirrored [.filename]#/# (root), [.filename]#/usr# and [.filename]#/var# dataset, and the rest of the disk space will be allocated for a man:zpool[8] mirrored ZFS file system.
Please note, that the ZFS file system will be configured after the FreeBSD operating system is successfully installed and booted.
The following example will describe how to create slices and labels, initialize man:gmirror[8] on each partition and how to create a UFS2 file system in each mirrored partition:
@ -222,7 +254,9 @@ The following example will describe how to create slices and labels, initialize
=== System Installation
This is the most important part. This section will describe how to actually install the minimal distribution of FreeBSD on the hard drives that we have prepared in the previous section. To accomplish this goal, all file systems need to be mounted so `sysinstall` may write the contents of FreeBSD to the hard drives:
This is the most important part.
This section will describe how to actually install the minimal distribution of FreeBSD on the hard drives that we have prepared in the previous section.
To accomplish this goal, all file systems need to be mounted so `sysinstall` may write the contents of FreeBSD to the hard drives:
[source,shell]
....
@ -232,30 +266,40 @@ This is the most important part. This section will describe how to actually inst
# mount /dev/mirror/usr /mnt/usr
....
When you are done, start man:sysinstall[8]. Select the [.guimenuitem]#Custom# installation from the main menu. Select [.guimenuitem]#Options# and press kbd:[Enter]. With the help of arrow keys, move the cursor on the `Install Root` item, press kbd:[Space] and change it to [.filename]#/mnt#. Press kbd:[Enter] to submit your changes and exit the [.guimenuitem]#Options# menu by pressing kbd:[q].
When you are done, start man:sysinstall[8].
Select the [.guimenuitem]#Custom# installation from the main menu.
Select [.guimenuitem]#Options# and press kbd:[Enter].
With the help of arrow keys, move the cursor on the `Install Root` item, press kbd:[Space] and change it to [.filename]#/mnt#.
Press kbd:[Enter] to submit your changes and exit the [.guimenuitem]#Options# menu by pressing kbd:[q].
[WARNING]
====
Note that this step is very important and if skipped, `sysinstall` will be unable to install FreeBSD.
====
Go to the [.guimenuitem]#Distributions# menu, move the cursor with the arrow keys to `Minimal`, and check it by pressing kbd:[Space]. This article uses the Minimal distribution in order to save network traffic, because the system itself will be installed over ftp. Exit this menu by choosing `Exit`.
Go to the [.guimenuitem]#Distributions# menu, move the cursor with the arrow keys to `Minimal`, and check it by pressing kbd:[Space].
This article uses the Minimal distribution in order to save network traffic, because the system itself will be installed over ftp.
Exit this menu by choosing `Exit`.
[NOTE]
====
The [.guimenuitem]#Partition# and [.guimenuitem]#Label# menus will be skipped, as these are useless now.
====
In the [.guimenuitem]#Media# menu, select `FTP`. Select the nearest mirror and let `sysinstall` assume that the network is already configured. You will be returned back to the [.guimenuitem]#Custom# menu.
In the [.guimenuitem]#Media# menu, select `FTP`.
Select the nearest mirror and let `sysinstall` assume that the network is already configured.
You will be returned back to the [.guimenuitem]#Custom# menu.
Finally, perform the system installation by selecting the last option, [.guimenuitem]#Commit#. Exit `sysinstall` when it finishes the installation.
Finally, perform the system installation by selecting the last option, [.guimenuitem]#Commit#.
Exit `sysinstall` when it finishes the installation.
=== Post Installation Steps
The FreeBSD operating system should be installed now; however, the process is not finished yet. It is necessary to perform some post installation steps in order to allow FreeBSD to boot in the future and to be able to log in to the system.
The FreeBSD operating system should be installed now; however, the process is not finished yet.
It is necessary to perform some post installation steps in order to allow FreeBSD to boot in the future and to be able to log in to the system.
You must now man:chroot[8] into the freshly installed system in order to finish the installation. Use the following command:
You must now man:chroot[8] into the freshly installed system in order to finish the installation.
Use the following command:
[source,shell]
....
@ -299,14 +343,17 @@ zfs_load="YES"
* Add additional users to the system using the man:adduser[8] tool. Do not forget to add a user to the `wheel` group so you may obtain root access after the reboot.
* Double-check all your settings.
The system should now be ready for the next boot. Use the man:reboot[8] command to reboot your system.
The system should now be ready for the next boot.
Use the man:reboot[8] command to reboot your system.
[[zfs]]
== ZFS
If your system survived the reboot, it should now be possible to log in. Welcome to the fresh FreeBSD installation, performed remotely without the use of a remote console!
If your system survived the reboot, it should now be possible to log in.
Welcome to the fresh FreeBSD installation, performed remotely without the use of a remote console!
The only remaining step is to configure man:zpool[8] and create some man:zfs[8] file systems. Creating and administering ZFS is very straightforward. First, create a mirrored pool:
The only remaining step is to configure man:zpool[8] and create some man:zfs[8] file systems.
Creating and administering ZFS is very straightforward. First, create a mirrored pool:
[source,shell]
....
@ -325,4 +372,5 @@ Next, create some file systems:
# zfs set mountpoint=/usr/src tank/src
....
That is all. If you are interested in more details about ZFS on FreeBSD, please refer to the https://wiki.freebsd.org/ZFS[ZFS] section of the FreeBSD Wiki.
That is all.
If you are interested in more details about ZFS on FreeBSD, please refer to the https://wiki.freebsd.org/ZFS[ZFS] section of the FreeBSD Wiki.

321
documentation/content/en/articles/serial-uart/_index.adoc
View file

@ -34,11 +34,15 @@ toc::[]
_Copyright (R) 1996 `{uhclem}`, All Rights Reserved. 13 January 1996._
The Universal Asynchronous Receiver/Transmitter (UART) controller is the key component of the serial communications subsystem of a computer. The UART takes bytes of data and transmits the individual bits in a sequential fashion. At the destination, a second UART re-assembles the bits into complete bytes.
The Universal Asynchronous Receiver/Transmitter (UART) controller is the key component of the serial communications subsystem of a computer.
The UART takes bytes of data and transmits the individual bits in a sequential fashion.
At the destination, a second UART re-assembles the bits into complete bytes.
Serial transmission is commonly used with modems and for non-networked communication between computers, terminals and other devices.
There are two primary forms of serial transmission: Synchronous and Asynchronous. Depending on the modes that are supported by the hardware, the name of the communication sub-system will usually include a `A` if it supports Asynchronous communications, and a `S` if it supports Synchronous communications. Both forms are described below.
There are two primary forms of serial transmission: Synchronous and Asynchronous.
Depending on the modes that are supported by the hardware, the name of the communication sub-system will usually include a `A` if it supports Asynchronous communications, and a `S` if it supports Synchronous communications.
Both forms are described below.
Some common acronyms are:
@ -50,27 +54,44 @@ USART Universal Synchronous-Asynchronous Receiver/Transmitter
=== Synchronous Serial Transmission
Synchronous serial transmission requires that the sender and receiver share a clock with one another, or that the sender provide a strobe or other timing signal so that the receiver knows when to "read" the next bit of the data. In most forms of serial Synchronous communication, if there is no data available at a given instant to transmit, a fill character must be sent instead so that data is always being transmitted. Synchronous communication is usually more efficient because only data bits are transmitted between sender and receiver, and synchronous communication can be more costly if extra wiring and circuits are required to share a clock signal between the sender and receiver.
Synchronous serial transmission requires that the sender and receiver share a clock with one another, or that the sender provide a strobe or other timing signal so that the receiver knows when to "read" the next bit of the data.
In most forms of serial Synchronous communication, if there is no data available at a given instant to transmit, a fill character must be sent instead so that data is always being transmitted.
Synchronous communication is usually more efficient because only data bits are transmitted between sender and receiver, and synchronous communication can be more costly if extra wiring and circuits are required to share a clock signal between the sender and receiver.
A form of Synchronous transmission is used with printers and fixed disk devices in that the data is sent on one set of wires while a clock or strobe is sent on a different wire. Printers and fixed disk devices are not normally serial devices because most fixed disk interface standards send an entire word of data for each clock or strobe signal by using a separate wire for each bit of the word. In the PC industry, these are known as Parallel devices.
A form of Synchronous transmission is used with printers and fixed disk devices in that the data is sent on one set of wires while a clock or strobe is sent on a different wire.
Printers and fixed disk devices are not normally serial devices because most fixed disk interface standards send an entire word of data for each clock or strobe signal by using a separate wire for each bit of the word.
In the PC industry, these are known as Parallel devices.
The standard serial communications hardware in the PC does not support Synchronous operations. This mode is described here for comparison purposes only.
The standard serial communications hardware in the PC does not support Synchronous operations.
This mode is described here for comparison purposes only.
=== Asynchronous Serial Transmission
Asynchronous transmission allows data to be transmitted without the sender having to send a clock signal to the receiver. Instead, the sender and receiver must agree on timing parameters in advance and special bits are added to each word which are used to synchronize the sending and receiving units.
Asynchronous transmission allows data to be transmitted without the sender having to send a clock signal to the receiver.
Instead, the sender and receiver must agree on timing parameters in advance and special bits are added to each word which are used to synchronize the sending and receiving units.
When a word is given to the UART for Asynchronous transmissions, a bit called the "Start Bit" is added to the beginning of each word that is to be transmitted. The Start Bit is used to alert the receiver that a word of data is about to be sent, and to force the clock in the receiver into synchronization with the clock in the transmitter. These two clocks must be accurate enough to not have the frequency drift by more than 10% during the transmission of the remaining bits in the word. (This requirement was set in the days of mechanical teleprinters and is easily met by modern electronic equipment.)
When a word is given to the UART for Asynchronous transmissions, a bit called the "Start Bit" is added to the beginning of each word that is to be transmitted.
The Start Bit is used to alert the receiver that a word of data is about to be sent, and to force the clock in the receiver into synchronization with the clock in the transmitter.
These two clocks must be accurate enough to not have the frequency drift by more than 10% during the transmission of the remaining bits in the word.
(This requirement was set in the days of mechanical teleprinters and is easily met by modern electronic equipment.)
After the Start Bit, the individual bits of the word of data are sent, with the Least Significant Bit (LSB) being sent first. Each bit in the transmission is transmitted for exactly the same amount of time as all of the other bits, and the receiver "looks" at the wire at approximately halfway through the period assigned to each bit to determine if the bit is a `1` or a `0`. For example, if it takes two seconds to send each bit, the receiver will examine the signal to determine if it is a `1` or a `0` after one second has passed, then it will wait two seconds and then examine the value of the next bit, and so on.
After the Start Bit, the individual bits of the word of data are sent, with the Least Significant Bit (LSB) being sent first.
Each bit in the transmission is transmitted for exactly the same amount of time as all of the other bits, and the receiver "looks" at the wire at approximately halfway through the period assigned to each bit to determine if the bit is a `1` or a `0`.
For example, if it takes two seconds to send each bit, the receiver will examine the signal to determine if it is a `1` or a `0` after one second has passed, then it will wait two seconds and then examine the value of the next bit, and so on.
The sender does not know when the receiver has "looked" at the value of the bit. The sender only knows when the clock says to begin transmitting the next bit of the word.
The sender does not know when the receiver has "looked" at the value of the bit.
The sender only knows when the clock says to begin transmitting the next bit of the word.
When the entire data word has been sent, the transmitter may add a Parity Bit that the transmitter generates. The Parity Bit may be used by the receiver to perform simple error checking. Then at least one Stop Bit is sent by the transmitter.
When the entire data word has been sent, the transmitter may add a Parity Bit that the transmitter generates.
The Parity Bit may be used by the receiver to perform simple error checking.
Then at least one Stop Bit is sent by the transmitter.
When the receiver has received all of the bits in the data word, it may check for the Parity Bits (both sender and receiver must agree on whether a Parity Bit is to be used), and then the receiver looks for a Stop Bit. If the Stop Bit does not appear when it is supposed to, the UART considers the entire word to be garbled and will report a Framing Error to the host processor when the data word is read. The usual cause of a Framing Error is that the sender and receiver clocks were not running at the same speed, or that the signal was interrupted.
When the receiver has received all of the bits in the data word, it may check for the Parity Bits (both sender and receiver must agree on whether a Parity Bit is to be used), and then the receiver looks for a Stop Bit.
If the Stop Bit does not appear when it is supposed to, the UART considers the entire word to be garbled and will report a Framing Error to the host processor when the data word is read.
The usual cause of a Framing Error is that the sender and receiver clocks were not running at the same speed, or that the signal was interrupted.
Regardless of whether the data was received correctly or not, the UART automatically discards the Start, Parity and Stop bits. If the sender and receiver are configured identically, these bits are not passed to the host.
Regardless of whether the data was received correctly or not, the UART automatically discards the Start, Parity and Stop bits.
If the sender and receiver are configured identically, these bits are not passed to the host.
If another word is ready for transmission, the Start Bit for the new word can be sent as soon as the Stop Bit for the previous word has been sent.
@ -78,54 +99,78 @@ As asynchronous data is "self synchronizing", if there is no data to transmit, t
=== Other UART Functions
In addition to the basic job of converting data from parallel to serial for transmission and from serial to parallel on reception, a UART will usually provide additional circuits for signals that can be used to indicate the state of the transmission media, and to regulate the flow of data in the event that the remote device is not prepared to accept more data. For example, when the device connected to the UART is a modem, the modem may report the presence of a carrier on the phone line while the computer may be able to instruct the modem to reset itself or to not take calls by raising or lowering one more of these extra signals. The function of each of these additional signals is defined in the EIA RS232-C standard.
In addition to the basic job of converting data from parallel to serial for transmission and from serial to parallel on reception, a UART will usually provide additional circuits for signals that can be used to indicate the state of the transmission media, and to regulate the flow of data in the event that the remote device is not prepared to accept more data.
For example, when the device connected to the UART is a modem, the modem may report the presence of a carrier on the phone line while the computer may be able to instruct the modem to reset itself or to not take calls by raising or lowering one more of these extra signals.
The function of each of these additional signals is defined in the EIA RS232-C standard.
=== The RS232-C and V.24 Standards
In most computer systems, the UART is connected to circuitry that generates signals that comply with the EIA RS232-C specification. There is also a CCITT standard named V.24 that mirrors the specifications included in RS232-C.
In most computer systems, the UART is connected to circuitry that generates signals that comply with the EIA RS232-C specification.
There is also a CCITT standard named V.24 that mirrors the specifications included in RS232-C.
==== RS232-C Bit Assignments (Marks and Spaces)
In RS232-C, a value of `1` is called a `Mark` and a value of `0` is called a `Space`. When a communication line is idle, the line is said to be "Marking", or transmitting continuous `1` values.
In RS232-C, a value of `1` is called a `Mark` and a value of `0` is called a `Space`.
When a communication line is idle, the line is said to be "Marking", or transmitting continuous `1` values.
The Start bit always has a value of `0` (a Space). The Stop Bit always has a value of `1` (a Mark). This means that there will always be a Mark (1) to Space (0) transition on the line at the start of every word, even when multiple word are transmitted back to back. This guarantees that sender and receiver can resynchronize their clocks regardless of the content of the data bits that are being transmitted.
The Start bit always has a value of `0` (a Space).
The Stop Bit always has a value of `1` (a Mark).
This means that there will always be a Mark (1) to Space (0) transition on the line at the start of every word, even when multiple word are transmitted back to back.
This guarantees that sender and receiver can resynchronize their clocks regardless of the content of the data bits that are being transmitted.
The idle time between Stop and Start bits does not have to be an exact multiple (including zero) of the bit rate of the communication link, but most UARTs are designed this way for simplicity.
In RS232-C, the "Marking" signal (a `1`) is represented by a voltage between -2 VDC and -12 VDC, and a "Spacing" signal (a `0`) is represented by a voltage between 0 and +12 VDC. The transmitter is supposed to send +12 VDC or -12 VDC, and the receiver is supposed to allow for some voltage loss in long cables. Some transmitters in low power devices (like portable computers) sometimes use only +5 VDC and -5 VDC, but these values are still acceptable to a RS232-C receiver, provided that the cable lengths are short.
In RS232-C, the "Marking" signal (a `1`) is represented by a voltage between -2 VDC and -12 VDC, and a "Spacing" signal (a `0`) is represented by a voltage between 0 and +12 VDC.
The transmitter is supposed to send +12 VDC or -12 VDC, and the receiver is supposed to allow for some voltage loss in long cables.
Some transmitters in low power devices (like portable computers) sometimes use only +5 VDC and -5 VDC, but these values are still acceptable to a RS232-C receiver, provided that the cable lengths are short.
==== RS232-C Break Signal
RS232-C also specifies a signal called a `Break`, which is caused by sending continuous Spacing values (no Start or Stop bits). When there is no electricity present on the data circuit, the line is considered to be sending `Break`.
RS232-C also specifies a signal called a `Break`, which is caused by sending continuous Spacing values (no Start or Stop bits).
When there is no electricity present on the data circuit, the line is considered to be sending `Break`.
The `Break` signal must be of a duration longer than the time it takes to send a complete byte plus Start, Stop and Parity bits. Most UARTs can distinguish between a Framing Error and a Break, but if the UART cannot do this, the Framing Error detection can be used to identify Breaks.
The `Break` signal must be of a duration longer than the time it takes to send a complete byte plus Start, Stop and Parity bits.
Most UARTs can distinguish between a Framing Error and a Break, but if the UART cannot do this, the Framing Error detection can be used to identify Breaks.
In the days of teleprinters, when numerous printers around the country were wired in series (such as news services), any unit could cause a `Break` by temporarily opening the entire circuit so that no current flowed. This was used to allow a location with urgent news to interrupt some other location that was currently sending information.
In the days of teleprinters, when numerous printers around the country were wired in series (such as news services), any unit could cause a `Break` by temporarily opening the entire circuit so that no current flowed.
This was used to allow a location with urgent news to interrupt some other location that was currently sending information.
In modern systems there are two types of Break signals. If the Break is longer than 1.6 seconds, it is considered a "Modem Break", and some modems can be programmed to terminate the conversation and go on-hook or enter the modems' command mode when the modem detects this signal. If the Break is smaller than 1.6 seconds, it signifies a Data Break and it is up to the remote computer to respond to this signal. Sometimes this form of Break is used as an Attention or Interrupt signal and sometimes is accepted as a substitute for the ASCII CONTROL-C character.
In modern systems there are two types of Break signals.
If the Break is longer than 1.6 seconds, it is considered a "Modem Break", and some modems can be programmed to terminate the conversation and go on-hook or enter the modems' command mode when the modem detects this signal.
If the Break is smaller than 1.6 seconds, it signifies a Data Break and it is up to the remote computer to respond to this signal.
Sometimes this form of Break is used as an Attention or Interrupt signal and sometimes is accepted as a substitute for the ASCII CONTROL-C character.
Marks and Spaces are also equivalent to "Holes" and "No Holes" in paper tape systems.
[NOTE]
====
Breaks cannot be generated from paper tape or from any other byte value, since bytes are always sent with Start and Stop bit. The UART is usually capable of generating the continuous Spacing signal in response to a special command from the host processor.
Breaks cannot be generated from paper tape or from any other byte value, since bytes are always sent with Start and Stop bit.
The UART is usually capable of generating the continuous Spacing signal in response to a special command from the host processor.
====
==== RS232-C DTE and DCE Devices
The RS232-C specification defines two types of equipment: the Data Terminal Equipment (DTE) and the Data Carrier Equipment (DCE). Usually, the DTE device is the terminal (or computer), and the DCE is a modem. Across the phone line at the other end of a conversation, the receiving modem is also a DCE device and the computer that is connected to that modem is a DTE device. The DCE device receives signals on the pins that the DTE device transmits on, and vice versa.
The RS232-C specification defines two types of equipment: the Data Terminal Equipment (DTE) and the Data Carrier Equipment (DCE).
Usually, the DTE device is the terminal (or computer), and the DCE is a modem.
Across the phone line at the other end of a conversation, the receiving modem is also a DCE device and the computer that is connected to that modem is a DTE device.
The DCE device receives signals on the pins that the DTE device transmits on, and vice versa.
When two devices that are both DTE or both DCE must be connected together without a modem or a similar media translator between them, a NULL modem must be used. The NULL modem electrically re-arranges the cabling so that the transmitter output is connected to the receiver input on the other device, and vice versa. Similar translations are performed on all of the control signals so that each device will see what it thinks are DCE (or DTE) signals from the other device.
When two devices that are both DTE or both DCE must be connected together without a modem or a similar media translator between them, a NULL modem must be used.
The NULL modem electrically re-arranges the cabling so that the transmitter output is connected to the receiver input on the other device, and vice versa.
Similar translations are performed on all of the control signals so that each device will see what it thinks are DCE (or DTE) signals from the other device.
The number of signals generated by the DTE and DCE devices are not symmetrical. The DTE device generates fewer signals for the DCE device than the DTE device receives from the DCE.
The number of signals generated by the DTE and DCE devices are not symmetrical.
The DTE device generates fewer signals for the DCE device than the DTE device receives from the DCE.
==== RS232-C Pin Assignments
The EIA RS232-C specification (and the ITU equivalent, V.24) calls for a twenty-five pin connector (usually a DB25) and defines the purpose of most of the pins in that connector.
In the IBM Personal Computer and similar systems, a subset of RS232-C signals are provided via nine pin connectors (DB9). The signals that are not included on the PC connector deal mainly with synchronous operation, and this transmission mode is not supported by the UART that IBM selected for use in the IBM PC.
In the IBM Personal Computer and similar systems, a subset of RS232-C signals are provided via nine pin connectors (DB9).
The signals that are not included on the PC connector deal mainly with synchronous operation, and this transmission mode is not supported by the UART that IBM selected for use in the IBM PC.
Depending on the computer manufacturer, a DB25, a DB9, or both types of connector may be used for RS232-C communications. (The IBM PC also uses a DB25 connector for the parallel printer interface which causes some confusion.)
Depending on the computer manufacturer, a DB25, a DB9, or both types of connector may be used for RS232-C communications.
(The IBM PC also uses a DB25 connector for the parallel printer interface which causes some confusion.)
Below is a table of the RS232-C signal assignments in the DB25 and DB9 connectors.
@ -343,29 +388,46 @@ Below is a table of the RS232-C signal assignments in the DB25 and DB9 connector
=== Bits, Baud and Symbols
Baud is a measurement of transmission speed in asynchronous communication. Due to advances in modem communication technology, this term is frequently misused when describing the data rates in newer devices.
Baud is a measurement of transmission speed in asynchronous communication.
Due to advances in modem communication technology, this term is frequently misused when describing the data rates in newer devices.
Traditionally, a Baud Rate represents the number of bits that are actually being sent over the media, not the amount of data that is actually moved from one DTE device to the other. The Baud count includes the overhead bits Start, Stop and Parity that are generated by the sending UART and removed by the receiving UART. This means that seven-bit words of data actually take 10 bits to be completely transmitted. Therefore, a modem capable of moving 300 bits per second from one place to another can normally only move 30 7-bit words if Parity is used and one Start and Stop bit are present.
Traditionally, a Baud Rate represents the number of bits that are actually being sent over the media, not the amount of data that is actually moved from one DTE device to the other.
The Baud count includes the overhead bits Start, Stop and Parity that are generated by the sending UART and removed by the receiving UART.
This means that seven-bit words of data actually take 10 bits to be completely transmitted.
Therefore, a modem capable of moving 300 bits per second from one place to another can normally only move 30 7-bit words if Parity is used and one Start and Stop bit are present.
If 8-bit data words are used and Parity bits are also used, the data rate falls to 27.27 words per second, because it now takes 11 bits to send the eight-bit words, and the modem still only sends 300 bits per second.
The formula for converting bytes per second into a baud rate and vice versa was simple until error-correcting modems came along. These modems receive the serial stream of bits from the UART in the host computer (even when internal modems are used the data is still frequently serialized) and converts the bits back into bytes. These bytes are then combined into packets and sent over the phone line using a Synchronous transmission method. This means that the Stop, Start, and Parity bits added by the UART in the DTE (the computer) were removed by the modem before transmission by the sending modem. When these bytes are received by the remote modem, the remote modem adds Start, Stop and Parity bits to the words, converts them to a serial format and then sends them to the receiving UART in the remote computer, who then strips the Start, Stop and Parity bits.
The formula for converting bytes per second into a baud rate and vice versa was simple until error-correcting modems came along.
These modems receive the serial stream of bits from the UART in the host computer (even when internal modems are used the data is still frequently serialized) and converts the bits back into bytes.
These bytes are then combined into packets and sent over the phone line using a Synchronous transmission method.
This means that the Stop, Start, and Parity bits added by the UART in the DTE (the computer) were removed by the modem before transmission by the sending modem.
When these bytes are received by the remote modem, the remote modem adds Start, Stop and Parity bits to the words, converts them to a serial format and then sends them to the receiving UART in the remote computer, who then strips the Start, Stop and Parity bits.
The reason all these extra conversions are done is so that the two modems can perform error correction, which means that the receiving modem is able to ask the sending modem to resend a block of data that was not received with the correct checksum. This checking is handled by the modems, and the DTE devices are usually unaware that the process is occurring.
The reason all these extra conversions are done is so that the two modems can perform error correction, which means that the receiving modem is able to ask the sending modem to resend a block of data that was not received with the correct checksum.
This checking is handled by the modems, and the DTE devices are usually unaware that the process is occurring.
By striping the Start, Stop and Parity bits, the additional bits of data that the two modems must share between themselves to perform error-correction are mostly concealed from the effective transmission rate seen by the sending and receiving DTE equipment. For example, if a modem sends ten 7-bit words to another modem without including the Start, Stop and Parity bits, the sending modem will be able to add 30 bits of its own information that the receiving modem can use to do error-correction without impacting the transmission speed of the real data.
By striping the Start, Stop and Parity bits, the additional bits of data that the two modems must share between themselves to perform error-correction are mostly concealed from the effective transmission rate seen by the sending and receiving DTE equipment.
For example, if a modem sends ten 7-bit words to another modem without including the Start, Stop and Parity bits, the sending modem will be able to add 30 bits of its own information that the receiving modem can use to do error-correction without impacting the transmission speed of the real data.
The use of the term Baud is further confused by modems that perform compression. A single 8-bit word passed over the telephone line might represent a dozen words that were transmitted to the sending modem. The receiving modem will expand the data back to its original content and pass that data to the receiving DTE.
The use of the term Baud is further confused by modems that perform compression.
A single 8-bit word passed over the telephone line might represent a dozen words that were transmitted to the sending modem.
The receiving modem will expand the data back to its original content and pass that data to the receiving DTE.
Modern modems also include buffers that allow the rate that bits move across the phone line (DCE to DCE) to be a different speed than the speed that the bits move between the DTE and DCE on both ends of the conversation. Normally the speed between the DTE and DCE is higher than the DCE to DCE speed because of the use of compression by the modems.
Modern modems also include buffers that allow the rate that bits move across the phone line (DCE to DCE) to be a different speed than the speed that the bits move between the DTE and DCE on both ends of the conversation.
Normally the speed between the DTE and DCE is higher than the DCE to DCE speed because of the use of compression by the modems.
As the number of bits needed to describe a byte varied during the trip between the two machines plus the differing bits-per-seconds speeds that are used present on the DTE-DCE and DCE-DCE links, the usage of the term Baud to describe the overall communication speed causes problems and can misrepresent the true transmission speed. So Bits Per Second (bps) is the correct term to use to describe the transmission rate seen at the DCE to DCE interface and Baud or Bits Per Second are acceptable terms to use when a connection is made between two systems with a wired connection, or if a modem is in use that is not performing error-correction or compression.
As the number of bits needed to describe a byte varied during the trip between the two machines plus the differing bits-per-seconds speeds that are used present on the DTE-DCE and DCE-DCE links, the usage of the term Baud to describe the overall communication speed causes problems and can misrepresent the true transmission speed.
So Bits Per Second (bps) is the correct term to use to describe the transmission rate seen at the DCE to DCE interface and Baud or Bits Per Second are acceptable terms to use when a connection is made between two systems with a wired connection, or if a modem is in use that is not performing error-correction or compression.
Modern high speed modems (2400, 9600, 14,400, and 19,200bps) in reality still operate at or below 2400 baud, or more accurately, 2400 Symbols per second. High speed modem are able to encode more bits of data into each Symbol using a technique called Constellation Stuffing, which is why the effective bits per second rate of the modem is higher, but the modem continues to operate within the limited audio bandwidth that the telephone system provides. Modems operating at 28,800 and higher speeds have variable Symbol rates, but the technique is the same.
Modern high speed modems (2400, 9600, 14,400, and 19,200bps) in reality still operate at or below 2400 baud, or more accurately, 2400 Symbols per second.
High speed modem are able to encode more bits of data into each Symbol using a technique called Constellation Stuffing, which is why the effective bits per second rate of the modem is higher, but the modem continues to operate within the limited audio bandwidth that the telephone system provides.
Modems operating at 28,800 and higher speeds have variable Symbol rates, but the technique is the same.
=== The IBM Personal Computer UART
Starting with the original IBM Personal Computer, IBM selected the National Semiconductor INS8250 UART for use in the IBM PC Parallel/Serial Adapter. Subsequent generations of compatible computers from IBM and other vendors continued to use the INS8250 or improved versions of the National Semiconductor UART family.
Starting with the original IBM Personal Computer, IBM selected the National Semiconductor INS8250 UART for use in the IBM PC Parallel/Serial Adapter.
Subsequent generations of compatible computers from IBM and other vendors continued to use the INS8250 or improved versions of the National Semiconductor UART family.
==== National Semiconductor UART Family Tree
@ -386,20 +448,27 @@ INS8250 -> INS8250B
....
INS8250::
This part was used in the original IBM PC and IBM PC/XT. The original name for this part was the INS8250 ACE (Asynchronous Communications Element) and it is made from NMOS technology.
This part was used in the original IBM PC and IBM PC/XT.
The original name for this part was the INS8250 ACE (Asynchronous Communications Element) and it is made from NMOS technology.
+
The 8250 uses eight I/O ports and has a one-byte send and a one-byte receive buffer. This original UART has several race conditions and other flaws. The original IBM BIOS includes code to work around these flaws, but this made the BIOS dependent on the flaws being present, so subsequent parts like the 8250A, 16450 or 16550 could not be used in the original IBM PC or IBM PC/XT.
The 8250 uses eight I/O ports and has a one-byte send and a one-byte receive buffer.
This original UART has several race conditions and other flaws.
The original IBM BIOS includes code to work around these flaws, but this made the BIOS dependent on the flaws being present, so subsequent parts like the 8250A, 16450 or 16550 could not be used in the original IBM PC or IBM PC/XT.
INS8250-B::
This is the slower speed of the INS8250 made from NMOS technology. It contains the same problems as the original INS8250.
This is the slower speed of the INS8250 made from NMOS technology.
It contains the same problems as the original INS8250.
INS8250A::
An improved version of the INS8250 using XMOS technology with various functional flaws corrected. The INS8250A was used initially in PC clone computers by vendors who used "clean" BIOS designs. Due to the corrections in the chip, this part could not be used with a BIOS compatible with the INS8250 or INS8250B.
An improved version of the INS8250 using XMOS technology with various functional flaws corrected.
The INS8250A was used initially in PC clone computers by vendors who used "clean" BIOS designs.
Due to the corrections in the chip, this part could not be used with a BIOS compatible with the INS8250 or INS8250B.
INS82C50A::
This is a CMOS version (low power consumption) of the INS8250A and has similar functional characteristics.
NS16450::
Same as NS8250A with improvements so it can be used with faster CPU bus designs. IBM used this part in the IBM AT and updated the IBM BIOS to no longer rely on the bugs in the INS8250.
Same as NS8250A with improvements so it can be used with faster CPU bus designs.
IBM used this part in the IBM AT and updated the IBM BIOS to no longer rely on the bugs in the INS8250.
NS16C450::
This is a CMOS version (low power consumption) of the NS16450.
@ -408,19 +477,25 @@ NS16550::
Same as NS16450 with a 16-byte send and receive buffer but the buffer design was flawed and could not be reliably be used.
NS16550A::
Same as NS16550 with the buffer flaws corrected. The 16550A and its successors have become the most popular UART design in the PC industry, mainly due to its ability to reliably handle higher data rates on operating systems with sluggish interrupt response times.
Same as NS16550 with the buffer flaws corrected.
The 16550A and its successors have become the most popular UART design in the PC industry, mainly due to its ability to reliably handle higher data rates on operating systems with sluggish interrupt response times.
NS16C552::
This component consists of two NS16C550A CMOS UARTs in a single package.
PC16550D::
Same as NS16550A with subtle flaws corrected. This is revision D of the 16550 family and is the latest design available from National Semiconductor.
Same as NS16550A with subtle flaws corrected.
This is revision D of the 16550 family and is the latest design available from National Semiconductor.
==== The NS16550AF and the PC16550D are the same thing
National reorganized their part numbering system a few years ago, and the NS16550AFN no longer exists by that name. (If you have a NS16550AFN, look at the date code on the part, which is a four digit number that usually starts with a nine. The first two digits of the number are the year, and the last two digits are the week in that year when the part was packaged. If you have a NS16550AFN, it is probably a few years old.)
National reorganized their part numbering system a few years ago, and the NS16550AFN no longer exists by that name.
(If you have a NS16550AFN, look at the date code on the part, which is a four digit number that usually starts with a nine.
The first two digits of the number are the year, and the last two digits are the week in that year when the part was packaged.
If you have a NS16550AFN, it is probably a few years old.)
The new numbers are like PC16550DV, with minor differences in the suffix letters depending on the package material and its shape. (A description of the numbering system can be found below.)
The new numbers are like PC16550DV, with minor differences in the suffix letters depending on the package material and its shape.
(A description of the numbering system can be found below.)
It is important to understand that in some stores, you may pay $15(US) for a NS16550AFN made in 1990 and in the next bin are the new PC16550DN parts with minor fixes that National has made since the AFN part was in production, the PC16550DN was probably made in the past six months and it costs half (as low as $5(US) in volume) as much as the NS16550AFN because they are readily available.
@ -451,29 +526,44 @@ The `_p_` is the package-type field. The types are:
|(lead plastic chip carrier) J lead type
|===
The _g_ is the product grade field. If an `I` precedes the package-type letter, it indicates an "industrial" grade part, which has higher specs than a standard part but not as high as Military Specification (Milspec) component. This is an optional field.
The _g_ is the product grade field.
If an `I` precedes the package-type letter, it indicates an "industrial" grade part, which has higher specs than a standard part but not as high as Military Specification (Milspec) component.
This is an optional field.
So what we used to call a NS16550AFN (DIP Package) is now called a PC16550DN or PC16550DIN.
=== Other Vendors and Similar UARTs
Over the years, the 8250, 8250A, 16450 and 16550 have been licensed or copied by other chip vendors. In the case of the 8250, 8250A and 16450, the exact circuit (the "megacell") was licensed to many vendors, including Western Digital and Intel. Other vendors reverse-engineered the part or produced emulations that had similar behavior.
Over the years, the 8250, 8250A, 16450 and 16550 have been licensed or copied by other chip vendors.
In the case of the 8250, 8250A and 16450, the exact circuit (the "megacell") was licensed to many vendors, including Western Digital and Intel.
Other vendors reverse-engineered the part or produced emulations that had similar behavior.
In internal modems, the modem designer will frequently emulate the 8250A/16450 with the modem microprocessor, and the emulated UART will frequently have a hidden buffer consisting of several hundred bytes. Due to the size of the buffer, these emulations can be as reliable as a 16550A in their ability to handle high speed data. However, most operating systems will still report that the UART is only a 8250A or 16450, and may not make effective use of the extra buffering present in the emulated UART unless special drivers are used.
In internal modems, the modem designer will frequently emulate the 8250A/16450 with the modem microprocessor, and the emulated UART will frequently have a hidden buffer consisting of several hundred bytes.
Due to the size of the buffer, these emulations can be as reliable as a 16550A in their ability to handle high speed data.
However, most operating systems will still report that the UART is only a 8250A or 16450, and may not make effective use of the extra buffering present in the emulated UART unless special drivers are used.
Some modem makers are driven by market forces to abandon a design that has hundreds of bytes of buffer and instead use a 16550A UART so that the product will compare favorably in market comparisons even though the effective performance may be lowered by this action.
A common misconception is that all parts with "16550A" written on them are identical in performance. There are differences, and in some cases, outright flaws in most of these 16550A clones.
A common misconception is that all parts with "16550A" written on them are identical in performance.
There are differences, and in some cases, outright flaws in most of these 16550A clones.
When the NS16550 was developed, the National Semiconductor obtained several patents on the design and they also limited licensing, making it harder for other vendors to provide a chip with similar features. As a result of the patents, reverse-engineered designs and emulations had to avoid infringing the claims covered by the patents. Subsequently, these copies almost never perform exactly the same as the NS16550A or PC16550D, which are the parts most computer and modem makers want to buy but are sometimes unwilling to pay the price required to get the genuine part.
When the NS16550 was developed, the National Semiconductor obtained several patents on the design and they also limited licensing, making it harder for other vendors to provide a chip with similar features.
As a result of the patents, reverse-engineered designs and emulations had to avoid infringing the claims covered by the patents.
Subsequently, these copies almost never perform exactly the same as the NS16550A or PC16550D, which are the parts most computer and modem makers want to buy but are sometimes unwilling to pay the price required to get the genuine part.
Some of the differences in the clone 16550A parts are unimportant, while others can prevent the device from being used at all with a given operating system or driver. These differences may show up when using other drivers, or when particular combinations of events occur that were not well tested or considered in the Windows(R) driver. This is because most modem vendors and 16550-clone makers use the Microsoft drivers from Windows(R) for Workgroups 3.11 and the Microsoft(R) MS-DOS(R) utility as the primary tests for compatibility with the NS16550A. This over-simplistic criteria means that if a different operating system is used, problems could appear due to subtle differences between the clones and genuine components.
Some of the differences in the clone 16550A parts are unimportant, while others can prevent the device from being used at all with a given operating system or driver.
These differences may show up when using other drivers, or when particular combinations of events occur that were not well tested or considered in the Windows(R) driver.
This is because most modem vendors and 16550-clone makers use the Microsoft drivers from Windows(R) for Workgroups 3.11 and the Microsoft(R) MS-DOS(R) utility as the primary tests for compatibility with the NS16550A.
This over-simplistic criteria means that if a different operating system is used, problems could appear due to subtle differences between the clones and genuine components.
National Semiconductor has made available a program named COMTEST that performs compatibility tests independent of any OS drivers. It should be remembered that the purpose of this type of program is to demonstrate the flaws in the products of the competition, so the program will report major as well as extremely subtle differences in behavior in the part being tested.
National Semiconductor has made available a program named COMTEST that performs compatibility tests independent of any OS drivers.
It should be remembered that the purpose of this type of program is to demonstrate the flaws in the products of the competition, so the program will report major as well as extremely subtle differences in behavior in the part being tested.
In a series of tests performed by the author of this document in 1994, components made by National Semiconductor, TI, StarTech, and CMD as well as megacells and emulations embedded in internal modems were tested with COMTEST. A difference count for some of these components is listed below. Since these tests were performed in 1994, they may not reflect the current performance of the given product from a vendor.
In a series of tests performed by the author of this document in 1994, components made by National Semiconductor, TI, StarTech, and CMD as well as megacells and emulations embedded in internal modems were tested with COMTEST. A difference count for some of these components is listed below.
Since these tests were performed in 1994, they may not reflect the current performance of the given product from a vendor.
It should be noted that COMTEST normally aborts when an excessive number or certain types of problems have been detected. As part of this testing, COMTEST was modified so that it would not abort no matter how many differences were encountered.
It should be noted that COMTEST normally aborts when an excessive number or certain types of problems have been detected.
As part of this testing, COMTEST was modified so that it would not abort no matter how many differences were encountered.
[.informaltable]
[cols="1,1,1", frame="none", options="header"]
@ -517,16 +607,25 @@ It should be noted that COMTEST normally aborts when an excessive number or cert
[NOTE]
====
To date, the author of this document has not found any non-National parts that report zero differences using the COMTEST program. It should also be noted that National has had five versions of the 16550 over the years and the newest parts behave a bit differently than the classic NS16550AFN that is considered the benchmark for functionality. COMTEST appears to turn a blind eye to the differences within the National product line and reports no errors on the National parts (except for the original 16550) even when there are official erratas that describe bugs in the A, B and C revisions of the parts, so this bias in COMTEST must be taken into account.
To date, the author of this document has not found any non-National parts that report zero differences using the COMTEST program.
It should also be noted that National has had five versions of the 16550 over the years and the newest parts behave a bit differently than the classic NS16550AFN that is considered the benchmark for functionality.
COMTEST appears to turn a blind eye to the differences within the National product line and reports no errors on the National parts (except for the original 16550) even when there are official erratas that describe bugs in the A, B and C revisions of the parts, so this bias in COMTEST must be taken into account.
====
It is important to understand that a simple count of differences from COMTEST does not reveal a lot about what differences are important and which are not. For example, about half of the differences reported in the two modems listed above that have internal UARTs were caused by the clone UARTs not supporting five- and six-bit character modes. The real 16550, 16450, and 8250 UARTs all support these modes and COMTEST checks the functionality of these modes so over fifty differences are reported. However, almost no modern modem supports five- or six-bit characters, particularly those with error-correction and compression capabilities. This means that the differences related to five- and six-bit character modes can be discounted.
It is important to understand that a simple count of differences from COMTEST does not reveal a lot about what differences are important and which are not.
For example, about half of the differences reported in the two modems listed above that have internal UARTs were caused by the clone UARTs not supporting five- and six-bit character modes.
The real 16550, 16450, and 8250 UARTs all support these modes and COMTEST checks the functionality of these modes so over fifty differences are reported.
However, almost no modern modem supports five- or six-bit characters, particularly those with error-correction and compression capabilities.
This means that the differences related to five- and six-bit character modes can be discounted.
Many of the differences COMTEST reports have to do with timing. In many of the clone designs, when the host reads from one port, the status bits in some other port may not update in the same amount of time (some faster, some slower) as a _real_ NS16550AFN and COMTEST looks for these differences. This means that the number of differences can be misleading in that one device may only have one or two differences but they are extremely serious, and some other device that updates the status registers faster or slower than the reference part (that would probably never affect the operation of a properly written driver) could have dozens of differences reported.
Many of the differences COMTEST reports have to do with timing.
In many of the clone designs, when the host reads from one port, the status bits in some other port may not update in the same amount of time (some faster, some slower) as a _real_ NS16550AFN and COMTEST looks for these differences.
This means that the number of differences can be misleading in that one device may only have one or two differences but they are extremely serious, and some other device that updates the status registers faster or slower than the reference part (that would probably never affect the operation of a properly written driver) could have dozens of differences reported.
COMTEST can be used as a screening tool to alert the administrator to the presence of potentially incompatible components that might cause problems or have to be handled as a special case.
If you run COMTEST on a 16550 that is in a modem or a modem is attached to the serial port, you need to first issue a ATE0&W command to the modem so that the modem will not echo any of the test characters. If you forget to do this, COMTEST will report at least this one difference:
If you run COMTEST on a 16550 that is in a modem or a modem is attached to the serial port, you need to first issue a ATE0&W command to the modem so that the modem will not echo any of the test characters.
If you forget to do this, COMTEST will report at least this one difference:
[source,shell]
....
@ -535,9 +634,15 @@ Error (6)...Timeout interrupt failed: IIR = c1 LSR = 61
=== 8250/16450/16550 Registers
The 8250/16450/16550 UART occupies eight contiguous I/O port addresses. In the IBM PC, there are two defined locations for these eight ports and they are known collectively as [.filename]#COM1# and [.filename]#COM2#. The makers of PC-clones and add-on cards have created two additional areas known as [.filename]#COM3# and [.filename]#COM4#, but these extra COM ports conflict with other hardware on some systems. The most common conflict is with video adapters that provide IBM 8514 emulation.
The 8250/16450/16550 UART occupies eight contiguous I/O port addresses.
In the IBM PC, there are two defined locations for these eight ports and they are known collectively as [.filename]#COM1# and [.filename]#COM2#.
The makers of PC-clones and add-on cards have created two additional areas known as [.filename]#COM3# and [.filename]#COM4#, but these extra COM ports conflict with other hardware on some systems.
The most common conflict is with video adapters that provide IBM 8514 emulation.
[.filename]#COM1# is located from 0x3f8 to 0x3ff and normally uses IRQ 4. [.filename]#COM2# is located from 0x2f8 to 0x2ff and normally uses IRQ 3. [.filename]#COM3# is located from 0x3e8 to 0x3ef and has no standardized IRQ. [.filename]#COM4# is located from 0x2e8 to 0x2ef and has no standardized IRQ.
[.filename]#COM1# is located from 0x3f8 to 0x3ff and normally uses IRQ 4.
[.filename]#COM2# is located from 0x2f8 to 0x2ff and normally uses IRQ 3.
[.filename]#COM3# is located from 0x3e8 to 0x3ef and has no standardized IRQ.
[.filename]#COM4# is located from 0x2e8 to 0x2ef and has no standardized IRQ.
A description of the I/O ports of the 8250/16450/16550 UART is provided below.
@ -691,29 +796,41 @@ Bit 0 -> Delta Clear To Send (DCTS). Set to "1" if the -CTS line has changed sta
=== Beyond the 16550A UART
Although National Semiconductor has not offered any components compatible with the 16550 that provide additional features, various other vendors have. Some of these components are described below. It should be understood that to effectively utilize these improvements, drivers may have to be provided by the chip vendor since most of the popular operating systems do not support features beyond those provided by the 16550.
Although National Semiconductor has not offered any components compatible with the 16550 that provide additional features, various other vendors have.
Some of these components are described below.
It should be understood that to effectively utilize these improvements, drivers may have to be provided by the chip vendor since most of the popular operating systems do not support features beyond those provided by the 16550.
ST16650::
By default this part is similar to the NS16550A, but an extended 32-byte send and receive buffer can be optionally enabled. Made by StarTech.
By default this part is similar to the NS16550A, but an extended 32-byte send and receive buffer can be optionally enabled.
Made by StarTech.
TIL16660::
By default this part behaves similar to the NS16550A, but an extended 64-byte send and receive buffer can be optionally enabled. Made by Texas Instruments.
By default this part behaves similar to the NS16550A, but an extended 64-byte send and receive buffer can be optionally enabled.
Made by Texas Instruments.
Hayes ESP::
This proprietary plug-in card contains a 2048-byte send and receive buffer, and supports data rates to 230.4Kbit/sec. Made by Hayes.
This proprietary plug-in card contains a 2048-byte send and receive buffer, and supports data rates to 230.4Kbit/sec.
Made by Hayes.
In addition to these "dumb" UARTs, many vendors produce intelligent serial communication boards. This type of design usually provides a microprocessor that interfaces with several UARTs, processes and buffers the data, and then alerts the main PC processor when necessary. As the UARTs are not directly accessed by the PC processor in this type of communication system, it is not necessary for the vendor to use UARTs that are compatible with the 8250, 16450, or the 16550 UART. This leaves the designer free to components that may have better performance characteristics.
In addition to these "dumb" UARTs, many vendors produce intelligent serial communication boards.
This type of design usually provides a microprocessor that interfaces with several UARTs, processes and buffers the data, and then alerts the main PC processor when necessary.
As the UARTs are not directly accessed by the PC processor in this type of communication system, it is not necessary for the vendor to use UARTs that are compatible with the 8250, 16450, or the 16550 UART.
This leaves the designer free to components that may have better performance characteristics.
[[sio]]
== Configuring the [.filename]#sio# driver
The [.filename]#sio# driver provides support for NS8250-, NS16450-, NS16550 and NS16550A-based EIA RS-232C (CCITT V.24) communications interfaces. Several multiport cards are supported as well. See the man:sio[4] manual page for detailed technical documentation.
The [.filename]#sio# driver provides support for NS8250-, NS16450-, NS16550 and NS16550A-based EIA RS-232C (CCITT V.24) communications interfaces.
Several multiport cards are supported as well.
See the man:sio[4] manual page for detailed technical documentation.
=== Digi International (DigiBoard) PC/8
_Contributed by `{awebster}`. 26 August 1995._
Here is a config snippet from a machine with a Digi International PC/8 with 16550. It has 8 modems connected to these 8 lines, and they work just great. Do not forget to add `options COM_MULTIPORT` or it will not work very well!
Here is a config snippet from a machine with a Digi International PC/8 with 16550.
It has 8 modems connected to these 8 lines, and they work just great.
Do not forget to add `options COM_MULTIPORT` or it will not work very well!
[.programlisting]
....
@ -738,9 +855,12 @@ The procedures to make a Boca 16 port board with FreeBSD are pretty straightforw
. You either need the kernel sources installed so you can recompile the necessary options or you will need someone else to compile it for you. The 2.0.5 default kernel does _not_ come with multiport support enabled and you will need to add a device entry for each port anyways.
. Two, you will need to know the interrupt and IO setting for your Boca Board so you can set these options properly in the kernel.
One important note - the actual UART chips for the Boca 16 are in the connector box, not on the internal board itself. So if you have it unplugged, probes of those ports will fail. I have never tested booting with the box unplugged and plugging it back in, and I suggest you do not either.
One important note - the actual UART chips for the Boca 16 are in the connector box, not on the internal board itself.
So if you have it unplugged, probes of those ports will fail.
I have never tested booting with the box unplugged and plugging it back in, and I suggest you do not either.
If you do not already have a custom kernel configuration file set up, refer to link:{handbook}#kernelconfig[Kernel Configuration] chapter of the FreeBSD Handbook for general procedures. The following are the specifics for the Boca 16 board and assume you are using the kernel name MYKERNEL and editing with vi.
If you do not already have a custom kernel configuration file set up, refer to link:{handbook}#kernelconfig[Kernel Configuration] chapter of the FreeBSD Handbook for general procedures.
The following are the specifics for the Boca 16 board and assume you are using the kernel name MYKERNEL and editing with vi.
[.procedure]
====
@ -764,14 +884,18 @@ device sio15 at isa? port 0x170 flags 0x1005
device sio16 at isa? port 0x178 flags 0x1005 irq 3
....
+
The flags entry _must_ be changed from this example unless you are using the exact same sio assignments. Flags are set according to 0x``__MYY__`` where _M_ indicates the minor number of the master port (the last port on a Boca 16) and _YY_ indicates if FIFO is enabled or disabled(enabled), IRQ sharing is used(yes) and if there is an AST/4 compatible IRQ control register(no). In this example,
The flags entry _must_ be changed from this example unless you are using the exact same sio assignments.
Flags are set according to 0x``__MYY__`` where _M_ indicates the minor number of the master port (the last port on a Boca 16) and _YY_ indicates if FIFO is enabled or disabled(enabled), IRQ sharing is used(yes) and if there is an AST/4 compatible IRQ control register(no).
In this example,
+
[.programlisting]
....
flags
0x1005
....
indicates that the master port is sio16. If I added another board and assigned sio17 through sio28, the flags for all 16 ports on _that_ board would be 0x1C05, where 1C indicates the minor number of the master port. Do not change the 05 setting.
indicates that the master port is sio16.
If I added another board and assigned sio17 through sio28, the flags for all 16 ports on _that_ board would be 0x1C05, where 1C indicates the minor number of the master port.
Do not change the 05 setting.
. Save and complete the kernel configuration, recompile, install and reboot. Presuming you have successfully installed the recompiled kernel and have it set to the correct address and IRQ, your boot message should indicate the successful probe of the Boca ports as follows: (obviously the sio numbers, IO and IRQ could be different)
+
[source,shell]
@ -848,7 +972,9 @@ _Contributed by Helge Oldach_ mailto:hmo@sep.hamburg.com[hmo@sep.hamburg.com], S
Ever wondered about FreeBSD support for your 20$ multi-I/O card with two (or more) COM ports, sharing IRQs? Here is how:
Usually the only option to support these kind of boards is to use a distinct IRQ for each port. For example, if your CPU board has an on-board [.filename]#COM1# port (aka [.filename]#sio0#-I/O address 0x3F8 and IRQ 4) and you have an extension board with two UARTs, you will commonly need to configure them as [.filename]#COM2# (aka [.filename]#sio1#-I/O address 0x2F8 and IRQ 3), and the third port (aka [.filename]#sio2#) as I/O 0x3E8 and IRQ 5. Obviously this is a waste of IRQ resources, as it should be basically possible to run both extension board ports using a single IRQ with the `COM_MULTIPORT` configuration described in the previous sections.
Usually the only option to support these kind of boards is to use a distinct IRQ for each port.
For example, if your CPU board has an on-board [.filename]#COM1# port (aka [.filename]#sio0#-I/O address 0x3F8 and IRQ 4) and you have an extension board with two UARTs, you will commonly need to configure them as [.filename]#COM2# (aka [.filename]#sio1#-I/O address 0x2F8 and IRQ 3), and the third port (aka [.filename]#sio2#) as I/O 0x3E8 and IRQ 5.
Obviously this is a waste of IRQ resources, as it should be basically possible to run both extension board ports using a single IRQ with the `COM_MULTIPORT` configuration described in the previous sections.
Such cheap I/O boards commonly have a 4 by 3 jumper matrix for the COM ports, similar to the following:
@ -862,11 +988,16 @@ Port B |
IRQ 2 3 4 5
....
Shown here is port A wired for IRQ 5 and port B wired for IRQ 3. The IRQ columns on your specific board may vary-other boards may supply jumpers for IRQs 3, 4, 5, and 7 instead.
Shown here is port A wired for IRQ 5 and port B wired for IRQ 3.
The IRQ columns on your specific board may vary-other boards may supply jumpers for IRQs 3, 4, 5, and 7 instead.
One could conclude that wiring both ports for IRQ 3 using a handcrafted wire-made jumper covering all three connection points in the IRQ 3 column would solve the issue, but no. You cannot duplicate IRQ 3 because the output drivers of each UART are wired in a "totem pole" fashion, so if one of the UARTs drives IRQ 3, the output signal will not be what you would expect. Depending on the implementation of the extension board or your motherboard, the IRQ 3 line will continuously stay up, or always stay low.
One could conclude that wiring both ports for IRQ 3 using a handcrafted wire-made jumper covering all three connection points in the IRQ 3 column would solve the issue, but no.
You cannot duplicate IRQ 3 because the output drivers of each UART are wired in a "totem pole" fashion, so if one of the UARTs drives IRQ 3, the output signal will not be what you would expect.
Depending on the implementation of the extension board or your motherboard, the IRQ 3 line will continuously stay up, or always stay low.
You need to decouple the IRQ drivers for the two UARTs, so that the IRQ line of the board only goes up if (and only if) one of the UARTs asserts a IRQ, and stays low otherwise. The solution was proposed by Joerg Wunsch mailto:j@ida.interface-business.de[j@ida.interface-business.de]: To solder up a wired-or consisting of two diodes (Germanium or Schottky-types strongly preferred) and a 1 kOhm resistor. Here is the schematic, starting from the 4 by 3 jumper field above:
You need to decouple the IRQ drivers for the two UARTs, so that the IRQ line of the board only goes up if (and only if) one of the UARTs asserts a IRQ, and stays low otherwise.
The solution was proposed by Joerg Wunsch mailto:j@ida.interface-business.de[j@ida.interface-business.de]: To solder up a wired-or consisting of two diodes (Germanium or Schottky-types strongly preferred) and a 1 kOhm resistor.
Here is the schematic, starting from the 4 by 3 jumper field above:
[.programlisting]
....
@ -883,9 +1014,11 @@ Port B `-------------------+ ==+==
IRQ 2 3 4 5 Diode
....
The cathodes of the diodes are connected to a common point, together with a 1 kOhm pull-down resistor. It is essential to connect the resistor to ground to avoid floating of the IRQ line on the bus.
The cathodes of the diodes are connected to a common point, together with a 1 kOhm pull-down resistor.
It is essential to connect the resistor to ground to avoid floating of the IRQ line on the bus.
Now we are ready to configure a kernel. Staying with this example, we would configure:
Now we are ready to configure a kernel.
Staying with this example, we would configure:
[.programlisting]
....
@ -897,7 +1030,9 @@ device sio1 at isa? port "IO_COM2" flags 0x205
device sio2 at isa? port "IO_COM3" flags 0x205 irq 3
....
Note that the `flags` setting for [.filename]#sio1# and [.filename]#sio2# is truly essential; refer to man:sio[4] for details. (Generally, the `2` in the "flags" attribute refers to [.filename]#sio#`2` which holds the IRQ, and you surely want a `5` low nibble.) With kernel verbose mode turned on this should yield something similar to this:
Note that the `flags` setting for [.filename]#sio1# and [.filename]#sio2# is truly essential; refer to man:sio[4] for details.
(Generally, the `2` in the "flags" attribute refers to [.filename]#sio#`2` which holds the IRQ, and you surely want a `5` low nibble.)
With kernel verbose mode turned on this should yield something similar to this:
[source,shell]
....
@ -912,14 +1047,17 @@ sio2 at 0x3e8-0x3ef irq 3 flags 0x205 on isa
sio2: type 16550A (multiport master)
....
Though [.filename]#/sys/i386/isa/sio.c# is somewhat cryptic with its use of the "irq maps" array above, the basic idea is that you observe `0x1` in the first, third, and fourth place. This means that the corresponding IRQ was set upon output and cleared after, which is just what we would expect. If your kernel does not display this behavior, most likely there is something wrong with your wiring.
Though [.filename]#/sys/i386/isa/sio.c# is somewhat cryptic with its use of the "irq maps" array above, the basic idea is that you observe `0x1` in the first, third, and fourth place.
This means that the corresponding IRQ was set upon output and cleared after, which is just what we would expect.
If your kernel does not display this behavior, most likely there is something wrong with your wiring.
[[cy]]
== Configuring the [.filename]#cy# driver
_Contributed by Alex Nash. 6 June 1996._
The Cyclades multiport cards are based on the [.filename]#cy# driver instead of the usual [.filename]#sio# driver used by other multiport cards. Configuration is a simple matter of:
The Cyclades multiport cards are based on the [.filename]#cy# driver instead of the usual [.filename]#sio# driver used by other multiport cards.
Configuration is a simple matter of:
[.procedure]
====
@ -955,7 +1093,9 @@ ttyc7 "/usr/libexec/getty std.38400" unknown on insecure
_Contributed by `{nsayer}`. 25 March 1998._
The Specialix SI/XIO and SX multiport cards use the [.filename]#si# driver. A single machine can have up to 4 host cards. The following host cards are supported:
The Specialix SI/XIO and SX multiport cards use the [.filename]#si# driver.
A single machine can have up to 4 host cards.
The following host cards are supported:
* ISA SI/XIO host card (2 versions)
* EISA SI/XIO host card
@ -963,9 +1103,15 @@ The Specialix SI/XIO and SX multiport cards use the [.filename]#si# driver. A si
* ISA SX host card
* PCI SX host card
Although the SX and SI/XIO host cards look markedly different, their functionality are basically the same. The host cards do not use I/O locations, but instead require a 32K chunk of memory. The factory configuration for ISA cards places this at `0xd0000-0xd7fff`. They also require an IRQ. PCI cards will, of course, auto-configure themselves.
Although the SX and SI/XIO host cards look markedly different, their functionality are basically the same.
The host cards do not use I/O locations, but instead require a 32K chunk of memory.
The factory configuration for ISA cards places this at `0xd0000-0xd7fff`.
They also require an IRQ.
PCI cards will, of course, auto-configure themselves.
You can attach up to 4 external modules to each host card. The external modules contain either 4 or 8 serial ports. They come in the following varieties:
You can attach up to 4 external modules to each host card.
The external modules contain either 4 or 8 serial ports.
They come in the following varieties:
* SI 4 or 8 port modules. Up to 57600 bps on each port supported.
* XIO 8 port modules. Up to 115200 bps on each port supported. One type of XIO module has 7 serial and 1 parallel port.
@ -994,7 +1140,9 @@ After adding the configuration entry, rebuild and install your new kernel.
The following step, is not necessary if you are using man:devfs[5] in FreeBSD 5._X_.
====
After rebooting with the new kernel, you need to make the device nodes in [.filename]#/dev#. The [.filename]#MAKEDEV# script will take care of this for you. Count how many total ports you have and type:
After rebooting with the new kernel, you need to make the device nodes in [.filename]#/dev#.
The [.filename]#MAKEDEV# script will take care of this for you.
Count how many total ports you have and type:
[source,shell]
....
@ -1011,4 +1159,5 @@ If you want login prompts to appear on these ports, you will need to add lines l
ttyA01 "/usr/libexec/getty std.9600" vt100 on insecure
....
Change the terminal type as appropriate. For modems, `dialup` or `unknown` is fine.
Change the terminal type as appropriate.
For modems, `dialup` or `unknown` is fine.

107
documentation/content/en/articles/solid-state/_index.adoc
View file

@ -23,9 +23,11 @@ Abstract
This article covers the use of solid state disk devices in FreeBSD to create embedded systems.
Embedded systems have the advantage of increased stability due to the lack of integral moving parts (hard drives). Account must be taken, however, for the generally low disk space available in the system and the durability of the storage medium.
Embedded systems have the advantage of increased stability due to the lack of integral moving parts (hard drives).
Account must be taken, however, for the generally low disk space available in the system and the durability of the storage medium.
Specific topics to be covered include the types and attributes of solid state media suitable for disk use in FreeBSD, kernel options that are of interest in such an environment, the [.filename]#rc.initdiskless# mechanisms that automate the initialization of such systems and the need for read-only filesystems, and building filesystems from scratch. The article will conclude with some general strategies for small and read-only FreeBSD environments.
Specific topics to be covered include the types and attributes of solid state media suitable for disk use in FreeBSD, kernel options that are of interest in such an environment, the [.filename]#rc.initdiskless# mechanisms that automate the initialization of such systems and the need for read-only filesystems, and building filesystems from scratch.
The article will conclude with some general strategies for small and read-only FreeBSD environments.
'''
@ -34,9 +36,19 @@ toc::[]
[[intro]]
== Solid State Disk Devices
The scope of this article will be limited to solid state disk devices made from flash memory. Flash memory is a solid state memory (no moving parts) that is non-volatile (the memory maintains data even after all power sources have been disconnected). Flash memory can withstand tremendous physical shock and is reasonably fast (the flash memory solutions covered in this article are slightly slower than a EIDE hard disk for write operations, and much faster for read operations). One very important aspect of flash memory, the ramifications of which will be discussed later in this article, is that each sector has a limited rewrite capacity. You can only write, erase, and write again to a sector of flash memory a certain number of times before the sector becomes permanently unusable. Although many flash memory products automatically map bad blocks, and although some even distribute write operations evenly throughout the unit, the fact remains that there exists a limit to the amount of writing that can be done to the device. Competitive units have between 1,000,000 and 10,000,000 writes per sector in their specification. This figure varies due to the temperature of the environment.
The scope of this article will be limited to solid state disk devices made from flash memory.
Flash memory is a solid state memory (no moving parts) that is non-volatile (the memory maintains data even after all power sources have been disconnected).
Flash memory can withstand tremendous physical shock and is reasonably fast (the flash memory solutions covered in this article are slightly slower than a EIDE hard disk for write operations, and much faster for read operations).
One very important aspect of flash memory, the ramifications of which will be discussed later in this article, is that each sector has a limited rewrite capacity.
You can only write, erase, and write again to a sector of flash memory a certain number of times before the sector becomes permanently unusable.
Although many flash memory products automatically map bad blocks, and although some even distribute write operations evenly throughout the unit, the fact remains that there exists a limit to the amount of writing that can be done to the device.
Competitive units have between 1,000,000 and 10,000,000 writes per sector in their specification.
This figure varies due to the temperature of the environment.
Specifically, we will be discussing ATA compatible compact-flash units, which are quite popular as storage media for digital cameras. Of particular interest is the fact that they pin out directly to the IDE bus and are compatible with the ATA command set. Therefore, with a very simple and low-cost adaptor, these devices can be attached directly to an IDE bus in a computer. Once implemented in this manner, operating systems such as FreeBSD see the device as a normal hard disk (albeit small).
Specifically, we will be discussing ATA compatible compact-flash units, which are quite popular as storage media for digital cameras.
Of particular interest is the fact that they pin out directly to the IDE bus and are compatible with the ATA command set.
Therefore, with a very simple and low-cost adaptor, these devices can be attached directly to an IDE bus in a computer.
Once implemented in this manner, operating systems such as FreeBSD see the device as a normal hard disk (albeit small).
Other solid state disk solutions do exist, but their expense, obscurity, and relative unease of use places them beyond the scope of this article.
@ -45,7 +57,10 @@ Other solid state disk solutions do exist, but their expense, obscurity, and rel
A few kernel options are of specific interest to those creating an embedded FreeBSD system.
All embedded FreeBSD systems that use flash memory as system disk will be interested in memory disks and memory filesystems. As a result of the limited number of writes that can be done to flash memory, the disk and the filesystems on the disk will most likely be mounted read-only. In this environment, filesystems such as [.filename]#/tmp# and [.filename]#/var# are mounted as memory filesystems to allow the system to create logs and update counters and temporary files. Memory filesystems are a critical component to a successful solid state FreeBSD implementation.
All embedded FreeBSD systems that use flash memory as system disk will be interested in memory disks and memory filesystems.
As a result of the limited number of writes that can be done to flash memory, the disk and the filesystems on the disk will most likely be mounted read-only.
In this environment, filesystems such as [.filename]#/tmp# and [.filename]#/var# are mounted as memory filesystems to allow the system to create logs and update counters and temporary files.
Memory filesystems are a critical component to a successful solid state FreeBSD implementation.
You should make sure the following lines exist in your kernel configuration file:
@ -61,7 +76,9 @@ pseudo-device md # memory disk
The post-boot initialization of an embedded FreeBSD system is controlled by [.filename]#/etc/rc.initdiskless#.
[.filename]#/etc/rc.d/var# mounts [.filename]#/var# as a memory filesystem, makes a configurable list of directories in [.filename]#/var# with the man:mkdir[1] command, and changes modes on some of those directories. In the execution of [.filename]#/etc/rc.d/var#, one other [.filename]#rc.conf# variable comes into play - `varsize`. A [.filename]#/var# partition is created by [.filename]#/etc/rc.d/var# based on the value of this variable in [.filename]#rc.conf#:
[.filename]#/etc/rc.d/var# mounts [.filename]#/var# as a memory filesystem, makes a configurable list of directories in [.filename]#/var# with the man:mkdir[1] command, and changes modes on some of those directories.
In the execution of [.filename]#/etc/rc.d/var#, one other [.filename]#rc.conf# variable comes into play - `varsize`.
A [.filename]#/var# partition is created by [.filename]#/etc/rc.d/var# based on the value of this variable in [.filename]#rc.conf#:
[.programlisting]
....
@ -70,7 +87,12 @@ varsize=8192
Remember that this value is in sectors by default.
The fact that [.filename]#/var# is a read-write filesystem is an important distinction, as the [.filename]#/# partition (and any other partitions you may have on your flash media) should be mounted read-only. Remember that in <<intro>> we detailed the limitations of flash memory - specifically the limited write capability. The importance of not mounting filesystems on flash media read-write, and the importance of not using a swap file, cannot be overstated. A swap file on a busy system can burn through a piece of flash media in less than one year. Heavy logging or temporary file creation and destruction can do the same. Therefore, in addition to removing the `swap` entry from your [.filename]#/etc/fstab#, you should also change the Options field for each filesystem to `ro` as follows:
The fact that [.filename]#/var# is a read-write filesystem is an important distinction, as the [.filename]#/# partition (and any other partitions you may have on your flash media) should be mounted read-only.
Remember that in <<intro>> we detailed the limitations of flash memory - specifically the limited write capability.
The importance of not mounting filesystems on flash media read-write, and the importance of not using a swap file, cannot be overstated.
A swap file on a busy system can burn through a piece of flash media in less than one year.
Heavy logging or temporary file creation and destruction can do the same.
Therefore, in addition to removing the `swap` entry from your [.filename]#/etc/fstab#, you should also change the Options field for each filesystem to `ro` as follows:
[.programlisting]
....
@ -78,7 +100,9 @@ The fact that [.filename]#/var# is a read-write filesystem is an important disti
/dev/ad0s1a / ufs ro 1 1
....
A few applications in the average system will immediately begin to fail as a result of this change. For instance, cron will not run properly as a result of missing cron tabs in the [.filename]#/var# created by [.filename]#/etc/rc.d/var#, and syslog and dhcp will encounter problems as well as a result of the read-only filesystem and missing items in the [.filename]#/var# that [.filename]#/etc/rc.d/var# has created. These are only temporary problems though, and are addressed, along with solutions to the execution of other common software packages in <<strategies>>.
A few applications in the average system will immediately begin to fail as a result of this change.
For instance, cron will not run properly as a result of missing cron tabs in the [.filename]#/var# created by [.filename]#/etc/rc.d/var#, and syslog and dhcp will encounter problems as well as a result of the read-only filesystem and missing items in the [.filename]#/var# that [.filename]#/etc/rc.d/var# has created.
These are only temporary problems though, and are addressed, along with solutions to the execution of other common software packages in <<strategies>>.
An important thing to remember is that a filesystem that was mounted read-only with [.filename]#/etc/fstab# can be made read-write at any time by issuing the command:
@ -98,32 +122,49 @@ and can be toggled back to read-only with the command:
Since ATA compatible compact-flash cards are seen by FreeBSD as normal IDE hard drives, you could theoretically install FreeBSD from the network using the kern and mfsroot floppies or from a CD.
However, even a small installation of FreeBSD using normal installation procedures can produce a system in size of greater than 200 megabytes. Most people will be using smaller flash memory devices (128 megabytes is considered fairly large - 32 or even 16 megabytes is common), so an installation using normal mechanisms is not possible-there is simply not enough disk space for even the smallest of conventional installations.
However, even a small installation of FreeBSD using normal installation procedures can produce a system in size of greater than 200 megabytes.
Most people will be using smaller flash memory devices (128 megabytes is considered fairly large - 32 or even 16 megabytes is common), so an installation using normal mechanisms is not possible-there is simply not enough disk space for even the smallest of conventional installations.
The easiest way to overcome this space limitation is to install FreeBSD using conventional means to a normal hard disk. After the installation is complete, pare down the operating system to a size that will fit onto your flash media, then tar the entire filesystem. The following steps will guide you through the process of preparing a piece of flash memory for your tarred filesystem. Remember, because a normal installation is not being performed, operations such as partitioning, labeling, file-system creation, etc. need to be performed by hand. In addition to the kern and mfsroot floppy disks, you will also need to use the fixit floppy.
The easiest way to overcome this space limitation is to install FreeBSD using conventional means to a normal hard disk.
After the installation is complete, pare down the operating system to a size that will fit onto your flash media, then tar the entire filesystem.
The following steps will guide you through the process of preparing a piece of flash memory for your tarred filesystem.
Remember, because a normal installation is not being performed, operations such as partitioning, labeling, file-system creation, etc. need to be performed by hand.
In addition to the kern and mfsroot floppy disks, you will also need to use the fixit floppy.
[.procedure]
====
. Partitioning Your Flash Media Device
+
After booting with the kern and mfsroot floppies, choose `custom` from the installation menu. In the custom installation menu, choose `partition`. In the partition menu, you should delete all existing partitions using kbd:[d]. After deleting all existing partitions, create a partition using kbd:[c] and accept the default value for the size of the partition. When asked for the type of the partition, make sure the value is set to `165`. Now write this partition table to the disk by pressing kbd:[w] (this is a hidden option on this screen). If you are using an ATA compatible compact flash card, you should choose the FreeBSD Boot Manager. Now press kbd:[q] to quit the partition menu. You will be shown the boot manager menu once more - repeat the choice you made earlier.
After booting with the kern and mfsroot floppies, choose `custom` from the installation menu.
In the custom installation menu, choose `partition`.
In the partition menu, you should delete all existing partitions using kbd:[d].
After deleting all existing partitions, create a partition using kbd:[c] and accept the default value for the size of the partition.
When asked for the type of the partition, make sure the value is set to `165`.
Now write this partition table to the disk by pressing kbd:[w] (this is a hidden option on this screen).
If you are using an ATA compatible compact flash card, you should choose the FreeBSD Boot Manager.
Now press kbd:[q] to quit the partition menu.
You will be shown the boot manager menu once more - repeat the choice you made earlier.
. Creating Filesystems on Your Flash Memory Device
+
Exit the custom installation menu, and from the main installation menu choose the `fixit` option. After entering the fixit environment, enter the following command:
Exit the custom installation menu, and from the main installation menu choose the `fixit` option.
After entering the fixit environment, enter the following command:
+
[source,shell]
....
# disklabel -e /dev/ad0c
....
+
At this point you will have entered the vi editor under the auspices of the disklabel command. Next, you need to add an `a:` line at the end of the file. This `a:` line should look like:
At this point you will have entered the vi editor under the auspices of the disklabel command.
Next, you need to add an `a:` line at the end of the file. This `a:` line should look like:
+
[.programlisting]
....
a: 123456 0 4.2BSD 0 0
....
+
Where _123456_ is a number that is exactly the same as the number in the existing `c:` entry for size. Basically you are duplicating the existing `c:` line as an `a:` line, making sure that fstype is `4.2BSD`. Save the file and exit.
Where _123456_ is a number that is exactly the same as the number in the existing `c:` entry for size.
Basically you are duplicating the existing `c:` line as an `a:` line, making sure that fstype is `4.2BSD`.
Save the file and exit.
+
[source,shell]
....
@ -140,7 +181,8 @@ Mount the newly prepared flash media:
# mount /dev/ad0a /flash
....
+
Bring this machine up on the network so we may transfer our tar file and explode it onto our flash media filesystem. One example of how to do this is:
Bring this machine up on the network so we may transfer our tar file and explode it onto our flash media filesystem.
One example of how to do this is:
+
[source,shell]
....
@ -148,7 +190,10 @@ Bring this machine up on the network so we may transfer our tar file and explode
# route add default 192.168.0.1
....
+
Now that the machine is on the network, transfer your tar file. You may be faced with a bit of a dilemma at this point - if your flash memory part is 128 megabytes, for instance, and your tar file is larger than 64 megabytes, you cannot have your tar file on the flash media at the same time as you explode it - you will run out of space. One solution to this problem, if you are using FTP, is to untar the file while it is transferred over FTP. If you perform your transfer in this manner, you will never have the tar file and the tar contents on your disk at the same time:
Now that the machine is on the network, transfer your tar file.
You may be faced with a bit of a dilemma at this point - if your flash memory part is 128 megabytes, for instance, and your tar file is larger than 64 megabytes, you cannot have your tar file on the flash media at the same time as you explode it - you will run out of space.
One solution to this problem, if you are using FTP, is to untar the file while it is transferred over FTP.
If you perform your transfer in this manner, you will never have the tar file and the tar contents on your disk at the same time:
+
[source,shell]
....
@ -177,17 +222,23 @@ Assuming that you configured your filesystem correctly when it was built on the
[[strategies]]
== System Strategies for Small and Read Only Environments
In <<ro-fs>>, it was pointed out that the [.filename]#/var# filesystem constructed by [.filename]#/etc/rc.d/var# and the presence of a read-only root filesystem causes problems with many common software packages used with FreeBSD. In this article, suggestions for successfully running cron, syslog, ports installations, and the Apache web server will be provided.
In <<ro-fs>>, it was pointed out that the [.filename]#/var# filesystem constructed by [.filename]#/etc/rc.d/var# and the presence of a read-only root filesystem causes problems with many common software packages used with FreeBSD.
In this article, suggestions for successfully running cron, syslog, ports installations, and the Apache web server will be provided.
=== Cron
Upon boot, [.filename]#/var# gets populated by [.filename]#/etc/rc.d/var# using the list from [.filename]#/etc/mtree/BSD.var.dist#, so the [.filename]#cron#, [.filename]#cron/tabs#, [.filename]#at#, and a few other standard directories get created.
However, this does not solve the problem of maintaining cron tabs across reboots. When the system reboots, the [.filename]#/var# filesystem that is in memory will disappear and any cron tabs you may have had in it will also disappear. Therefore, one solution would be to create cron tabs for the users that need them, mount your [.filename]#/# filesystem as read-write and copy those cron tabs to somewhere safe, like [.filename]#/etc/tabs#, then add a line to the end of [.filename]#/etc/rc.initdiskless# that copies those crontabs into [.filename]#/var/cron/tabs# after that directory has been created during system initialization. You may also need to add a line that changes modes and permissions on the directories you create and the files you copy with [.filename]#/etc/rc.initdiskless#.
However, this does not solve the problem of maintaining cron tabs across reboots.
When the system reboots, the [.filename]#/var# filesystem that is in memory will disappear and any cron tabs you may have had in it will also disappear.
Therefore, one solution would be to create cron tabs for the users that need them, mount your [.filename]#/# filesystem as read-write and copy those cron tabs to somewhere safe, like [.filename]#/etc/tabs#, then add a line to the end of [.filename]#/etc/rc.initdiskless# that copies those crontabs into [.filename]#/var/cron/tabs# after that directory has been created during system initialization.
You may also need to add a line that changes modes and permissions on the directories you create and the files you copy with [.filename]#/etc/rc.initdiskless#.
=== Syslog
[.filename]#syslog.conf# specifies the locations of certain log files that exist in [.filename]#/var/log#. These files are not created by [.filename]#/etc/rc.d/var# upon system initialization. Therefore, somewhere in [.filename]#/etc/rc.d/var#, after the section that creates the directories in [.filename]#/var#, you will need to add something like this:
[.filename]#syslog.conf# specifies the locations of certain log files that exist in [.filename]#/var/log#.
These files are not created by [.filename]#/etc/rc.d/var# upon system initialization.
Therefore, somewhere in [.filename]#/etc/rc.d/var#, after the section that creates the directories in [.filename]#/var#, you will need to add something like this:
[source,shell]
....
@ -197,11 +248,15 @@ However, this does not solve the problem of maintaining cron tabs across reboots
=== Ports Installation
Before discussing the changes necessary to successfully use the ports tree, a reminder is necessary regarding the read-only nature of your filesystems on the flash media. Since they are read-only, you will need to temporarily mount them read-write using the mount syntax shown in <<ro-fs>>. You should always remount those filesystems read-only when you are done with any maintenance - unnecessary writes to the flash media could considerably shorten its lifespan.
Before discussing the changes necessary to successfully use the ports tree, a reminder is necessary regarding the read-only nature of your filesystems on the flash media.
Since they are read-only, you will need to temporarily mount them read-write using the mount syntax shown in <<ro-fs>>.
You should always remount those filesystems read-only when you are done with any maintenance - unnecessary writes to the flash media could considerably shorten its lifespan.
To make it possible to enter a ports directory and successfully run `make install`, we must create a packages directory on a non-memory filesystem that will keep track of our packages across reboots. As it is necessary to mount your filesystems as read-write for the installation of a package anyway, it is sensible to assume that an area on the flash media can also be used for package information to be written to.
To make it possible to enter a ports directory and successfully run `make install`, we must create a packages directory on a non-memory filesystem that will keep track of our packages across reboots.
As it is necessary to mount your filesystems as read-write for the installation of a package anyway, it is sensible to assume that an area on the flash media can also be used for package information to be written to.
First, create a package database directory. This is normally in [.filename]#/var/db/pkg#, but we cannot place it there as it will disappear every time the system is booted.
First, create a package database directory.
This is normally in [.filename]#/var/db/pkg#, but we cannot place it there as it will disappear every time the system is booted.
[source,shell]
....
@ -221,10 +276,14 @@ Now, any time that you mount your filesystems as read-write and install a packag
[NOTE]
====
The steps in this section are only necessary if Apache is set up to write its pid or log information outside of [.filename]#/var#. By default, Apache keeps its pid file in [.filename]#/var/run/httpd.pid# and its log files in [.filename]#/var/log#.
The steps in this section are only necessary if Apache is set up to write its pid or log information outside of [.filename]#/var#.
By default, Apache keeps its pid file in [.filename]#/var/run/httpd.pid# and its log files in [.filename]#/var/log#.
====
It is now assumed that Apache keeps its log files in a directory [.filename]#apache_log_dir# outside of [.filename]#/var#. When this directory lives on a read-only filesystem, Apache will not be able to save any log files, and may have problems working. If so, it is necessary to add a new directory to the list of directories in [.filename]#/etc/rc.d/var# to create in [.filename]#/var#, and to link [.filename]#apache_log_dir# to [.filename]#/var/log/apache#. It is also necessary to set permissions and ownership on this new directory.
It is now assumed that Apache keeps its log files in a directory [.filename]#apache_log_dir# outside of [.filename]#/var#.
When this directory lives on a read-only filesystem, Apache will not be able to save any log files, and may have problems working.
If so, it is necessary to add a new directory to the list of directories in [.filename]#/etc/rc.d/var# to create in [.filename]#/var#, and to link [.filename]#apache_log_dir# to [.filename]#/var/log/apache#.
It is also necessary to set permissions and ownership on this new directory.
First, add the directory `log/apache` to the list of directories to be created in [.filename]#/etc/rc.d/var#.

243
documentation/content/en/articles/vinum/_index.adoc
View file

@ -17,7 +17,7 @@ authors:
include::shared/en/urls.adoc[]
ifeval::["{backend}" == "html5"]
:imagesdir: ../../images/articles/vinum/
:imagesdir: ../../../images/articles/vinum/
endif::[]
ifeval::["{backend}" == "pdf"]
@ -35,41 +35,72 @@ toc::[]
[[vinum-synopsis]]
== Synopsis
No matter the type of disks, there are always potential problems. The disks can be too small, too slow, or too unreliable to meet the system's requirements. While disks are getting bigger, so are data storage requirements. Often a file system is needed that is bigger than a disk's capacity. Various solutions to these problems have been proposed and implemented.
No matter the type of disks, there are always potential problems.
The disks can be too small, too slow, or too unreliable to meet the system's requirements.
While disks are getting bigger, so are data storage requirements.
Often a file system is needed that is bigger than a disk's capacity.
Various solutions to these problems have been proposed and implemented.
One method is through the use of multiple, and sometimes redundant, disks. In addition to supporting various cards and controllers for hardware Redundant Array of Independent Disks RAID systems, the base FreeBSD system includes the [.filename]#vinum# volume manager, a block device driver that implements virtual disk drives and addresses these three problems. [.filename]#vinum# provides more flexibility, performance, and reliability than traditional disk storage and implements `RAID`-0, `RAID`-1, and `RAID`-5 models, both individually and in combination.
One method is through the use of multiple, and sometimes redundant, disks.
In addition to supporting various cards and controllers for hardware Redundant Array of Independent Disks RAID systems, the base FreeBSD system includes the [.filename]#vinum# volume manager, a block device driver that implements virtual disk drives and addresses these three problems.
[.filename]#vinum# provides more flexibility, performance, and reliability than traditional disk storage and implements `RAID`-0, `RAID`-1, and `RAID`-5 models, both individually and in combination.
This chapter provides an overview of potential problems with traditional disk storage, and an introduction to the [.filename]#vinum# volume manager.
[NOTE]
====
Starting with FreeBSD 5, [.filename]#vinum# has been rewritten in order to fit into the link:{handbook}#geom[GEOM architecture], while retaining the original ideas, terminology, and on-disk metadata. This rewrite is called _gvinum_ (for _GEOM vinum_). While this chapter uses the term [.filename]#vinum#, any command invocations should be performed with `gvinum`. The name of the kernel module has changed from the original [.filename]#vinum.ko# to [.filename]#geom_vinum.ko#, and all device nodes reside under [.filename]#/dev/gvinum# instead of [.filename]#/dev/vinum#. As of FreeBSD 6, the original [.filename]#vinum# implementation is no longer available in the code base.
Starting with FreeBSD 5, [.filename]#vinum# has been rewritten in order to fit into the link:{handbook}#geom[GEOM architecture], while retaining the original ideas, terminology, and on-disk metadata.
This rewrite is called _gvinum_ (for _GEOM vinum_).
While this chapter uses the term [.filename]#vinum#, any command invocations should be performed with `gvinum`.
The name of the kernel module has changed from the original [.filename]#vinum.ko# to [.filename]#geom_vinum.ko#, and all device nodes reside under [.filename]#/dev/gvinum# instead of [.filename]#/dev/vinum#.
As of FreeBSD 6, the original [.filename]#vinum# implementation is no longer available in the code base.
====
[[vinum-access-bottlenecks]]
== Access Bottlenecks
Modern systems frequently need to access data in a highly concurrent manner. For example, large FTP or HTTP servers can maintain thousands of concurrent sessions and have multiple 100 Mbit/s connections to the outside world, well beyond the sustained transfer rate of most disks.
Modern systems frequently need to access data in a highly concurrent manner.
For example, large FTP or HTTP servers can maintain thousands of concurrent sessions and have multiple 100 Mbit/s connections to the outside world, well beyond the sustained transfer rate of most disks.
Current disk drives can transfer data sequentially at up to 70 MB/s, but this value is of little importance in an environment where many independent processes access a drive, and where they may achieve only a fraction of these values. In such cases, it is more interesting to view the problem from the viewpoint of the disk subsystem. The important parameter is the load that a transfer places on the subsystem, or the time for which a transfer occupies the drives involved in the transfer.
Current disk drives can transfer data sequentially at up to 70 MB/s, but this value is of little importance in an environment where many independent processes access a drive, and where they may achieve only a fraction of these values.
In such cases, it is more interesting to view the problem from the viewpoint of the disk subsystem.
The important parameter is the load that a transfer places on the subsystem, or the time for which a transfer occupies the drives involved in the transfer.
In any disk transfer, the drive must first position the heads, wait for the first sector to pass under the read head, and then perform the transfer. These actions can be considered to be atomic as it does not make any sense to interrupt them.
In any disk transfer, the drive must first position the heads, wait for the first sector to pass under the read head, and then perform the transfer.
These actions can be considered to be atomic as it does not make any sense to interrupt them.
[[vinum-latency]] Consider a typical transfer of about 10 kB: the current generation of high-performance disks can position the heads in an average of 3.5 ms. The fastest drives spin at 15,000 rpm, so the average rotational latency (half a revolution) is 2 ms. At 70 MB/s, the transfer itself takes about 150 μs, almost nothing compared to the positioning time. In such a case, the effective transfer rate drops to a little over 1 MB/s and is clearly highly dependent on the transfer size.
[[vinum-latency]] Consider a typical transfer of about 10 kB: the current generation of high-performance disks can position the heads in an average of 3.5 ms.
The fastest drives spin at 15,000 rpm, so the average rotational latency (half a revolution) is 2 ms.
At 70 MB/s, the transfer itself takes about 150 μs, almost nothing compared to the positioning time.
In such a case, the effective transfer rate drops to a little over 1 MB/s and is clearly highly dependent on the transfer size.
The traditional and obvious solution to this bottleneck is "more spindles": rather than using one large disk, use several smaller disks with the same aggregate storage space. Each disk is capable of positioning and transferring independently, so the effective throughput increases by a factor close to the number of disks used.
The traditional and obvious solution to this bottleneck is "more spindles": rather than using one large disk, use several smaller disks with the same aggregate storage space.
Each disk is capable of positioning and transferring independently, so the effective throughput increases by a factor close to the number of disks used.
The actual throughput improvement is smaller than the number of disks involved. Although each drive is capable of transferring in parallel, there is no way to ensure that the requests are evenly distributed across the drives. Inevitably the load on one drive will be higher than on another.
The actual throughput improvement is smaller than the number of disks involved.
Although each drive is capable of transferring in parallel, there is no way to ensure that the requests are evenly distributed across the drives.
Inevitably the load on one drive will be higher than on another.
The evenness of the load on the disks is strongly dependent on the way the data is shared across the drives. In the following discussion, it is convenient to think of the disk storage as a large number of data sectors which are addressable by number, rather like the pages in a book. The most obvious method is to divide the virtual disk into groups of consecutive sectors the size of the individual physical disks and store them in this manner, rather like taking a large book and tearing it into smaller sections. This method is called _concatenation_ and has the advantage that the disks are not required to have any specific size relationships. It works well when the access to the virtual disk is spread evenly about its address space. When access is concentrated on a smaller area, the improvement is less marked. <<vinum-concat, Concatenated Organization>> illustrates the sequence in which storage units are allocated in a concatenated organization.
The evenness of the load on the disks is strongly dependent on the way the data is shared across the drives.
In the following discussion, it is convenient to think of the disk storage as a large number of data sectors which are addressable by number, rather like the pages in a book.
The most obvious method is to divide the virtual disk into groups of consecutive sectors the size of the individual physical disks and store them in this manner, rather like taking a large book and tearing it into smaller sections.
This method is called _concatenation_ and has the advantage that the disks are not required to have any specific size relationships.
It works well when the access to the virtual disk is spread evenly about its address space.
When access is concentrated on a smaller area, the improvement is less marked.
<<vinum-concat, Concatenated Organization>> illustrates the sequence in which storage units are allocated in a concatenated organization.
[[vinum-concat]]
.Concatenated Organization
image::vinum-concat.png[]
An alternative mapping is to divide the address space into smaller, equal-sized components and store them sequentially on different devices. For example, the first 256 sectors may be stored on the first disk, the next 256 sectors on the next disk and so on. After filling the last disk, the process repeats until the disks are full. This mapping is called _striping_ or RAID-0.
An alternative mapping is to divide the address space into smaller, equal-sized components and store them sequentially on different devices.
For example, the first 256 sectors may be stored on the first disk, the next 256 sectors on the next disk and so on.
After filling the last disk, the process repeats until the disks are full.
This mapping is called _striping_ or RAID-0.
`RAID` offers various forms of fault tolerance, though RAID-0 is somewhat misleading as it provides no redundancy. Striping requires somewhat more effort to locate the data, and it can cause additional I/O load where a transfer is spread over multiple disks, but it can also provide a more constant load across the disks. <<vinum-striped, Striped Organization>> illustrates the sequence in which storage units are allocated in a striped organization.
`RAID` offers various forms of fault tolerance, though RAID-0 is somewhat misleading as it provides no redundancy.
Striping requires somewhat more effort to locate the data, and it can cause additional I/O load where a transfer is spread over multiple disks, but it can also provide a more constant load across the disks.
<<vinum-striped, Striped Organization>> illustrates the sequence in which storage units are allocated in a striped organization.
[[vinum-striped]]
.Striped Organization
@ -78,22 +109,32 @@ image::vinum-striped.png[]
[[vinum-data-integrity]]
== Data Integrity
The final problem with disks is that they are unreliable. Although reliability has increased tremendously over the last few years, disk drives are still the most likely core component of a server to fail. When they do, the results can be catastrophic and replacing a failed disk drive and restoring data can result in server downtime.
The final problem with disks is that they are unreliable.
Although reliability has increased tremendously over the last few years, disk drives are still the most likely core component of a server to fail.
When they do, the results can be catastrophic and replacing a failed disk drive and restoring data can result in server downtime.
One approach to this problem is _mirroring_, or `RAID-1`, which keeps two copies of the data on different physical hardware. Any write to the volume writes to both disks; a read can be satisfied from either, so if one drive fails, the data is still available on the other drive.
One approach to this problem is _mirroring_, or `RAID-1`, which keeps two copies of the data on different physical hardware.
Any write to the volume writes to both disks; a read can be satisfied from either, so if one drive fails, the data is still available on the other drive.
Mirroring has two problems:
* It requires twice as much disk storage as a non-redundant solution.
* Writes must be performed to both drives, so they take up twice the bandwidth of a non-mirrored volume. Reads do not suffer from a performance penalty and can even be faster.
An alternative solution is _parity_, implemented in `RAID` levels 2, 3, 4 and 5. Of these, `RAID-5` is the most interesting. As implemented in [.filename]#vinum#, it is a variant on a striped organization which dedicates one block of each stripe to parity one of the other blocks. As implemented by [.filename]#vinum#, a `RAID-5` plex is similar to a striped plex, except that it implements `RAID-5` by including a parity block in each stripe. As required by `RAID-5`, the location of this parity block changes from one stripe to the next. The numbers in the data blocks indicate the relative block numbers.
An alternative solution is _parity_, implemented in `RAID` levels 2, 3, 4 and 5.
Of these, `RAID-5` is the most interesting.
As implemented in [.filename]#vinum#, it is a variant on a striped organization which dedicates one block of each stripe to parity one of the other blocks.
As implemented by [.filename]#vinum#, a `RAID-5` plex is similar to a striped plex, except that it implements `RAID-5` by including a parity block in each stripe.
As required by `RAID-5`, the location of this parity block changes from one stripe to the next.
The numbers in the data blocks indicate the relative block numbers.
[[vinum-raid5-org]]
.`RAID`-5 Organization
image::vinum-raid5-org.png[]
Compared to mirroring, `RAID-5` has the advantage of requiring significantly less storage space. Read access is similar to that of striped organizations, but write access is significantly slower, approximately 25% of the read performance. If one drive fails, the array can continue to operate in degraded mode where a read from one of the remaining accessible drives continues normally, but a read from the failed drive is recalculated from the corresponding block from all the remaining drives.
Compared to mirroring, `RAID-5` has the advantage of requiring significantly less storage space.
Read access is similar to that of striped organizations, but write access is significantly slower, approximately 25% of the read performance.
If one drive fails, the array can continue to operate in degraded mode where a read from one of the remaining accessible drives continues normally, but a read from the failed drive is recalculated from the corresponding block from all the remaining drives.
[[vinum-objects]]
== [.filename]#vinum# Objects
@ -109,13 +150,17 @@ The following sections describe the way these objects provide the functionality
=== Volume Size Considerations
Plexes can include multiple subdisks spread over all drives in the [.filename]#vinum# configuration. As a result, the size of an individual drive does not limit the size of a plex or a volume.
Plexes can include multiple subdisks spread over all drives in the [.filename]#vinum# configuration.
As a result, the size of an individual drive does not limit the size of a plex or a volume.
=== Redundant Data Storage
[.filename]#vinum# implements mirroring by attaching multiple plexes to a volume. Each plex is a representation of the data in a volume. A volume may contain between one and eight plexes.
[.filename]#vinum# implements mirroring by attaching multiple plexes to a volume.
Each plex is a representation of the data in a volume.
A volume may contain between one and eight plexes.
Although a plex represents the complete data of a volume, it is possible for parts of the representation to be physically missing, either by design (by not defining a subdisk for parts of the plex) or by accident (as a result of the failure of a drive). As long as at least one plex can provide the data for the complete address range of the volume, the volume is fully functional.
Although a plex represents the complete data of a volume, it is possible for parts of the representation to be physically missing, either by design (by not defining a subdisk for parts of the plex) or by accident (as a result of the failure of a drive).
As long as at least one plex can provide the data for the complete address range of the volume, the volume is fully functional.
=== Which Plex Organization?
@ -152,11 +197,15 @@ Although a plex represents the complete data of a volume, it is possible for par
[[vinum-examples]]
== Some Examples
[.filename]#vinum# maintains a _configuration database_ which describes the objects known to an individual system. Initially, the user creates the configuration database from one or more configuration files using man:gvinum[8]. [.filename]#vinum# stores a copy of its configuration database on each disk _device_ under its control. This database is updated on each state change, so that a restart accurately restores the state of each [.filename]#vinum# object.
[.filename]#vinum# maintains a _configuration database_ which describes the objects known to an individual system.
Initially, the user creates the configuration database from one or more configuration files using man:gvinum[8].
[.filename]#vinum# stores a copy of its configuration database on each disk _device_ under its control.
This database is updated on each state change, so that a restart accurately restores the state of each [.filename]#vinum# object.
=== The Configuration File
The configuration file describes individual [.filename]#vinum# objects. The definition of a simple volume might be:
The configuration file describes individual [.filename]#vinum# objects.
The definition of a simple volume might be:
[.programlisting]
....
@ -194,19 +243,26 @@ Subdisks: 1 (16 configured)
S myvol.p0.s0 State: up PO: 0 B Size: 512 MB
....
This output shows the brief listing format of man:gvinum[8]. It is represented graphically in <<vinum-simple-vol, A Simple [.filename]#vinum# Volume>>.
This output shows the brief listing format of man:gvinum[8].
It is represented graphically in <<vinum-simple-vol, A Simple [.filename]#vinum# Volume>>.
[[vinum-simple-vol]]
.A Simple [.filename]#vinum# Volume
image::vinum-simple-vol.png[]
This figure, and the ones which follow, represent a volume, which contains the plexes, which in turn contains the subdisks. In this example, the volume contains one plex, and the plex contains one subdisk.
This figure, and the ones which follow, represent a volume, which contains the plexes, which in turn contains the subdisks.
In this example, the volume contains one plex, and the plex contains one subdisk.
This particular volume has no specific advantage over a conventional disk partition. It contains a single plex, so it is not redundant. The plex contains a single subdisk, so there is no difference in storage allocation from a conventional disk partition. The following sections illustrate various more interesting configuration methods.
This particular volume has no specific advantage over a conventional disk partition.
It contains a single plex, so it is not redundant.
The plex contains a single subdisk, so there is no difference in storage allocation from a conventional disk partition.
The following sections illustrate various more interesting configuration methods.
=== Increased Resilience: Mirroring
The resilience of a volume can be increased by mirroring. When laying out a mirrored volume, it is important to ensure that the subdisks of each plex are on different drives, so that a drive failure will not take down both plexes. The following configuration mirrors a volume:
The resilience of a volume can be increased by mirroring.
When laying out a mirrored volume, it is important to ensure that the subdisks of each plex are on different drives, so that a drive failure will not take down both plexes.
The following configuration mirrors a volume:
[.programlisting]
....
@ -218,7 +274,8 @@ The resilience of a volume can be increased by mirroring. When laying out a mirr
sd length 512m drive b
....
In this example, it was not necessary to specify a definition of drive _a_ again, since [.filename]#vinum# keeps track of all objects in its configuration database. After processing this definition, the configuration looks like:
In this example, it was not necessary to specify a definition of drive _a_ again, since [.filename]#vinum# keeps track of all objects in its configuration database.
After processing this definition, the configuration looks like:
[.programlisting]
....
@ -249,11 +306,14 @@ In this example, it was not necessary to specify a definition of drive _a_ again
.A Mirrored [.filename]#vinum# Volume
image::vinum-mirrored-vol.png[]
In this example, each plex contains the full 512 MB of address space. As in the previous example, each plex contains only a single subdisk.
In this example, each plex contains the full 512 MB of address space.
As in the previous example, each plex contains only a single subdisk.
=== Optimizing Performance
The mirrored volume in the previous example is more resistant to failure than an unmirrored volume, but its performance is less as each write to the volume requires a write to both drives, using up a greater proportion of the total disk bandwidth. Performance considerations demand a different approach: instead of mirroring, the data is striped across as many disk drives as possible. The following configuration shows a volume with a plex striped across four disk drives:
The mirrored volume in the previous example is more resistant to failure than an unmirrored volume, but its performance is less as each write to the volume requires a write to both drives, using up a greater proportion of the total disk bandwidth.
Performance considerations demand a different approach: instead of mirroring, the data is striped across as many disk drives as possible.
The following configuration shows a volume with a plex striped across four disk drives:
[.programlisting]
....
@ -267,7 +327,8 @@ The mirrored volume in the previous example is more resistant to failure than an
sd length 128m drive d
....
As before, it is not necessary to define the drives which are already known to [.filename]#vinum#. After processing this definition, the configuration looks like:
As before, it is not necessary to define the drives which are already known to [.filename]#vinum#.
After processing this definition, the configuration looks like:
[.programlisting]
....
@ -304,11 +365,13 @@ As before, it is not necessary to define the drives which are already known to [
.A Striped [.filename]#vinum# Volume
image::vinum-striped-vol.png[]
This volume is represented in <<vinum-striped-vol, A Striped [.filename]#vinum# Volume>>. The darkness of the stripes indicates the position within the plex address space, where the lightest stripes come first and the darkest last.
This volume is represented in <<vinum-striped-vol, A Striped [.filename]#vinum# Volume>>.
The darkness of the stripes indicates the position within the plex address space, where the lightest stripes come first and the darkest last.
=== Resilience and Performance
[[vinum-resilience]]With sufficient hardware, it is possible to build volumes which show both increased resilience and increased performance compared to standard UNIX(R) partitions. A typical configuration file might be:
[[vinum-resilience]]With sufficient hardware, it is possible to build volumes which show both increased resilience and increased performance compared to standard UNIX(R) partitions.
A typical configuration file might be:
[.programlisting]
....
@ -327,7 +390,8 @@ This volume is represented in <<vinum-striped-vol, A Striped [.filename]#vinum#
sd length 102480k drive b
....
The subdisks of the second plex are offset by two drives from those of the first plex. This helps to ensure that writes do not go to the same subdisks even if a transfer goes over two drives.
The subdisks of the second plex are offset by two drives from those of the first plex.
This helps to ensure that writes do not go to the same subdisks even if a transfer goes over two drives.
<<vinum-raid10-vol, A Mirrored, Striped [.filename]#vinum# Volume>> represents the structure of this volume.
@ -338,11 +402,14 @@ image::vinum-raid10-vol.png[]
[[vinum-object-naming]]
== Object Naming
[.filename]#vinum# assigns default names to plexes and subdisks, although they may be overridden. Overriding the default names is not recommended as it does not bring a significant advantage and it can cause confusion.
[.filename]#vinum# assigns default names to plexes and subdisks, although they may be overridden.
Overriding the default names is not recommended as it does not bring a significant advantage and it can cause confusion.
Names may contain any non-blank character, but it is recommended to restrict them to letters, digits and the underscore characters. The names of volumes, plexes, and subdisks may be up to 64 characters long, and the names of drives may be up to 32 characters long.
Names may contain any non-blank character, but it is recommended to restrict them to letters, digits and the underscore characters.
The names of volumes, plexes, and subdisks may be up to 64 characters long, and the names of drives may be up to 32 characters long.
[.filename]#vinum# objects are assigned device nodes in the hierarchy [.filename]#/dev/gvinum#. The configuration shown above would cause [.filename]#vinum# to create the following device nodes:
[.filename]#vinum# objects are assigned device nodes in the hierarchy [.filename]#/dev/gvinum#.
The configuration shown above would cause [.filename]#vinum# to create the following device nodes:
* Device entries for each volume. These are the main devices used by [.filename]#vinum#. The configuration above would include the devices [.filename]#/dev/gvinum/myvol#, [.filename]#/dev/gvinum/mirror#, [.filename]#/dev/gvinum/striped#, [.filename]#/dev/gvinum/raid5# and [.filename]#/dev/gvinum/raid10#.
* All volumes get direct entries under [.filename]#/dev/gvinum/#.
@ -385,11 +452,18 @@ After processing this file, man:gvinum[8] creates the following structure in [.f
crwxr-xr-- 1 root wheel 91, 0x20300002 Apr 13 16:46 s64.p0.s3
....
Although it is recommended that plexes and subdisks should not be allocated specific names, [.filename]#vinum# drives must be named. This makes it possible to move a drive to a different location and still recognize it automatically. Drive names may be up to 32 characters long.
Although it is recommended that plexes and subdisks should not be allocated specific names, [.filename]#vinum# drives must be named.
This makes it possible to move a drive to a different location and still recognize it automatically.
Drive names may be up to 32 characters long.
=== Creating File Systems
Volumes appear to the system to be identical to disks, with one exception. Unlike UNIX(R) drives, [.filename]#vinum# does not partition volumes, which thus do not contain a partition table. This has required modification to some disk utilities, notably man:newfs[8], so that it does not try to interpret the last letter of a [.filename]#vinum# volume name as a partition identifier. For example, a disk drive may have a name like [.filename]#/dev/ad0a# or [.filename]#/dev/da2h#. These names represent the first partition ([.filename]#a#) on the first (0) IDE disk ([.filename]#ad#) and the eighth partition ([.filename]#h#) on the third (2) SCSI disk ([.filename]#da#) respectively. By contrast, a [.filename]#vinum# volume might be called [.filename]#/dev/gvinum/concat#, which has no relationship with a partition name.
Volumes appear to the system to be identical to disks, with one exception.
Unlike UNIX(R) drives, [.filename]#vinum# does not partition volumes, which thus do not contain a partition table.
This has required modification to some disk utilities, notably man:newfs[8], so that it does not try to interpret the last letter of a [.filename]#vinum# volume name as a partition identifier.
For example, a disk drive may have a name like [.filename]#/dev/ad0a# or [.filename]#/dev/da2h#.
These names represent the first partition ([.filename]#a#) on the first (0) IDE disk ([.filename]#ad#) and the eighth partition ([.filename]#h#) on the third (2) SCSI disk ([.filename]#da#) respectively.
By contrast, a [.filename]#vinum# volume might be called [.filename]#/dev/gvinum/concat#, which has no relationship with a partition name.
In order to create a file system on this volume, use man:newfs[8]:
@ -401,11 +475,16 @@ In order to create a file system on this volume, use man:newfs[8]:
[[vinum-config]]
== Configuring [.filename]#vinum#
The [.filename]#GENERIC# kernel does not contain [.filename]#vinum#. It is possible to build a custom kernel which includes [.filename]#vinum#, but this is not recommended. The standard way to start [.filename]#vinum# is as a kernel module. man:kldload[8] is not needed because when man:gvinum[8] starts, it checks whether the module has been loaded, and if it is not, it loads it automatically.
The [.filename]#GENERIC# kernel does not contain [.filename]#vinum#.
It is possible to build a custom kernel which includes [.filename]#vinum#, but this is not recommended.
The standard way to start [.filename]#vinum# is as a kernel module.
man:kldload[8] is not needed because when man:gvinum[8] starts, it checks whether the module has been loaded, and if it is not, it loads it automatically.
=== Startup
[.filename]#vinum# stores configuration information on the disk slices in essentially the same form as in the configuration files. When reading from the configuration database, [.filename]#vinum# recognizes a number of keywords which are not allowed in the configuration files. For example, a disk configuration might contain the following text:
[.filename]#vinum# stores configuration information on the disk slices in essentially the same form as in the configuration files.
When reading from the configuration database, [.filename]#vinum# recognizes a number of keywords which are not allowed in the configuration files.
For example, a disk configuration might contain the following text:
[.programlisting]
....
@ -430,19 +509,27 @@ sd name bigraid.p0.s3 drive d plex bigraid.p0 state initializing len 4194304b dr
sd name bigraid.p0.s4 drive e plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 16777216b
....
The obvious differences here are the presence of explicit location information and naming, both of which are allowed but discouraged, and the information on the states. [.filename]#vinum# does not store information about drives in the configuration information. It finds the drives by scanning the configured disk drives for partitions with a [.filename]#vinum# label. This enables [.filename]#vinum# to identify drives correctly even if they have been assigned different UNIX(R) drive IDs.
The obvious differences here are the presence of explicit location information and naming, both of which are allowed but discouraged, and the information on the states.
[.filename]#vinum# does not store information about drives in the configuration information.
It finds the drives by scanning the configured disk drives for partitions with a [.filename]#vinum# label.
This enables [.filename]#vinum# to identify drives correctly even if they have been assigned different UNIX(R) drive IDs.
[[vinum-rc-startup]]
==== Automatic Startup
_Gvinum_ always features an automatic startup once the kernel module is loaded, via man:loader.conf[5]. To load the _Gvinum_ module at boot time, add `geom_vinum_load="YES"` to [.filename]#/boot/loader.conf#.
_Gvinum_ always features an automatic startup once the kernel module is loaded, via man:loader.conf[5].
To load the _Gvinum_ module at boot time, add `geom_vinum_load="YES"` to [.filename]#/boot/loader.conf#.
When [.filename]#vinum# is started with `gvinum start`, [.filename]#vinum# reads the configuration database from one of the [.filename]#vinum# drives. Under normal circumstances, each drive contains an identical copy of the configuration database, so it does not matter which drive is read. After a crash, however, [.filename]#vinum# must determine which drive was updated most recently and read the configuration from this drive. It then updates the configuration, if necessary, from progressively older drives.
When [.filename]#vinum# is started with `gvinum start`, [.filename]#vinum# reads the configuration database from one of the [.filename]#vinum# drives.
Under normal circumstances, each drive contains an identical copy of the configuration database, so it does not matter which drive is read.
After a crash, however, [.filename]#vinum# must determine which drive was updated most recently and read the configuration from this drive.
It then updates the configuration, if necessary, from progressively older drives.
[[vinum-root]]
== Using [.filename]#vinum# for the Root File System
For a machine that has fully-mirrored file systems using [.filename]#vinum#, it is desirable to also mirror the root file system. Setting up such a configuration is less trivial than mirroring an arbitrary file system because:
For a machine that has fully-mirrored file systems using [.filename]#vinum#, it is desirable to also mirror the root file system.
Setting up such a configuration is less trivial than mirroring an arbitrary file system because:
* The root file system must be available very early during the boot process, so the [.filename]#vinum# infrastructure must already be available at this time.
* The volume containing the root file system also contains the system bootstrap and the kernel. These must be read using the host system's native utilities, such as the BIOS, which often cannot be taught about the details of [.filename]#vinum#.
@ -451,7 +538,8 @@ In the following sections, the term "root volume" is generally used to describe
=== Starting up [.filename]#vinum# Early Enough for the Root File System
[.filename]#vinum# must be available early in the system boot as man:loader[8] must be able to load the vinum kernel module before starting the kernel. This can be accomplished by putting this line in [.filename]#/boot/loader.conf#:
[.filename]#vinum# must be available early in the system boot as man:loader[8] must be able to load the vinum kernel module before starting the kernel.
This can be accomplished by putting this line in [.filename]#/boot/loader.conf#:
[.programlisting]
....
@ -460,14 +548,20 @@ geom_vinum_load="YES"
=== Making a [.filename]#vinum#-based Root Volume Accessible to the Bootstrap
The current FreeBSD bootstrap is only 7.5 KB of code and does not understand the internal [.filename]#vinum# structures. This means that it cannot parse the [.filename]#vinum# configuration data or figure out the elements of a boot volume. Thus, some workarounds are necessary to provide the bootstrap code with the illusion of a standard `a` partition that contains the root file system.
The current FreeBSD bootstrap is only 7.5 KB of code and does not understand the internal [.filename]#vinum# structures.
This means that it cannot parse the [.filename]#vinum# configuration data or figure out the elements of a boot volume.
Thus, some workarounds are necessary to provide the bootstrap code with the illusion of a standard `a` partition that contains the root file system.
For this to be possible, the following requirements must be met for the root volume:
* The root volume must not be a stripe or `RAID`-5.
* The root volume must not contain more than one concatenated subdisk per plex.
Note that it is desirable and possible to use multiple plexes, each containing one replica of the root file system. The bootstrap process will only use one replica for finding the bootstrap and all boot files, until the kernel mounts the root file system. Each single subdisk within these plexes needs its own `a` partition illusion, for the respective device to be bootable. It is not strictly needed that each of these faked `a` partitions is located at the same offset within its device, compared with other devices containing plexes of the root volume. However, it is probably a good idea to create the [.filename]#vinum# volumes that way so the resulting mirrored devices are symmetric, to avoid confusion.
Note that it is desirable and possible to use multiple plexes, each containing one replica of the root file system.
The bootstrap process will only use one replica for finding the bootstrap and all boot files, until the kernel mounts the root file system.
Each single subdisk within these plexes needs its own `a` partition illusion, for the respective device to be bootable.
It is not strictly needed that each of these faked `a` partitions is located at the same offset within its device, compared with other devices containing plexes of the root volume.
However, it is probably a good idea to create the [.filename]#vinum# volumes that way so the resulting mirrored devices are symmetric, to avoid confusion.
In order to set up these `a` partitions for each device containing part of the root volume, the following is required:
@ -480,7 +574,8 @@ In order to set up these `a` partitions for each device containing part of the r
# gvinum l -rv root
....
+
[.filename]#vinum# offsets and sizes are measured in bytes. They must be divided by 512 in order to obtain the block numbers that are to be used by `bsdlabel`.
[.filename]#vinum# offsets and sizes are measured in bytes.
They must be divided by 512 in order to obtain the block numbers that are to be used by `bsdlabel`.
. Run this command for each device that participates in the root volume:
+
[source,shell]
@ -490,11 +585,17 @@ In order to set up these `a` partitions for each device containing part of the r
+
_devname_ must be either the name of the disk, like [.filename]#da0# for disks without a slice table, or the name of the slice, like [.filename]#ad0s1#.
+
If there is already an `a` partition on the device from a pre-[.filename]#vinum# root file system, it should be renamed to something else so that it remains accessible (just in case), but will no longer be used by default to bootstrap the system. A currently mounted root file system cannot be renamed, so this must be executed either when being booted from a "Fixit" media, or in a two-step process where, in a mirror, the disk that is not been currently booted is manipulated first.
If there is already an `a` partition on the device from a pre-[.filename]#vinum# root file system, it should be renamed to something else so that it remains accessible (just in case), but will no longer be used by default to bootstrap the system.
A currently mounted root file system cannot be renamed, so this must be executed either when being booted from a "Fixit" media, or in a two-step process where, in a mirror, the disk that is not been currently booted is manipulated first.
+
The offset of the [.filename]#vinum# partition on this device (if any) must be added to the offset of the respective root volume subdisk on this device. The resulting value will become the `offset` value for the new `a` partition. The `size` value for this partition can be taken verbatim from the calculation above. The `fstype` should be `4.2BSD`. The `fsize`, `bsize`, and `cpg` values should be chosen to match the actual file system, though they are fairly unimportant within this context.
The offset of the [.filename]#vinum# partition on this device (if any) must be added to the offset of the respective root volume subdisk on this device.
The resulting value will become the `offset` value for the new `a` partition.
The `size` value for this partition can be taken verbatim from the calculation above.
The `fstype` should be `4.2BSD`.
The `fsize`, `bsize`, and `cpg` values should be chosen to match the actual file system, though they are fairly unimportant within this context.
+
That way, a new `a` partition will be established that overlaps the [.filename]#vinum# partition on this device. `bsdlabel` will only allow for this overlap if the [.filename]#vinum# partition has properly been marked using the `vinum` fstype.
That way, a new `a` partition will be established that overlaps the [.filename]#vinum# partition on this device.
`bsdlabel` will only allow for this overlap if the [.filename]#vinum# partition has properly been marked using the `vinum` fstype.
. A faked `a` partition now exists on each device that has one replica of the root volume. It is highly recommendable to verify the result using a command like:
+
[source,shell]
@ -503,9 +604,11 @@ That way, a new `a` partition will be established that overlaps the [.filename]#
....
====
It should be remembered that all files containing control information must be relative to the root file system in the [.filename]#vinum# volume which, when setting up a new [.filename]#vinum# root volume, might not match the root file system that is currently active. So in particular, [.filename]#/etc/fstab# and [.filename]#/boot/loader.conf# need to be taken care of.
It should be remembered that all files containing control information must be relative to the root file system in the [.filename]#vinum# volume which, when setting up a new [.filename]#vinum# root volume, might not match the root file system that is currently active.
So in particular, [.filename]#/etc/fstab# and [.filename]#/boot/loader.conf# need to be taken care of.
At next reboot, the bootstrap should figure out the appropriate control information from the new [.filename]#vinum#-based root file system, and act accordingly. At the end of the kernel initialization process, after all devices have been announced, the prominent notice that shows the success of this setup is a message like:
At next reboot, the bootstrap should figure out the appropriate control information from the new [.filename]#vinum#-based root file system, and act accordingly.
At the end of the kernel initialization process, after all devices have been announced, the prominent notice that shows the success of this setup is a message like:
[source,shell]
....
@ -532,7 +635,10 @@ Subdisk root.p1.s0:
Drive disk1 (/dev/da1h) at offset 135680 (132 kB)
....
The values to note are `135680` for the offset, relative to partition [.filename]#/dev/da0h#. This translates to 265 512-byte disk blocks in `bsdlabel`'s terms. Likewise, the size of this root volume is 245760 512-byte blocks. [.filename]#/dev/da1h#, containing the second replica of this root volume, has a symmetric setup.
The values to note are `135680` for the offset, relative to partition [.filename]#/dev/da0h#.
This translates to 265 512-byte disk blocks in `bsdlabel`'s terms.
Likewise, the size of this root volume is 245760 512-byte blocks.
[.filename]#/dev/da1h#, containing the second replica of this root volume, has a symmetric setup.
The bsdlabel for these devices might look like:
@ -546,7 +652,9 @@ The bsdlabel for these devices might look like:
h: 71771672 16 vinum # (Cyl. 0*- 4467*)
....
It can be observed that the `size` parameter for the faked `a` partition matches the value outlined above, while the `offset` parameter is the sum of the offset within the [.filename]#vinum# partition `h`, and the offset of this partition within the device or slice. This is a typical setup that is necessary to avoid the problem described in <<vinum-root-panic, Nothing Boots, the Bootstrap Panics>>. The entire `a` partition is completely within the `h` partition containing all the [.filename]#vinum# data for this device.
It can be observed that the `size` parameter for the faked `a` partition matches the value outlined above, while the `offset` parameter is the sum of the offset within the [.filename]#vinum# partition `h`, and the offset of this partition within the device or slice.
This is a typical setup that is necessary to avoid the problem described in <<vinum-root-panic, Nothing Boots, the Bootstrap Panics>>.
The entire `a` partition is completely within the `h` partition containing all the [.filename]#vinum# data for this device.
In the above example, the entire device is dedicated to [.filename]#vinum# and there is no leftover pre-[.filename]#vinum# root partition.
@ -556,21 +664,34 @@ The following list contains a few known pitfalls and solutions.
==== System Bootstrap Loads, but System Does Not Boot
If for any reason the system does not continue to boot, the bootstrap can be interrupted by pressing kbd:[space] at the 10-seconds warning. The loader variable `vinum.autostart` can be examined by typing `show` and manipulated using `set` or `unset`.
If for any reason the system does not continue to boot, the bootstrap can be interrupted by pressing kbd:[space] at the 10-seconds warning.
The loader variable `vinum.autostart` can be examined by typing `show` and manipulated using `set` or `unset`.
If the [.filename]#vinum# kernel module was not yet in the list of modules to load automatically, type `load geom_vinum`.
When ready, the boot process can be continued by typing `boot -as` which `-as` requests the kernel to ask for the root file system to mount (`-a`) and make the boot process stop in single-user mode (`-s`), where the root file system is mounted read-only. That way, even if only one plex of a multi-plex volume has been mounted, no data inconsistency between plexes is being risked.
When ready, the boot process can be continued by typing `boot -as` which `-as` requests the kernel to ask for the root file system to mount (`-a`) and make the boot process stop in single-user mode (`-s`), where the root file system is mounted read-only.
That way, even if only one plex of a multi-plex volume has been mounted, no data inconsistency between plexes is being risked.
At the prompt asking for a root file system to mount, any device that contains a valid root file system can be entered. If [.filename]#/etc/fstab# is set up correctly, the default should be something like `ufs:/dev/gvinum/root`. A typical alternate choice would be something like `ufs:da0d` which could be a hypothetical partition containing the pre-[.filename]#vinum# root file system. Care should be taken if one of the alias `a` partitions is entered here, that it actually references the subdisks of the [.filename]#vinum# root device, because in a mirrored setup, this would only mount one piece of a mirrored root device. If this file system is to be mounted read-write later on, it is necessary to remove the other plex(es) of the [.filename]#vinum# root volume since these plexes would otherwise carry inconsistent data.
At the prompt asking for a root file system to mount, any device that contains a valid root file system can be entered.
If [.filename]#/etc/fstab# is set up correctly, the default should be something like `ufs:/dev/gvinum/root`.
A typical alternate choice would be something like `ufs:da0d` which could be a hypothetical partition containing the pre-[.filename]#vinum# root file system.
Care should be taken if one of the alias `a` partitions is entered here, that it actually references the subdisks of the [.filename]#vinum# root device, because in a mirrored setup, this would only mount one piece of a mirrored root device.
If this file system is to be mounted read-write later on, it is necessary to remove the other plex(es) of the [.filename]#vinum# root volume since these plexes would otherwise carry inconsistent data.
==== Only Primary Bootstrap Loads
If [.filename]#/boot/loader# fails to load, but the primary bootstrap still loads (visible by a single dash in the left column of the screen right after the boot process starts), an attempt can be made to interrupt the primary bootstrap by pressing kbd:[space]. This will make the bootstrap stop in link:{handbook}#boot-boot1[stage two]. An attempt can be made here to boot off an alternate partition, like the partition containing the previous root file system that has been moved away from `a`.
If [.filename]#/boot/loader# fails to load, but the primary bootstrap still loads (visible by a single dash in the left column of the screen right after the boot process starts), an attempt can be made to interrupt the primary bootstrap by pressing kbd:[space].
This will make the bootstrap stop in link:{handbook}#boot-boot1[stage two].
An attempt can be made here to boot off an alternate partition, like the partition containing the previous root file system that has been moved away from `a`.
[[vinum-root-panic]]
==== Nothing Boots, the Bootstrap Panics
This situation will happen if the bootstrap had been destroyed by the [.filename]#vinum# installation. Unfortunately, [.filename]#vinum# accidentally leaves only 4 KB at the beginning of its partition free before starting to write its [.filename]#vinum# header information. However, the stage one and two bootstraps plus the bsdlabel require 8 KB. So if a [.filename]#vinum# partition was started at offset 0 within a slice or disk that was meant to be bootable, the [.filename]#vinum# setup will trash the bootstrap.
This situation will happen if the bootstrap had been destroyed by the [.filename]#vinum# installation.
Unfortunately, [.filename]#vinum# accidentally leaves only 4 KB at the beginning of its partition free before starting to write its [.filename]#vinum# header information.
However, the stage one and two bootstraps plus the bsdlabel require 8 KB.
So if a [.filename]#vinum# partition was started at offset 0 within a slice or disk that was meant to be bootable, the [.filename]#vinum# setup will trash the bootstrap.
Similarly, if the above situation has been recovered, by booting from a "Fixit" media, and the bootstrap has been re-installed using `bsdlabel -B` as described in link:{handbook}#boot-boot1[stage two], the bootstrap will trash the [.filename]#vinum# header, and [.filename]#vinum# will no longer find its disk(s). Though no actual [.filename]#vinum# configuration data or data in [.filename]#vinum# volumes will be trashed, and it would be possible to recover all the data by entering exactly the same [.filename]#vinum# configuration data again, the situation is hard to fix. It is necessary to move the entire [.filename]#vinum# partition by at least 4 KB, in order to have the [.filename]#vinum# header and the system bootstrap no longer collide.
Similarly, if the above situation has been recovered, by booting from a "Fixit" media, and the bootstrap has been re-installed using `bsdlabel -B` as described in link:{handbook}#boot-boot1[stage two], the bootstrap will trash the [.filename]#vinum# header, and [.filename]#vinum# will no longer find its disk(s).
Though no actual [.filename]#vinum# configuration data or data in [.filename]#vinum# volumes will be trashed, and it would be possible to recover all the data by entering exactly the same [.filename]#vinum# configuration data again, the situation is hard to fix.
It is necessary to move the entire [.filename]#vinum# partition by at least 4 KB, in order to have the [.filename]#vinum# header and the system bootstrap no longer collide.

304
documentation/content/en/articles/vm-design/_index.adoc
View file

@ -18,7 +18,7 @@ trademarks: ["freebsd", "linux", "microsoft", "opengroup", "daemon-news", "gener
:experimental:
ifeval::["{backend}" == "html5"]
:imagesdir: ../../images/articles/vm-design/
:imagesdir: ../../../images/articles/vm-design/
endif::[]
ifeval::["{backend}" == "pdf"]
@ -32,7 +32,12 @@ endif::[]
[.abstract-title]
Abstract
The title is really just a fancy way of saying that I am going to attempt to describe the whole VM enchilada, hopefully in a way that everyone can follow. For the last year I have concentrated on a number of major kernel subsystems within FreeBSD, with the VM and Swap subsystems being the most interesting and NFS being "a necessary chore". I rewrote only small portions of the code. In the VM arena the only major rewrite I have done is to the swap subsystem. Most of my work was cleanup and maintenance, with only moderate code rewriting and no major algorithmic adjustments within the VM subsystem. The bulk of the VM subsystem's theoretical base remains unchanged and a lot of the credit for the modernization effort in the last few years belongs to John Dyson and David Greenman. Not being a historian like Kirk I will not attempt to tag all the various features with peoples names, since I will invariably get it wrong.
The title is really just a fancy way of saying that I am going to attempt to describe the whole VM enchilada, hopefully in a way that everyone can follow.
For the last year I have concentrated on a number of major kernel subsystems within FreeBSD, with the VM and Swap subsystems being the most interesting and NFS being "a necessary chore".
I rewrote only small portions of the code. In the VM arena the only major rewrite I have done is to the swap subsystem.
Most of my work was cleanup and maintenance, with only moderate code rewriting and no major algorithmic adjustments within the VM subsystem.
The bulk of the VM subsystem's theoretical base remains unchanged and a lot of the credit for the modernization effort in the last few years belongs to John Dyson and David Greenman.
Not being a historian like Kirk I will not attempt to tag all the various features with peoples names, since I will invariably get it wrong.
'''
@ -41,54 +46,133 @@ toc::[]
[[introduction]]
== Introduction
Before moving along to the actual design let's spend a little time on the necessity of maintaining and modernizing any long-living codebase. In the programming world, algorithms tend to be more important than code and it is precisely due to BSD's academic roots that a great deal of attention was paid to algorithm design from the beginning. More attention paid to the design generally leads to a clean and flexible codebase that can be fairly easily modified, extended, or replaced over time. While BSD is considered an "old" operating system by some people, those of us who work on it tend to view it more as a "mature" codebase which has various components modified, extended, or replaced with modern code. It has evolved, and FreeBSD is at the bleeding edge no matter how old some of the code might be. This is an important distinction to make and one that is unfortunately lost to many people. The biggest error a programmer can make is to not learn from history, and this is precisely the error that many other modern operating systems have made. Windows NT(R) is the best example of this, and the consequences have been dire. Linux also makes this mistake to some degree-enough that we BSD folk can make small jokes about it every once in a while, anyway. Linux's problem is simply one of a lack of experience and history to compare ideas against, a problem that is easily and rapidly being addressed by the Linux community in the same way it has been addressed in the BSD community-by continuous code development. The Windows NT(R) folk, on the other hand, repeatedly make the same mistakes solved by UNIX(R) decades ago and then spend years fixing them. Over and over again. They have a severe case of "not designed here" and "we are always right because our marketing department says so". I have little tolerance for anyone who cannot learn from history.
Before moving along to the actual design let's spend a little time on the necessity of maintaining and modernizing any long-living codebase.
In the programming world, algorithms tend to be more important than code and it is precisely due to BSD's academic roots that a great deal of attention was paid to algorithm design from the beginning.
More attention paid to the design generally leads to a clean and flexible codebase that can be fairly easily modified, extended, or replaced over time.
While BSD is considered an "old" operating system by some people, those of us who work on it tend to view it more as a "mature" codebase which has various components modified, extended, or replaced with modern code.
It has evolved, and FreeBSD is at the bleeding edge no matter how old some of the code might be.
This is an important distinction to make and one that is unfortunately lost to many people.
The biggest error a programmer can make is to not learn from history, and this is precisely the error that many other modern operating systems have made.
Windows NT(R) is the best example of this, and the consequences have been dire.
Linux also makes this mistake to some degree-enough that we BSD folk can make small jokes about it every once in a while, anyway.
Linux's problem is simply one of a lack of experience and history to compare ideas against, a problem that is easily and rapidly being addressed by the Linux community in the same way it has been addressed in the BSD community-by continuous code development.
The Windows NT(R) folk, on the other hand, repeatedly make the same mistakes solved by UNIX(R) decades ago and then spend years fixing them.
Over and over again.
They have a severe case of "not designed here" and "we are always right because our marketing department says so".
I have little tolerance for anyone who cannot learn from history.
Much of the apparent complexity of the FreeBSD design, especially in the VM/Swap subsystem, is a direct result of having to solve serious performance issues that occur under various conditions. These issues are not due to bad algorithmic design but instead rise from environmental factors. In any direct comparison between platforms, these issues become most apparent when system resources begin to get stressed. As I describe FreeBSD's VM/Swap subsystem the reader should always keep two points in mind:
Much of the apparent complexity of the FreeBSD design, especially in the VM/Swap subsystem, is a direct result of having to solve serious performance issues that occur under various conditions.
These issues are not due to bad algorithmic design but instead rise from environmental factors.
In any direct comparison between platforms, these issues become most apparent when system resources begin to get stressed.
As I describe FreeBSD's VM/Swap subsystem the reader should always keep two points in mind:
. The most important aspect of performance design is what is known as "Optimizing the Critical Path". It is often the case that performance optimizations add a little bloat to the code in order to make the critical path perform better.
. A solid, generalized design outperforms a heavily-optimized design over the long run. While a generalized design may end up being slower than an heavily-optimized design when they are first implemented, the generalized design tends to be easier to adapt to changing conditions and the heavily-optimized design winds up having to be thrown away.
Any codebase that will survive and be maintainable for years must therefore be designed properly from the beginning even if it costs some performance. Twenty years ago people were still arguing that programming in assembly was better than programming in a high-level language because it produced code that was ten times as fast. Today, the fallibility of that argument is obvious - as are the parallels to algorithmic design and code generalization.
Any codebase that will survive and be maintainable for years must therefore be designed properly from the beginning even if it costs some performance.
Twenty years ago people were still arguing that programming in assembly was better than programming in a high-level language because it produced code that was ten times as fast.
Today, the fallibility of that argument is obvious - as are the parallels to algorithmic design and code generalization.
[[vm-objects]]
== VM Objects
The best way to begin describing the FreeBSD VM system is to look at it from the perspective of a user-level process. Each user process sees a single, private, contiguous VM address space containing several types of memory objects. These objects have various characteristics. Program code and program data are effectively a single memory-mapped file (the binary file being run), but program code is read-only while program data is copy-on-write. Program BSS is just memory allocated and filled with zeros on demand, called demand zero page fill. Arbitrary files can be memory-mapped into the address space as well, which is how the shared library mechanism works. Such mappings can require modifications to remain private to the process making them. The fork system call adds an entirely new dimension to the VM management problem on top of the complexity already given.
The best way to begin describing the FreeBSD VM system is to look at it from the perspective of a user-level process.
Each user process sees a single, private, contiguous VM address space containing several types of memory objects.
These objects have various characteristics.
Program code and program data are effectively a single memory-mapped file (the binary file being run), but program code is read-only while program data is copy-on-write.
Program BSS is just memory allocated and filled with zeros on demand, called demand zero page fill.
Arbitrary files can be memory-mapped into the address space as well, which is how the shared library mechanism works.
Such mappings can require modifications to remain private to the process making them.
The fork system call adds an entirely new dimension to the VM management problem on top of the complexity already given.
A program binary data page (which is a basic copy-on-write page) illustrates the complexity. A program binary contains a preinitialized data section which is initially mapped directly from the program file. When a program is loaded into a process's VM space, this area is initially memory-mapped and backed by the program binary itself, allowing the VM system to free/reuse the page and later load it back in from the binary. The moment a process modifies this data, however, the VM system must make a private copy of the page for that process. Since the private copy has been modified, the VM system may no longer free it, because there is no longer any way to restore it later on.
A program binary data page (which is a basic copy-on-write page) illustrates the complexity.
A program binary contains a preinitialized data section which is initially mapped directly from the program file.
When a program is loaded into a process's VM space, this area is initially memory-mapped and backed by the program binary itself, allowing the VM system to free/reuse the page and later load it back in from the binary.
The moment a process modifies this data, however, the VM system must make a private copy of the page for that process.
Since the private copy has been modified, the VM system may no longer free it, because there is no longer any way to restore it later on.
You will notice immediately that what was originally a simple file mapping has become much more complex. Data may be modified on a page-by-page basis whereas the file mapping encompasses many pages at once. The complexity further increases when a process forks. When a process forks, the result is two processes-each with their own private address spaces, including any modifications made by the original process prior to the call to `fork()`. It would be silly for the VM system to make a complete copy of the data at the time of the `fork()` because it is quite possible that at least one of the two processes will only need to read from that page from then on, allowing the original page to continue to be used. What was a private page is made copy-on-write again, since each process (parent and child) expects their own personal post-fork modifications to remain private to themselves and not effect the other.
You will notice immediately that what was originally a simple file mapping has become much more complex.
Data may be modified on a page-by-page basis whereas the file mapping encompasses many pages at once.
The complexity further increases when a process forks.
When a process forks, the result is two processes-each with their own private address spaces, including any modifications made by the original process prior to the call to `fork()`.
It would be silly for the VM system to make a complete copy of the data at the time of the `fork()` because it is quite possible that at least one of the two processes will only need to read from that page from then on, allowing the original page to continue to be used.
What was a private page is made copy-on-write again, since each process (parent and child) expects their own personal post-fork modifications to remain private to themselves and not effect the other.
FreeBSD manages all of this with a layered VM Object model. The original binary program file winds up being the lowest VM Object layer. A copy-on-write layer is pushed on top of that to hold those pages which had to be copied from the original file. If the program modifies a data page belonging to the original file the VM system takes a fault and makes a copy of the page in the higher layer. When a process forks, additional VM Object layers are pushed on. This might make a little more sense with a fairly basic example. A `fork()` is a common operation for any *BSD system, so this example will consider a program that starts up, and forks. When the process starts, the VM system creates an object layer, let's call this A:
FreeBSD manages all of this with a layered VM Object model.
The original binary program file winds up being the lowest VM Object layer.
A copy-on-write layer is pushed on top of that to hold those pages which had to be copied from the original file.
If the program modifies a data page belonging to the original file the VM system takes a fault and makes a copy of the page in the higher layer.
When a process forks, additional VM Object layers are pushed on.
This might make a little more sense with a fairly basic example.
A `fork()` is a common operation for any *BSD system, so this example will consider a program that starts up, and forks.
When the process starts, the VM system creates an object layer, let's call this A:
image::fig1.png[A picture]
A represents the file-pages may be paged in and out of the file's physical media as necessary. Paging in from the disk is reasonable for a program, but we really do not want to page back out and overwrite the executable. The VM system therefore creates a second layer, B, that will be physically backed by swap space:
A represents the file-pages may be paged in and out of the file's physical media as necessary.
Paging in from the disk is reasonable for a program, but we really do not want to page back out and overwrite the executable.
The VM system therefore creates a second layer, B, that will be physically backed by swap space:
image::fig2.png[]
On the first write to a page after this, a new page is created in B, and its contents are initialized from A. All pages in B can be paged in or out to a swap device. When the program forks, the VM system creates two new object layers-C1 for the parent, and C2 for the child-that rest on top of B:
On the first write to a page after this, a new page is created in B, and its contents are initialized from A.
All pages in B can be paged in or out to a swap device.
When the program forks, the VM system creates two new object layers-C1 for the parent, and C2 for the child-that rest on top of B:
image::fig3.png[]
In this case, let's say a page in B is modified by the original parent process. The process will take a copy-on-write fault and duplicate the page in C1, leaving the original page in B untouched. Now, let's say the same page in B is modified by the child process. The process will take a copy-on-write fault and duplicate the page in C2. The original page in B is now completely hidden since both C1 and C2 have a copy and B could theoretically be destroyed if it does not represent a "real" file; however, this sort of optimization is not trivial to make because it is so fine-grained. FreeBSD does not make this optimization. Now, suppose (as is often the case) that the child process does an `exec()`. Its current address space is usually replaced by a new address space representing a new file. In this case, the C2 layer is destroyed:
In this case, let's say a page in B is modified by the original parent process.
The process will take a copy-on-write fault and duplicate the page in C1, leaving the original page in B untouched.
Now, let's say the same page in B is modified by the child process.
The process will take a copy-on-write fault and duplicate the page in C2.
The original page in B is now completely hidden since both C1 and C2 have a copy and B could theoretically be destroyed if it does not represent a "real" file; however, this sort of optimization is not trivial to make because it is so fine-grained.
FreeBSD does not make this optimization.
Now, suppose (as is often the case) that the child process does an `exec()`.
Its current address space is usually replaced by a new address space representing a new file.
In this case, the C2 layer is destroyed:
image::fig4.png[]
In this case, the number of children of B drops to one, and all accesses to B now go through C1. This means that B and C1 can be collapsed together. Any pages in B that also exist in C1 are deleted from B during the collapse. Thus, even though the optimization in the previous step could not be made, we can recover the dead pages when either of the processes exit or `exec()`.
In this case, the number of children of B drops to one, and all accesses to B now go through C1.
This means that B and C1 can be collapsed together.
Any pages in B that also exist in C1 are deleted from B during the collapse.
Thus, even though the optimization in the previous step could not be made, we can recover the dead pages when either of the processes exit or `exec()`.
This model creates a number of potential problems. The first is that you can wind up with a relatively deep stack of layered VM Objects which can cost scanning time and memory when you take a fault. Deep layering can occur when processes fork and then fork again (either parent or child). The second problem is that you can wind up with dead, inaccessible pages deep in the stack of VM Objects. In our last example if both the parent and child processes modify the same page, they both get their own private copies of the page and the original page in B is no longer accessible by anyone. That page in B can be freed.
This model creates a number of potential problems.
The first is that you can wind up with a relatively deep stack of layered VM Objects which can cost scanning time and memory when you take a fault.
Deep layering can occur when processes fork and then fork again (either parent or child).
The second problem is that you can wind up with dead, inaccessible pages deep in the stack of VM Objects.
In our last example if both the parent and child processes modify the same page, they both get their own private copies of the page and the original page in B is no longer accessible by anyone.
That page in B can be freed.
FreeBSD solves the deep layering problem with a special optimization called the "All Shadowed Case". This case occurs if either C1 or C2 take sufficient COW faults to completely shadow all pages in B. Lets say that C1 achieves this. C1 can now bypass B entirely, so rather then have C1->B->A and C2->B->A we now have C1->A and C2->B->A. But look what also happened-now B has only one reference (C2), so we can collapse B and C2 together. The end result is that B is deleted entirely and we have C1->A and C2->A. It is often the case that B will contain a large number of pages and neither C1 nor C2 will be able to completely overshadow it. If we fork again and create a set of D layers, however, it is much more likely that one of the D layers will eventually be able to completely overshadow the much smaller dataset represented by C1 or C2. The same optimization will work at any point in the graph and the grand result of this is that even on a heavily forked machine VM Object stacks tend to not get much deeper then 4. This is true of both the parent and the children and true whether the parent is doing the forking or whether the children cascade forks.
FreeBSD solves the deep layering problem with a special optimization called the "All Shadowed Case".
This case occurs if either C1 or C2 take sufficient COW faults to completely shadow all pages in B.
Lets say that C1 achieves this.
C1 can now bypass B entirely, so rather then have C1->B->A and C2->B->A we now have C1->A and C2->B->A.
But look what also happened-now B has only one reference (C2), so we can collapse B and C2 together.
The end result is that B is deleted entirely and we have C1->A and C2->A.
It is often the case that B will contain a large number of pages and neither C1 nor C2 will be able to completely overshadow it.
If we fork again and create a set of D layers, however, it is much more likely that one of the D layers will eventually be able to completely overshadow the much smaller dataset represented by C1 or C2.
The same optimization will work at any point in the graph and the grand result of this is that even on a heavily forked machine VM Object stacks tend to not get much deeper then 4.
This is true of both the parent and the children and true whether the parent is doing the forking or whether the children cascade forks.
The dead page problem still exists in the case where C1 or C2 do not completely overshadow B. Due to our other optimizations this case does not represent much of a problem and we simply allow the pages to be dead. If the system runs low on memory it will swap them out, eating a little swap, but that is it.
The dead page problem still exists in the case where C1 or C2 do not completely overshadow B.
Due to our other optimizations this case does not represent much of a problem and we simply allow the pages to be dead.
If the system runs low on memory it will swap them out, eating a little swap, but that is it.
The advantage to the VM Object model is that `fork()` is extremely fast, since no real data copying need take place. The disadvantage is that you can build a relatively complex VM Object layering that slows page fault handling down a little, and you spend memory managing the VM Object structures. The optimizations FreeBSD makes proves to reduce the problems enough that they can be ignored, leaving no real disadvantage.
The advantage to the VM Object model is that `fork()` is extremely fast, since no real data copying need take place.
The disadvantage is that you can build a relatively complex VM Object layering that slows page fault handling down a little, and you spend memory managing the VM Object structures.
The optimizations FreeBSD makes proves to reduce the problems enough that they can be ignored, leaving no real disadvantage.
[[swap-layers]]
== SWAP Layers
Private data pages are initially either copy-on-write or zero-fill pages. When a change, and therefore a copy, is made, the original backing object (usually a file) can no longer be used to save a copy of the page when the VM system needs to reuse it for other purposes. This is where SWAP comes in. SWAP is allocated to create backing store for memory that does not otherwise have it. FreeBSD allocates the swap management structure for a VM Object only when it is actually needed. However, the swap management structure has had problems historically:
Private data pages are initially either copy-on-write or zero-fill pages.
When a change, and therefore a copy, is made, the original backing object (usually a file) can no longer be used to save a copy of the page when the VM system needs to reuse it for other purposes.
This is where SWAP comes in.
SWAP is allocated to create backing store for memory that does not otherwise have it.
FreeBSD allocates the swap management structure for a VM Object only when it is actually needed.
However, the swap management structure has had problems historically:
* Under FreeBSD 3.X the swap management structure preallocates an array that encompasses the entire object requiring swap backing store-even if only a few pages of that object are swap-backed. This creates a kernel memory fragmentation problem when large objects are mapped, or processes with large runsizes (RSS) fork.
* Also, in order to keep track of swap space, a "list of holes" is kept in kernel memory, and this tends to get severely fragmented as well. Since the "list of holes" is a linear list, the swap allocation and freeing performance is a non-optimal O(n)-per-page.
@ -97,7 +181,8 @@ Private data pages are initially either copy-on-write or zero-fill pages. When a
* Also, the swap block map can become fragmented fairly easily resulting in non-contiguous allocations.
* Kernel memory must also be allocated on the fly for additional swap management structures when a swapout occurs.
It is evident from that list that there was plenty of room for improvement. For FreeBSD 4.X, I completely rewrote the swap subsystem:
It is evident from that list that there was plenty of room for improvement.
For FreeBSD 4.X, I completely rewrote the swap subsystem:
* Swap management structures are allocated through a hash table rather than a linear array giving them a fixed allocation size and much finer granularity.
* Rather then using a linearly linked list to keep track of swap space reservations, it now uses a bitmap of swap blocks arranged in a radix tree structure with free-space hinting in the radix node structures. This effectively makes swap allocation and freeing an O(1) operation.
@ -109,107 +194,218 @@ I did not take the final step of having an "allocating hint pointer" that would
[[freeing-pages]]
== When to free a page
Since the VM system uses all available memory for disk caching, there are usually very few truly-free pages. The VM system depends on being able to properly choose pages which are not in use to reuse for new allocations. Selecting the optimal pages to free is possibly the single-most important function any VM system can perform because if it makes a poor selection, the VM system may be forced to unnecessarily retrieve pages from disk, seriously degrading system performance.
Since the VM system uses all available memory for disk caching, there are usually very few truly-free pages.
The VM system depends on being able to properly choose pages which are not in use to reuse for new allocations.
Selecting the optimal pages to free is possibly the single-most important function any VM system can perform because if it makes a poor selection, the VM system may be forced to unnecessarily retrieve pages from disk, seriously degrading system performance.
How much overhead are we willing to suffer in the critical path to avoid freeing the wrong page? Each wrong choice we make will cost us hundreds of thousands of CPU cycles and a noticeable stall of the affected processes, so we are willing to endure a significant amount of overhead in order to be sure that the right page is chosen. This is why FreeBSD tends to outperform other systems when memory resources become stressed.
How much overhead are we willing to suffer in the critical path to avoid freeing the wrong page? Each wrong choice we make will cost us hundreds of thousands of CPU cycles and a noticeable stall of the affected processes, so we are willing to endure a significant amount of overhead in order to be sure that the right page is chosen.
This is why FreeBSD tends to outperform other systems when memory resources become stressed.
The free page determination algorithm is built upon a history of the use of memory pages. To acquire this history, the system takes advantage of a page-used bit feature that most hardware page tables have.
The free page determination algorithm is built upon a history of the use of memory pages.
To acquire this history, the system takes advantage of a page-used bit feature that most hardware page tables have.
In any case, the page-used bit is cleared and at some later point the VM system comes across the page again and sees that the page-used bit has been set. This indicates that the page is still being actively used. If the bit is still clear it is an indication that the page is not being actively used. By testing this bit periodically, a use history (in the form of a counter) for the physical page is developed. When the VM system later needs to free up some pages, checking this history becomes the cornerstone of determining the best candidate page to reuse.
In any case, the page-used bit is cleared and at some later point the VM system comes across the page again and sees that the page-used bit has been set.
This indicates that the page is still being actively used.
If the bit is still clear it is an indication that the page is not being actively used.
By testing this bit periodically, a use history (in the form of a counter) for the physical page is developed.
When the VM system later needs to free up some pages, checking this history becomes the cornerstone of determining the best candidate page to reuse.
For those platforms that do not have this feature, the system actually emulates a page-used bit. It unmaps or protects a page, forcing a page fault if the page is accessed again. When the page fault is taken, the system simply marks the page as having been used and unprotects the page so that it may be used. While taking such page faults just to determine if a page is being used appears to be an expensive proposition, it is much less expensive than reusing the page for some other purpose only to find that a process needs it back and then have to go to disk.
For those platforms that do not have this feature, the system actually emulates a page-used bit.
It unmaps or protects a page, forcing a page fault if the page is accessed again.
When the page fault is taken, the system simply marks the page as having been used and unprotects the page so that it may be used.
While taking such page faults just to determine if a page is being used appears to be an expensive proposition, it is much less expensive than reusing the page for some other purpose only to find that a process needs it back and then have to go to disk.
FreeBSD makes use of several page queues to further refine the selection of pages to reuse as well as to determine when dirty pages must be flushed to their backing store. Since page tables are dynamic entities under FreeBSD, it costs virtually nothing to unmap a page from the address space of any processes using it. When a page candidate has been chosen based on the page-use counter, this is precisely what is done. The system must make a distinction between clean pages which can theoretically be freed up at any time, and dirty pages which must first be written to their backing store before being reusable. When a page candidate has been found it is moved to the inactive queue if it is dirty, or the cache queue if it is clean. A separate algorithm based on the dirty-to-clean page ratio determines when dirty pages in the inactive queue must be flushed to disk. Once this is accomplished, the flushed pages are moved from the inactive queue to the cache queue. At this point, pages in the cache queue can still be reactivated by a VM fault at relatively low cost. However, pages in the cache queue are considered to be "immediately freeable" and will be reused in an LRU (least-recently used) fashion when the system needs to allocate new memory.
FreeBSD makes use of several page queues to further refine the selection of pages to reuse as well as to determine when dirty pages must be flushed to their backing store.
Since page tables are dynamic entities under FreeBSD, it costs virtually nothing to unmap a page from the address space of any processes using it.
When a page candidate has been chosen based on the page-use counter, this is precisely what is done.
The system must make a distinction between clean pages which can theoretically be freed up at any time, and dirty pages which must first be written to their backing store before being reusable.
When a page candidate has been found it is moved to the inactive queue if it is dirty, or the cache queue if it is clean.
A separate algorithm based on the dirty-to-clean page ratio determines when dirty pages in the inactive queue must be flushed to disk.
Once this is accomplished, the flushed pages are moved from the inactive queue to the cache queue.
At this point, pages in the cache queue can still be reactivated by a VM fault at relatively low cost.
However, pages in the cache queue are considered to be "immediately freeable" and will be reused in an LRU (least-recently used) fashion when the system needs to allocate new memory.
It is important to note that the FreeBSD VM system attempts to separate clean and dirty pages for the express reason of avoiding unnecessary flushes of dirty pages (which eats I/O bandwidth), nor does it move pages between the various page queues gratuitously when the memory subsystem is not being stressed. This is why you will see some systems with very low cache queue counts and high active queue counts when doing a `systat -vm` command. As the VM system becomes more stressed, it makes a greater effort to maintain the various page queues at the levels determined to be the most effective.
It is important to note that the FreeBSD VM system attempts to separate clean and dirty pages for the express reason of avoiding unnecessary flushes of dirty pages (which eats I/O bandwidth), nor does it move pages between the various page queues gratuitously when the memory subsystem is not being stressed.
This is why you will see some systems with very low cache queue counts and high active queue counts when doing a `systat -vm` command.
As the VM system becomes more stressed, it makes a greater effort to maintain the various page queues at the levels determined to be the most effective.
An urban myth has circulated for years that Linux did a better job avoiding swapouts than FreeBSD, but this in fact is not true. What was actually occurring was that FreeBSD was proactively paging out unused pages in order to make room for more disk cache while Linux was keeping unused pages in core and leaving less memory available for cache and process pages. I do not know whether this is still true today.
An urban myth has circulated for years that Linux did a better job avoiding swapouts than FreeBSD, but this in fact is not true.
What was actually occurring was that FreeBSD was proactively paging out unused pages in order to make room for more disk cache while Linux was keeping unused pages in core and leaving less memory available for cache and process pages.
I do not know whether this is still true today.
[[prefault-optimizations]]
== Pre-Faulting and Zeroing Optimizations
Taking a VM fault is not expensive if the underlying page is already in core and can simply be mapped into the process, but it can become expensive if you take a whole lot of them on a regular basis. A good example of this is running a program such as man:ls[1] or man:ps[1] over and over again. If the program binary is mapped into memory but not mapped into the page table, then all the pages that will be accessed by the program will have to be faulted in every time the program is run. This is unnecessary when the pages in question are already in the VM Cache, so FreeBSD will attempt to pre-populate a process's page tables with those pages that are already in the VM Cache. One thing that FreeBSD does not yet do is pre-copy-on-write certain pages on exec. For example, if you run the man:ls[1] program while running `vmstat 1` you will notice that it always takes a certain number of page faults, even when you run it over and over again. These are zero-fill faults, not program code faults (which were pre-faulted in already). Pre-copying pages on exec or fork is an area that could use more study.
Taking a VM fault is not expensive if the underlying page is already in core and can simply be mapped into the process, but it can become expensive if you take a whole lot of them on a regular basis.
A good example of this is running a program such as man:ls[1] or man:ps[1] over and over again.
If the program binary is mapped into memory but not mapped into the page table, then all the pages that will be accessed by the program will have to be faulted in every time the program is run.
This is unnecessary when the pages in question are already in the VM Cache, so FreeBSD will attempt to pre-populate a process's page tables with those pages that are already in the VM Cache.
One thing that FreeBSD does not yet do is pre-copy-on-write certain pages on exec.
For example, if you run the man:ls[1] program while running `vmstat 1` you will notice that it always takes a certain number of page faults, even when you run it over and over again.
These are zero-fill faults, not program code faults (which were pre-faulted in already).
Pre-copying pages on exec or fork is an area that could use more study.
A large percentage of page faults that occur are zero-fill faults. You can usually see this by observing the `vmstat -s` output. These occur when a process accesses pages in its BSS area. The BSS area is expected to be initially zero but the VM system does not bother to allocate any memory at all until the process actually accesses it. When a fault occurs the VM system must not only allocate a new page, it must zero it as well. To optimize the zeroing operation the VM system has the ability to pre-zero pages and mark them as such, and to request pre-zeroed pages when zero-fill faults occur. The pre-zeroing occurs whenever the CPU is idle but the number of pages the system pre-zeros is limited in order to avoid blowing away the memory caches. This is an excellent example of adding complexity to the VM system in order to optimize the critical path.
A large percentage of page faults that occur are zero-fill faults.
You can usually see this by observing the `vmstat -s` output.
These occur when a process accesses pages in its BSS area.
The BSS area is expected to be initially zero but the VM system does not bother to allocate any memory at all until the process actually accesses it.
When a fault occurs the VM system must not only allocate a new page, it must zero it as well.
To optimize the zeroing operation the VM system has the ability to pre-zero pages and mark them as such, and to request pre-zeroed pages when zero-fill faults occur.
The pre-zeroing occurs whenever the CPU is idle but the number of pages the system pre-zeros is limited in order to avoid blowing away the memory caches.
This is an excellent example of adding complexity to the VM system in order to optimize the critical path.
[[page-table-optimizations]]
== Page Table Optimizations
The page table optimizations make up the most contentious part of the FreeBSD VM design and they have shown some strain with the advent of serious use of `mmap()`. I think this is actually a feature of most BSDs though I am not sure when it was first introduced. There are two major optimizations. The first is that hardware page tables do not contain persistent state but instead can be thrown away at any time with only a minor amount of management overhead. The second is that every active page table entry in the system has a governing `pv_entry` structure which is tied into the `vm_page` structure. FreeBSD can simply iterate through those mappings that are known to exist while Linux must check all page tables that _might_ contain a specific mapping to see if it does, which can achieve O(n^2) overhead in certain situations. It is because of this that FreeBSD tends to make better choices on which pages to reuse or swap when memory is stressed, giving it better performance under load. However, FreeBSD requires kernel tuning to accommodate large-shared-address-space situations such as those that can occur in a news system because it may run out of `pv_entry` structures.
The page table optimizations make up the most contentious part of the FreeBSD VM design and they have shown some strain with the advent of serious use of `mmap()`.
I think this is actually a feature of most BSDs though I am not sure when it was first introduced.
There are two major optimizations.
The first is that hardware page tables do not contain persistent state but instead can be thrown away at any time with only a minor amount of management overhead.
The second is that every active page table entry in the system has a governing `pv_entry` structure which is tied into the `vm_page` structure.
FreeBSD can simply iterate through those mappings that are known to exist while Linux must check all page tables that _might_ contain a specific mapping to see if it does, which can achieve O(n^2) overhead in certain situations.
It is because of this that FreeBSD tends to make better choices on which pages to reuse or swap when memory is stressed, giving it better performance under load.
However, FreeBSD requires kernel tuning to accommodate large-shared-address-space situations such as those that can occur in a news system because it may run out of `pv_entry` structures.
Both Linux and FreeBSD need work in this area. FreeBSD is trying to maximize the advantage of a potentially sparse active-mapping model (not all processes need to map all pages of a shared library, for example), whereas Linux is trying to simplify its algorithms. FreeBSD generally has the performance advantage here at the cost of wasting a little extra memory, but FreeBSD breaks down in the case where a large file is massively shared across hundreds of processes. Linux, on the other hand, breaks down in the case where many processes are sparsely-mapping the same shared library and also runs non-optimally when trying to determine whether a page can be reused or not.
Both Linux and FreeBSD need work in this area.
FreeBSD is trying to maximize the advantage of a potentially sparse active-mapping model (not all processes need to map all pages of a shared library, for example), whereas Linux is trying to simplify its algorithms.
FreeBSD generally has the performance advantage here at the cost of wasting a little extra memory, but FreeBSD breaks down in the case where a large file is massively shared across hundreds of processes.
Linux, on the other hand, breaks down in the case where many processes are sparsely-mapping the same shared library and also runs non-optimally when trying to determine whether a page can be reused or not.
[[page-coloring-optimizations]]
== Page Coloring
We will end with the page coloring optimizations. Page coloring is a performance optimization designed to ensure that accesses to contiguous pages in virtual memory make the best use of the processor cache. In ancient times (i.e. 10+ years ago) processor caches tended to map virtual memory rather than physical memory. This led to a huge number of problems including having to clear the cache on every context switch in some cases, and problems with data aliasing in the cache. Modern processor caches map physical memory precisely to solve those problems. This means that two side-by-side pages in a processes address space may not correspond to two side-by-side pages in the cache. In fact, if you are not careful side-by-side pages in virtual memory could wind up using the same page in the processor cache-leading to cacheable data being thrown away prematurely and reducing CPU performance. This is true even with multi-way set-associative caches (though the effect is mitigated somewhat).
We will end with the page coloring optimizations.
Page coloring is a performance optimization designed to ensure that accesses to contiguous pages in virtual memory make the best use of the processor cache.
In ancient times (i.e. 10+ years ago) processor caches tended to map virtual memory rather than physical memory.
This led to a huge number of problems including having to clear the cache on every context switch in some cases, and problems with data aliasing in the cache.
Modern processor caches map physical memory precisely to solve those problems.
This means that two side-by-side pages in a processes address space may not correspond to two side-by-side pages in the cache.
In fact, if you are not careful side-by-side pages in virtual memory could wind up using the same page in the processor cache-leading to cacheable data being thrown away prematurely and reducing CPU performance.
This is true even with multi-way set-associative caches (though the effect is mitigated somewhat).
FreeBSD's memory allocation code implements page coloring optimizations, which means that the memory allocation code will attempt to locate free pages that are contiguous from the point of view of the cache. For example, if page 16 of physical memory is assigned to page 0 of a process's virtual memory and the cache can hold 4 pages, the page coloring code will not assign page 20 of physical memory to page 1 of a process's virtual memory. It would, instead, assign page 21 of physical memory. The page coloring code attempts to avoid assigning page 20 because this maps over the same cache memory as page 16 and would result in non-optimal caching. This code adds a significant amount of complexity to the VM memory allocation subsystem as you can well imagine, but the result is well worth the effort. Page Coloring makes VM memory as deterministic as physical memory in regards to cache performance.
FreeBSD's memory allocation code implements page coloring optimizations, which means that the memory allocation code will attempt to locate free pages that are contiguous from the point of view of the cache.
For example, if page 16 of physical memory is assigned to page 0 of a process's virtual memory and the cache can hold 4 pages, the page coloring code will not assign page 20 of physical memory to page 1 of a process's virtual memory.
It would, instead, assign page 21 of physical memory.
The page coloring code attempts to avoid assigning page 20 because this maps over the same cache memory as page 16 and would result in non-optimal caching.
This code adds a significant amount of complexity to the VM memory allocation subsystem as you can well imagine, but the result is well worth the effort.
Page Coloring makes VM memory as deterministic as physical memory in regards to cache performance.
[[conclusion]]
== Conclusion
Virtual memory in modern operating systems must address a number of different issues efficiently and for many different usage patterns. The modular and algorithmic approach that BSD has historically taken allows us to study and understand the current implementation as well as relatively cleanly replace large sections of the code. There have been a number of improvements to the FreeBSD VM system in the last several years, and work is ongoing.
Virtual memory in modern operating systems must address a number of different issues efficiently and for many different usage patterns.
The modular and algorithmic approach that BSD has historically taken allows us to study and understand the current implementation as well as relatively cleanly replace large sections of the code.
There have been a number of improvements to the FreeBSD VM system in the last several years, and work is ongoing.
[[allen-briggs-qa]]
== Bonus QA session by Allen Briggs
=== What is the interleaving algorithm that you refer to in your listing of the ills of the FreeBSD 3.X swap arrangements?
FreeBSD uses a fixed swap interleave which defaults to 4. This means that FreeBSD reserves space for four swap areas even if you only have one, two, or three. Since swap is interleaved the linear address space representing the "four swap areas" will be fragmented if you do not actually have four swap areas. For example, if you have two swap areas A and B FreeBSD's address space representation for that swap area will be interleaved in blocks of 16 pages:
FreeBSD uses a fixed swap interleave which defaults to 4.
This means that FreeBSD reserves space for four swap areas even if you only have one, two, or three.
Since swap is interleaved the linear address space representing the "four swap areas" will be fragmented if you do not actually have four swap areas.
For example, if you have two swap areas A and B FreeBSD's address space representation for that swap area will be interleaved in blocks of 16 pages:
....
A B C D A B C D A B C D A B C D
....
FreeBSD 3.X uses a "sequential list of free regions" approach to accounting for the free swap areas. The idea is that large blocks of free linear space can be represented with a single list node ([.filename]#kern/subr_rlist.c#). But due to the fragmentation the sequential list winds up being insanely fragmented. In the above example, completely unused swap will have A and B shown as "free" and C and D shown as "all allocated". Each A-B sequence requires a list node to account for because C and D are holes, so the list node cannot be combined with the next A-B sequence.
FreeBSD 3.X uses a "sequential list of free regions" approach to accounting for the free swap areas.
The idea is that large blocks of free linear space can be represented with a single list node ([.filename]#kern/subr_rlist.c#).
But due to the fragmentation the sequential list winds up being insanely fragmented.
In the above example, completely unused swap will have A and B shown as "free" and C and D shown as "all allocated".
Each A-B sequence requires a list node to account for because C and D are holes, so the list node cannot be combined with the next A-B sequence.
Why do we interleave our swap space instead of just tack swap areas onto the end and do something fancier? It is a whole lot easier to allocate linear swaths of an address space and have the result automatically be interleaved across multiple disks than it is to try to put that sophistication elsewhere.
The fragmentation causes other problems. Being a linear list under 3.X, and having such a huge amount of inherent fragmentation, allocating and freeing swap winds up being an O(N) algorithm instead of an O(1) algorithm. Combined with other factors (heavy swapping) and you start getting into O(N^2) and O(N^3) levels of overhead, which is bad. The 3.X system may also need to allocate KVM during a swap operation to create a new list node which can lead to a deadlock if the system is trying to pageout pages in a low-memory situation.
The fragmentation causes other problems.
Being a linear list under 3.X, and having such a huge amount of inherent fragmentation, allocating and freeing swap winds up being an O(N) algorithm instead of an O(1) algorithm.
Combined with other factors (heavy swapping) and you start getting into O(N^2) and O(N^3) levels of overhead, which is bad.
The 3.X system may also need to allocate KVM during a swap operation to create a new list node which can lead to a deadlock if the system is trying to pageout pages in a low-memory situation.
Under 4.X we do not use a sequential list. Instead we use a radix tree and bitmaps of swap blocks rather than ranged list nodes. We take the hit of preallocating all the bitmaps required for the entire swap area up front but it winds up wasting less memory due to the use of a bitmap (one bit per block) instead of a linked list of nodes. The use of a radix tree instead of a sequential list gives us nearly O(1) performance no matter how fragmented the tree becomes.
Under 4.X we do not use a sequential list.
Instead we use a radix tree and bitmaps of swap blocks rather than ranged list nodes.
We take the hit of preallocating all the bitmaps required for the entire swap area up front but it winds up wasting less memory due to the use of a bitmap (one bit per block) instead of a linked list of nodes.
The use of a radix tree instead of a sequential list gives us nearly O(1) performance no matter how fragmented the tree becomes.
=== How is the separation of clean and dirty (inactive) pages related to the situation where you see low cache queue counts and high active queue counts in systat -vm? Do the systat stats roll the active and dirty pages together for the active queue count?
Yes, that is confusing. The relationship is "goal" verses "reality". Our goal is to separate the pages but the reality is that if we are not in a memory crunch, we do not really have to.
Yes, that is confusing.
The relationship is "goal" verses "reality".
Our goal is to separate the pages but the reality is that if we are not in a memory crunch, we do not really have to.
What this means is that FreeBSD will not try very hard to separate out dirty pages (inactive queue) from clean pages (cache queue) when the system is not being stressed, nor will it try to deactivate pages (active queue -> inactive queue) when the system is not being stressed, even if they are not being used.
=== In man:ls[1] the / vmstat 1 example, would not some of the page faults be data page faults (COW from executable file to private page)? I.e., I would expect the page faults to be some zero-fill and some program data. Or are you implying that FreeBSD does do pre-COW for the program data?
A COW fault can be either zero-fill or program-data. The mechanism is the same either way because the backing program-data is almost certainly already in the cache. I am indeed lumping the two together. FreeBSD does not pre-COW program data or zero-fill, but it _does_ pre-map pages that exist in its cache.
A COW fault can be either zero-fill or program-data.
The mechanism is the same either way because the backing program-data is almost certainly already in the cache.
I am indeed lumping the two together.
FreeBSD does not pre-COW program data or zero-fill, but it _does_ pre-map pages that exist in its cache.
=== In your section on page table optimizations, can you give a little more detail about pv_entry and vm_page (or should vm_page be vm_pmap-as in 4.4, cf. pp. 180-181 of McKusick, Bostic, Karel, Quarterman)? Specifically, what kind of operation/reaction would require scanning the mappings?
A `vm_page` represents an (object,index#) tuple. A `pv_entry` represents a hardware page table entry (pte). If you have five processes sharing the same physical page, and three of those processes's page tables actually map the page, that page will be represented by a single `vm_page` structure and three `pv_entry` structures.
A `vm_page` represents an (object,index#) tuple. A `pv_entry` represents a hardware page table entry (pte).
If you have five processes sharing the same physical page, and three of those processes's page tables actually map the page, that page will be represented by a single `vm_page` structure and three `pv_entry` structures.
`pv_entry` structures only represent pages mapped by the MMU (one `pv_entry` represents one pte). This means that when we need to remove all hardware references to a `vm_page` (in order to reuse the page for something else, page it out, clear it, dirty it, and so forth) we can simply scan the linked list of pv_entry's associated with that vm_page to remove or modify the pte's from their page tables.
`pv_entry` structures only represent pages mapped by the MMU (one `pv_entry` represents one pte).
This means that when we need to remove all hardware references to a `vm_page` (in order to reuse the page for something else, page it out, clear it, dirty it, and so forth) we can simply scan the linked list of pv_entry's associated with that vm_page to remove or modify the pte's from their page tables.
Under Linux there is no such linked list. In order to remove all the hardware page table mappings for a `vm_page` linux must index into every VM object that _might_ have mapped the page. For example, if you have 50 processes all mapping the same shared library and want to get rid of page X in that library, you need to index into the page table for each of those 50 processes even if only 10 of them have actually mapped the page. So Linux is trading off the simplicity of its design against performance. Many VM algorithms which are O(1) or (small N) under FreeBSD wind up being O(N), O(N^2), or worse under Linux. Since the pte's representing a particular page in an object tend to be at the same offset in all the page tables they are mapped in, reducing the number of accesses into the page tables at the same pte offset will often avoid blowing away the L1 cache line for that offset, which can lead to better performance.
Under Linux there is no such linked list.
In order to remove all the hardware page table mappings for a `vm_page` linux must index into every VM object that _might_ have mapped the page.
For example, if you have 50 processes all mapping the same shared library and want to get rid of page X in that library, you need to index into the page table for each of those 50 processes even if only 10 of them have actually mapped the page.
So Linux is trading off the simplicity of its design against performance.
Many VM algorithms which are O(1) or (small N) under FreeBSD wind up being O(N), O(N^2), or worse under Linux.
Since the pte's representing a particular page in an object tend to be at the same offset in all the page tables they are mapped in, reducing the number of accesses into the page tables at the same pte offset will often avoid blowing away the L1 cache line for that offset, which can lead to better performance.
FreeBSD has added complexity (the `pv_entry` scheme) in order to increase performance (to limit page table accesses to _only_ those pte's that need to be modified).
But FreeBSD has a scaling problem that Linux does not in that there are a limited number of `pv_entry` structures and this causes problems when you have massive sharing of data. In this case you may run out of `pv_entry` structures even though there is plenty of free memory available. This can be fixed easily enough by bumping up the number of `pv_entry` structures in the kernel config, but we really need to find a better way to do it.
But FreeBSD has a scaling problem that Linux does not in that there are a limited number of `pv_entry` structures and this causes problems when you have massive sharing of data.
In this case you may run out of `pv_entry` structures even though there is plenty of free memory available.
This can be fixed easily enough by bumping up the number of `pv_entry` structures in the kernel config, but we really need to find a better way to do it.
In regards to the memory overhead of a page table verses the `pv_entry` scheme: Linux uses "permanent" page tables that are not throw away, but does not need a `pv_entry` for each potentially mapped pte. FreeBSD uses "throw away" page tables but adds in a `pv_entry` structure for each actually-mapped pte. I think memory utilization winds up being about the same, giving FreeBSD an algorithmic advantage with its ability to throw away page tables at will with very low overhead.
In regards to the memory overhead of a page table verses the `pv_entry` scheme: Linux uses "permanent" page tables that are not throw away, but does not need a `pv_entry` for each potentially mapped pte.
FreeBSD uses "throw away" page tables but adds in a `pv_entry` structure for each actually-mapped pte.
I think memory utilization winds up being about the same, giving FreeBSD an algorithmic advantage with its ability to throw away page tables at will with very low overhead.
=== Finally, in the page coloring section, it might help to have a little more description of what you mean here. I did not quite follow it.
Do you know how an L1 hardware memory cache works? I will explain: Consider a machine with 16MB of main memory but only 128K of L1 cache. Generally the way this cache works is that each 128K block of main memory uses the _same_ 128K of cache. If you access offset 0 in main memory and then offset 128K in main memory you can wind up throwing away the cached data you read from offset 0!
Do you know how an L1 hardware memory cache works? I will explain: Consider a machine with 16MB of main memory but only 128K of L1 cache.
Generally the way this cache works is that each 128K block of main memory uses the _same_ 128K of cache.
If you access offset 0 in main memory and then offset 128K in main memory you can wind up throwing away the cached data you read from offset 0!
Now, I am simplifying things greatly. What I just described is what is called a "direct mapped" hardware memory cache. Most modern caches are what are called 2-way-set-associative or 4-way-set-associative caches. The set-associatively allows you to access up to N different memory regions that overlap the same cache memory without destroying the previously cached data. But only N.
Now, I am simplifying things greatly.
What I just described is what is called a "direct mapped" hardware memory cache.
Most modern caches are what are called 2-way-set-associative or 4-way-set-associative caches.
The set-associatively allows you to access up to N different memory regions that overlap the same cache memory without destroying the previously cached data.
But only N.
So if I have a 4-way set associative cache I can access offset 0, offset 128K, 256K and offset 384K and still be able to access offset 0 again and have it come from the L1 cache. If I then access offset 512K, however, one of the four previously cached data objects will be thrown away by the cache.
So if I have a 4-way set associative cache I can access offset 0, offset 128K, 256K and offset 384K and still be able to access offset 0 again and have it come from the L1 cache.
If I then access offset 512K, however, one of the four previously cached data objects will be thrown away by the cache.
It is extremely important... _extremely_ important for most of a processor's memory accesses to be able to come from the L1 cache, because the L1 cache operates at the processor frequency. The moment you have an L1 cache miss and have to go to the L2 cache or to main memory, the processor will stall and potentially sit twiddling its fingers for _hundreds_ of instructions worth of time waiting for a read from main memory to complete. Main memory (the dynamic ram you stuff into a computer) is __slow__, when compared to the speed of a modern processor core.
It is extremely important... _extremely_ important for most of a processor's memory accesses to be able to come from the L1 cache, because the L1 cache operates at the processor frequency.
The moment you have an L1 cache miss and have to go to the L2 cache or to main memory, the processor will stall and potentially sit twiddling its fingers for _hundreds_ of instructions worth of time waiting for a read from main memory to complete.
Main memory (the dynamic ram you stuff into a computer) is __slow__, when compared to the speed of a modern processor core.
Ok, so now onto page coloring: All modern memory caches are what are known as _physical_ caches. They cache physical memory addresses, not virtual memory addresses. This allows the cache to be left alone across a process context switch, which is very important.
Ok, so now onto page coloring: All modern memory caches are what are known as _physical_ caches.
They cache physical memory addresses, not virtual memory addresses.
This allows the cache to be left alone across a process context switch, which is very important.
But in the UNIX(R) world you are dealing with virtual address spaces, not physical address spaces. Any program you write will see the virtual address space given to it. The actual _physical_ pages underlying that virtual address space are not necessarily physically contiguous! In fact, you might have two pages that are side by side in a processes address space which wind up being at offset 0 and offset 128K in _physical_ memory.
But in the UNIX(R) world you are dealing with virtual address spaces, not physical address spaces.
Any program you write will see the virtual address space given to it.
The actual _physical_ pages underlying that virtual address space are not necessarily physically contiguous!
In fact, you might have two pages that are side by side in a processes address space which wind up being at offset 0 and offset 128K in _physical_ memory.
A program normally assumes that two side-by-side pages will be optimally cached. That is, that you can access data objects in both pages without having them blow away each other's cache entry. But this is only true if the physical pages underlying the virtual address space are contiguous (insofar as the cache is concerned).
A program normally assumes that two side-by-side pages will be optimally cached.
That is, that you can access data objects in both pages without having them blow away each other's cache entry.
But this is only true if the physical pages underlying the virtual address space are contiguous (insofar as the cache is concerned).
This is what Page coloring does. Instead of assigning _random_ physical pages to virtual addresses, which may result in non-optimal cache performance, Page coloring assigns _reasonably-contiguous_ physical pages to virtual addresses. Thus programs can be written under the assumption that the characteristics of the underlying hardware cache are the same for their virtual address space as they would be if the program had been run directly in a physical address space.
This is what Page coloring does.
Instead of assigning _random_ physical pages to virtual addresses, which may result in non-optimal cache performance, Page coloring assigns _reasonably-contiguous_ physical pages to virtual addresses.
Thus programs can be written under the assumption that the characteristics of the underlying hardware cache are the same for their virtual address space as they would be if the program had been run directly in a physical address space.
Note that I say "reasonably" contiguous rather than simply "contiguous". From the point of view of a 128K direct mapped cache, the physical address 0 is the same as the physical address 128K. So two side-by-side pages in your virtual address space may wind up being offset 128K and offset 132K in physical memory, but could also easily be offset 128K and offset 4K in physical memory and still retain the same cache performance characteristics. So page-coloring does _not_ have to assign truly contiguous pages of physical memory to contiguous pages of virtual memory, it just needs to make sure it assigns contiguous pages from the point of view of cache performance and operation.
Note that I say "reasonably" contiguous rather than simply "contiguous".
From the point of view of a 128K direct mapped cache, the physical address 0 is the same as the physical address 128K.
So two side-by-side pages in your virtual address space may wind up being offset 128K and offset 132K in physical memory, but could also easily be offset 128K and offset 4K in physical memory and still retain the same cache performance characteristics.
So page-coloring does _not_ have to assign truly contiguous pages of physical memory to contiguous pages of virtual memory, it just needs to make sure it assigns contiguous pages from the point of view of cache performance and operation.