os/doc
3
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Migrate doc to Hugo/AsciiDoctor

Browse source 
I'm very pleased to announce the release of
our new website and documentation using
the new toolchain with Hugo and AsciiDoctor.

To get more information about the new toolchain
please read the FreeBSD Documentation Project Primer[1],
Hugo docs[2] and AsciiDoctor docs[3].

Acknowledgment:
Benedict Reuschling <bcr@>
Glen Barber <gjb@>
Hiroki Sato <hrs@>
Li-Wen Hsu <lwhsu@>
Sean Chittenden <seanc@>
The FreeBSD Foundation

[1] https://docs.FreeBSD.org/en/books/fdp-primer/
[2] https://gohugo.io/documentation/
[3] https://docs.asciidoctor.org/home/

Approved by:    doceng, core
This commit is contained in:
Sergio Carlavilla Delgado 2021-01-26 00:31:29 +01:00
parent 0cff342f42
commit 989d921f5d
14375 changed files with 1277937 additions and 3448703 deletions
Download patch file Download diff file Expand all files Collapse all files

101
documentation/tools/README Normal file
View file

@ -0,0 +1,101 @@
How to add your key to the list
===============================
The addkey.sh script will export your key and generate the correct
XML incantations to have your key listed in the "PGP Keys" chapter.
It was written for GnuPG, but shouldn't be difficult to modify to use
PGP if you absolutely won't use GnuPG.
For addkey.sh to perform its magic, gpg must be in your path, and the
key(s) you wish to export must be present in the default keyring or in
a keyring listed in your ~/.gnupg/options file.
The simplest case is when your login name (as reported by 'id -nu') is
the same as your freefall login name, and all the keys you wish to add
(and only those keys) have a UID that contains your FreeBSD.org email
address. For instance, in my case:
| % sh addkey.sh
| Retrieving key...
| pub 1024D/27848427 2000-10-13 Dag-Erling Smørgrav (low security key) <des@freebsd.org>
| pub 1024D/0512E49A 2001-06-26 Dag-Erling Smørgrav (FreeBSD) <des@freebsd.org>
| WARNING: Multiple keys; exporting all. If this is not what you want,
| WARNING: you should specify a key ID on the command line.
| Generating des.key...
| Adding key to entity list...
|
| Unless you are already listed there, you should now add the
| following text to pgpkeys-developers.xml (unless this is a role key
| or you are a core member. In that case add to pgpkeys-officers.xml
| or pgpkeys-core.xml). Remember to keep the list sorted by last name!
|
| <sect2 id="pgpkey-des">
| <title>&a.des.email;</title>
| &pgpkey.des;
| </sect2>
|
| Don't forget to 'svn add des.key' if this is a new entry,
| and check your diffs before committing!
If your login name is different from your freefall login name, specify
your freefall login name as the first argument to addkey.sh:
| % sh addkey.sh des
If you want to tune what keys are to be exported, specify the key IDs,
or strings to look for in the UID, just like you would to gpg:
| % sh addkey.sh des 0512E49A
You can specify as few or as many IDs as you want, and addkey.sh will
export all matching keys, and those only. If you didn't specify any
key IDs on the command line, addkey.sh will look for a key that
matches your FreeBSD.org email address.
The script will generate a file named <freefall-login>.key (des.key in
my case) with programlisting elements: one that contains the
fingerprints of all the keys you exported, and one that contains an
ASCII dump of those keys, as produced by 'gpg --armor --export'. It
will also add a line to pgpkeys.ent defining an external entity named
pgpkey.<freefall-login> (pgpkey.des in my case) that refers to the key
file.
As you can see from the example above, addkey.sh will also tell you
what keys it exported (check this carefully before committing!) and
provide instructions for actually adding the key to the handbook.
Beware that the keys are supposed to be ordered by surname; if you're
unsure where to add your entry, use finger(1) to check the surnames of
those around you.
Once you've successfully run addkey.sh, 'svn add' your key file, check
the diffs, and commit.
How to update your key
======================
There are two ways to update your key. The first is to follow the
above instructions as if you were adding your key for the first time -
except that you don't need to edit chapter.xml, because you're
already listed there. The addkey.sh script will snarf the RCS Id from
the old key file and generate a new one with whatever keys you
specified.
The second method, which you can use if you're not adding or removing
any keys, but just updating the ones that are already there (because
you've had your key signed, or you've added or removed UIDs), is to
execute the third line of your key file, which is a pre-cooked
addkey.sh command line with your freefall login and the numeric key
IDs of the keys contained in the file. Just do the equivalent of:
| % eval $(grep addkey des.key)
The exact command, of course, will depend on your shell, and on your
freefall login name.
-- des@freebsd.org
$FreeBSD$

156
documentation/tools/addkey.sh Executable file
View file

@ -0,0 +1,156 @@
#!/bin/sh
#
progname=$(basename $(realpath $0))
# Print an informational message
info() {
echo "$@" >&2
}
# Print a warning message
warning() {
echo "WARNING: $@" >&2
}
# Print an error message and exit
error() {
echo "ERROR: $@" >&2
exit 1
}
# Print usage message and exit
usage() {
echo "usage: ${progname} [user] [keyid ...]\n" >&2
exit 1
}
# Look for gpg
gpg=$(which gpg)
if [ -z "${gpg}" -o ! -x "${gpg}" ] ; then
error "gpg does not seem to be installed"
fi
gpg() {
LANG=C "${gpg}" \
--display-charset utf-8 \
--no-greeting \
--no-secmem-warning \
--keyid-format long \
--list-options no-show-uid-validity \
"$@"
}
# Look up key by key ID
getkeybyid() {
gpg --with-colons --list-keys "$1" 2>/dev/null | awk -F: \
'$5 ~ /^\([0-9A-F]{8}\)?'"$1"'$/i && $12 ~ /ESC/ { print $5 }'
}
# Look up key by email
getkeybyemail() {
gpg --with-colons --list-keys "$1" 2>/dev/null | awk -F: \
'$10 ~ /<'"$1"'>/i && $12 ~ /ESC/ { print $5 }'
}
# The first command-line argument can be a user name or a key ID.
if [ $# -gt 0 ] && expr "$1" : '^[a-z][0-9a-z-]*$' >/dev/null ; then
me="$1"
shift
fi
if [ -z "${me}" ] ; then
me=$(id -nu)
fi
if [ -z "${me}" ] ; then
error "Unable to determine user name."
fi
if ! expr "${me}" : '^[0-9a-z][0-9a-z-]*$' >/dev/null ; then
error "${me} does not seem like a valid user name."
fi
if [ $# -ne 0 ] ; then
# Verify the keys that were specified on the command line
for arg ; do
case $(expr "${arg}" : '^[0-9A-Fa-f]\{8,16\}$') in
8)
warning "${arg}: recommend using 16-digit keyid"
;&
16)
keyid=$(getkeybyid "${arg}")
if [ -n "${keyid}" ] ; then
keyids="${keyids} ${keyid}"
else
warning "${arg} not found"
fi
;;
*)
warning "${arg} does not appear to be a valid key ID"
;;
esac
done
else
# Search for keys by freebsd.org email
email="${me}@FreeBSD.org"
keyids=$(getkeybyemail "${email}")
case $(echo "${keyids}" | wc -w) in
0)
error "no keys found for ${email}"
;;
1)
;;
*)
warning "Multiple keys found for <${email}>; exporting all."
warning "If this is not what you want, specify a key ID" \
"on the command line."
;;
esac
fi
# :(
if [ -z "${keyids}" ] ; then
error "no valid keys were found"
fi
# Generate key file
keyfile="${me}.key"
info "Generating ${keyfile}..."
(
echo '<!--'
echo "sh ${progname} ${me}" ${keyids} ";"
echo '-->'
echo '<programlisting xmlns="http://docbook.org/ns/docbook" role="pgpfingerprint"><![CDATA['
gpg --fingerprint ${keyids}
echo ']]></programlisting>'
echo '<programlisting xmlns="http://docbook.org/ns/docbook" role="pgpkey"><![CDATA['
gpg --no-version --armor --export ${keyids}
echo ']]></programlisting>'
) >"${keyfile}"
info "Adding key to entity list..."
if ! grep -qwF "pgpkey.${me}" pgpkeys.ent ; then
mv pgpkeys.ent pgpkeys.ent.orig || exit 1
(
cat pgpkeys.ent.orig
echo "<!ENTITY pgpkey.${me} SYSTEM \"${keyfile}\">"
) | sort -u >pgpkeys.ent
fi
cat <<EOF
Unless you are already listed there, you should now add the following
text to pgpkeys-developers.xml. Remember to keep the list sorted by
last name!
<sect2 xmlns="http://docbook.org/ns/docbook" xml:id="pgpkey-${me}">
<title>&a.${me}.email;</title>
&pgpkey.${me};
</sect2>
If this is a role key or you are a core member, you should add it to
either pgpkeys-officers.xml or pgpkeys-core.xml instead.
If this is a new entry, don't forget to run the following commands
before committing:
% git add ${keyfile}
EOF

204
documentation/tools/books-toc-creator.py Normal file
View file

@ -0,0 +1,204 @@
# -*- coding: utf-8 -*-
"""
BSD 2-Clause License
Copyright (c) 2020-2021, The FreeBSD Project
Copyright (c) 2020-2021, Sergio Carlavilla <carlavilla@FreeBSD.org>
This script will generate the Table of Contents of the Handbook
"""
#!/usr/bin/env python3
import sys, getopt
import re
languages = []
def setAppendixTitle(language):
languages = {
'en': 'Appendix',
'de': 'Anhang',
'el': 'Παράρτημα',
'es': 'Apéndice',
'fr': 'Annexe',
'hu': 'függelék',
'it': 'Appendice',
'ja': '付録',
'mn': 'Хавсралт',
'nl': 'Bijlage',
'pl': 'Dodatek',
'pt-br': 'Apêndice',
'ru': 'Приложение',
'zh-cn': '附录',
'zh-tw': '附錄'
}
return languages.get(language)
def setPartTitle(language):
languages = {
'en': 'Part',
'de': 'Teil',
'el': 'Μέρος',
'es': 'Parte',
'fr': 'Partie',
'hu': 'rész',
'it': 'Parte',
'ja': 'パート',
'mn': 'хэсэг',
'nl': 'Deel',
'pl': 'Część',
'pt-br': 'Parte',
'ru': 'Часть',
'zh-cn': '部分',
'zh-tw': '部'
}
return languages.get(language)
def setChapterTitle(language):
languages = {
'en': 'Chapter',
'de': 'Kapitel',
'el': 'Κεφάλαιο',
'es': 'Capítulo',
'fr': 'Chapitre',
'hu': 'Fejezet',
'it': 'Capitolo',
'ja': '章',
'mn': 'Бүлэг',
'nl': 'Hoofdstuk',
'pl': 'Rozdział',
'pt-br': 'Capítulo',
'ru': 'Глава',
'zh-cn': '章',
'zh-tw': '章'
}
return languages.get(language)
def setTOCTitle(language):
languages = {
'en': 'Table of Contents',
'de': 'Inhaltsverzeichnis',
'el': 'Πίνακας Περιεχομένων',
'es': 'Tabla de contenidos',
'fr': 'Table des matières',
'hu': 'Tartalom',
'it': 'Indice',
'ja': '目次',
'mn': 'Гарчиг',
'nl': 'Inhoudsopgave',
'pl': 'Spis treści',
'pt-br': 'Índice',
'ru': 'Содержание',
'zh-cn': '目录',
'zh-tw': '內容目錄'
}
return languages.get(language)
def getPartNumber(number):
numbers = {
1: 'I',
2: 'II',
3: 'III',
4: 'IV',
5: 'V'
}
return numbers.get(number)
def checkIsPart(chapter):
if "part" in chapter:
return True
return False
def checkIsPreface(chapterContent):
if "[preface]" in chapterContent:
return True
return False
def checkIsAppendix(chapterContent):
if "[appendix]" in chapterContent:
return True
return False
def main(argv):
try:
opts, args = getopt.getopt(argv,"hl:",["language="])
except getopt.GetoptError:
print('books-toc-creator.py -l <language>')
sys.exit(2)
for opt, arg in opts:
if opt == '-h':
print('books-toc-creator.py -l <language>')
sys.exit()
elif opt in ("-l", "--language"):
languages = arg.split(',')
for language in languages:
with open('./content/{0}/books/handbook/chapters-order.adoc'.format(language), 'r', encoding = 'utf-8') as chaptersFile:
chapters = [line.strip() for line in chaptersFile]
toc = "// Code generated by the FreeBSD Documentation toolchain. DO NOT EDIT.\n"
toc += "// Please don't change this file manually but run `make` to update it.\n"
toc += "// For more information, please read the FreeBSD Documentation Project Primer\n\n"
toc += "[.toc]\n"
toc += "--\n"
toc += '[.toc-title]\n'
toc += setTOCTitle(language) + '\n\n'
chapterCounter = 1
subChapterCounter = 1
partCounter = 1
for chapter in chapters:
with open('./content/{0}/books/handbook/{1}'.format(language, chapter), 'r', encoding = 'utf-8') as chapterFile:
chapterContent = chapterFile.read().splitlines()
chapterFile.close()
chapter = chapter.replace("/_index.adoc", "").replace(".adoc", "")
if checkIsPart(chapter):
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^={1} [^!<\n]+", chapterLine):
toc += "* link:{0}[{1} {2}. {3}]\n".format(chapter, setPartTitle(language), getPartNumber(partCounter), chapterLine.replace("=", "").strip())
partCounter += 1
elif checkIsPreface(chapterContent):
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^={1} [^!<\n]+", chapterLine):
toc += "* link:{0}[{1}]\n".format(chapter, chapterLine.replace("=", "").strip())
elif checkIsAppendix(chapterContent):
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^={1} [^!<\n]+", chapterLine):
toc += "** link:{0}[{1} {2}]\n".format(chapter, setAppendixTitle(language), chapterLine.replace("=", "").strip())
elif re.match(r"^={2} [^\n]+", chapterLine):
toc += "*** link:{0}/#{1}[{2}]\n".format(chapter, chapterContent[lineNumber-2].replace("[[", "").replace("]]", ""), chapterLine.replace("==", "").lstrip())
else: # Normal chapter
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^={1} [^!<\n]+", chapterLine):
toc += "** link:{0}[{1} {2}. {3}]\n".format(chapter, setChapterTitle(language), chapterCounter, chapterLine.replace("=", "").strip())
elif re.match(r"^={2} [^\n]+", chapterLine):
toc += "*** link:{0}/#{1}[{2}.{3}. {4}]\n".format(chapter, chapterContent[lineNumber-2].replace("[[", "").replace("]]", ""), chapterCounter, subChapterCounter, chapterLine.replace("==", "").lstrip())
subChapterCounter += 1
chapterCounter += 1
subChapterCounter = 1 # Reset subChapterCounter
toc += "--\n"
with open('./content/{0}/books/handbook/toc.adoc'.format(language), 'w', encoding = 'utf-8') as tocFile:
tocFile.write(toc)
if __name__ == "__main__":
main(sys.argv[1:])

122
documentation/tools/books-toc-examples-creator.py Normal file
View file

@ -0,0 +1,122 @@
# -*- coding: utf-8 -*-
"""
BSD 2-Clause License
Copyright (c) 2020-2021, The FreeBSD Project
Copyright (c) 2020-2021, Sergio Carlavilla <carlavilla@FreeBSD.org>
This script will generate the Table of Contents of the Handbook
"""
#!/usr/bin/env python3
import sys, getopt
import re
languages = []
"""
To determine if a chapter is a chapter we are going to check if it is
anything else, an appendix, a part, the preface ... and if it is not
any of those, it will be a chapter.
It may not be the best option, but it works :)
"""
def checkIsChapter(chapter, chapterContent):
if "part" in chapter:
return False
elif "[preface]" in chapterContent:
return False
elif "[appendix]" in chapterContent:
return False
else:
return True
def setTOCTitle(language):
languages = {
'en': 'List of Examples',
'de': 'Liste der Beispiele',
'el': 'Κατάλογος Παραδειγμάτων',
'es': 'Lista de ejemplos',
'fr': 'Liste des exemples',
'hu': 'A példák listája',
'it': 'Lista delle tabelle',
'ja': '例の一覧',
'mn': 'Жишээний жагсаалт',
'nl': 'Lijst van voorbeelden',
'pl': 'Spis przykładów',
'pt-br': 'Lista de Exemplos',
'ru': 'Список примеров',
'zh-cn': '范例清单',
'zh-tw': '範例目錄'
}
return languages.get(language)
def main(argv):
try:
opts, args = getopt.getopt(argv,"hl:",["language="])
except getopt.GetoptError:
print('books-toc-examples-creator.py -l <language>')
sys.exit(2)
for opt, arg in opts:
if opt == '-h':
print('books-toc-examples-creator.py -l <language>')
sys.exit()
elif opt in ("-l", "--language"):
languages = arg.split(',')
for language in languages:
with open('./content/{}/books/books.adoc'.format(language), 'r', encoding = 'utf-8') as booksFile:
books = [line.strip() for line in booksFile]
for book in books:
with open('./content/{0}/books/{1}/chapters-order.adoc'.format(language, book), 'r', encoding = 'utf-8') as chaptersFile:
chapters = [line.strip() for line in chaptersFile]
toc = "// Code generated by the FreeBSD Documentation toolchain. DO NOT EDIT.\n"
toc += "// Please don't change this file manually but run `make` to update it.\n"
toc += "// For more information, please read the FreeBSD Documentation Project Primer\n\n"
toc += "[.toc]\n"
toc += "--\n"
toc += '[.toc-title]\n'
toc += setTOCTitle(language) + '\n\n'
chapterCounter = 1
exampleCounter = 1
for chapter in chapters:
with open('./content/{0}/books/{1}/{2}'.format(language, book, chapter), 'r', encoding = 'utf-8') as chapterFile:
chapterContent = chapterFile.read().splitlines()
chapterFile.close()
chapter = chapter.replace("/_index.adoc", "").replace(".adoc", "").replace("/chapter.adoc", "")
exampleId = ""
exampleTitle = ""
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^\[example\]+", chapterLine) and re.match(r"^[.]{1}[^\n]+", chapterContent[lineNumber-2]) and re.match(r"^\[\[[^\n]+\]\]", chapterContent[lineNumber-3]):
exampleTitle = chapterContent[lineNumber-2]
exampleId = chapterContent[lineNumber-3]
if book == "handbook":
toc += "* {0}.{1} link:{2}#{3}[{4}]\n".format(chapterCounter, exampleCounter, chapter, exampleId.replace("[[", "").replace("]]", ""), exampleTitle[1:])
else:
toc += "* {0}.{1} link:{2}#{3}[{4}]\n".format(chapterCounter, exampleCounter, "", exampleId.replace("[[", "").replace("]]", ""), exampleTitle[1:])
exampleCounter += 1
else:
exampleId = ""
exampleTitle = ""
if checkIsChapter(chapter, chapterContent):
chapterCounter += 1
exampleCounter = 1 # Reset example counter
toc += "--\n"
with open('./content/{0}/books/{1}/toc-examples.adoc'.format(language, book), 'w', encoding = 'utf-8') as tocFile:
tocFile.write(toc)
if __name__ == "__main__":
main(sys.argv[1:])

121
documentation/tools/books-toc-figures-creator.py Normal file
View file

@ -0,0 +1,121 @@
# -*- coding: utf-8 -*-
"""
BSD 2-Clause License
Copyright (c) 2020-2021, The FreeBSD Project
Copyright (c) 2020-2021, Sergio Carlavilla <carlavilla@FreeBSD.org>
This script will generate the Table of Contents of the Handbook
"""
#!/usr/bin/env python3
import sys, getopt
import re
languages = []
"""
To determine if a chapter is a chapter we are going to check if it is
anything else, an appendix, a part, the preface ... and if it is not
any of those, it will be a chapter.
It may not be the best option, but it works :)
"""
def checkIsChapter(chapter, chapterContent):
if "part" in chapter:
return False
elif "[preface]" in chapterContent:
return False
elif "[appendix]" in chapterContent:
return False
else:
return True
def setTOCTitle(language):
languages = {
'en': 'List of Figures',
'de': 'Abbildungsverzeichnis',
'el': 'Κατάλογος Σχημάτων',
'es': 'Lista de figuras',
'fr': 'Liste des illustrations',
'hu': 'Az ábrák listája',
'it': 'Lista delle figure',
'ja': '図の一覧',
'mn': 'Зургийн жагсаалт',
'nl': 'Lijst van afbeeldingen',
'pl': 'Spis rysunków',
'pt-br': 'Lista de Figuras',
'ru': 'Список иллюстраций',
'zh-cn': '插图清单',
'zh-tw': '附圖目錄'
}
return languages.get(language)
def main(argv):
try:
opts, args = getopt.getopt(argv,"hl:",["language="])
except getopt.GetoptError:
print('books-toc-figures-creator.py -l <language>')
sys.exit(2)
for opt, arg in opts:
if opt == '-h':
print('books-toc-figures-creator.py -l <language>')
sys.exit()
elif opt in ("-l", "--language"):
languages = arg.split(',')
for language in languages:
with open('./content/{}/books/books.adoc'.format(language), 'r', encoding = 'utf-8') as booksFile:
books = [line.strip() for line in booksFile]
for book in books:
with open('./content/{0}/books/{1}/chapters-order.adoc'.format(language, book), 'r', encoding = 'utf-8') as chaptersFile:
chapters = [line.strip() for line in chaptersFile]
toc = "// Code generated by the FreeBSD Documentation toolchain. DO NOT EDIT.\n"
toc += "// Please don't change this file manually but run `make` to update it.\n"
toc += "// For more information, please read the FreeBSD Documentation Project Primer\n\n"
toc += "[.toc]\n"
toc += "--\n"
toc += '[.toc-title]\n'
toc += setTOCTitle(language) + '\n\n'
chapterCounter = 1
figureCounter = 1
for chapter in chapters:
with open('./content/{0}/books/{1}/{2}'.format(language, book, chapter), 'r', encoding = 'utf-8') as chapterFile:
chapterContent = chapterFile.read().splitlines()
chapterFile.close()
chapter = chapter.replace("/_index.adoc", "").replace(".adoc", "").replace("/chapter.adoc", "")
figureId = ""
figureTitle = ""
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^image::+", chapterLine) and re.match(r"^[.]{1}[^\n]+", chapterContent[lineNumber-2]) and re.match(r"^\[\[[^\n]+\]\]", chapterContent[lineNumber-3]):
figureTitle = chapterContent[lineNumber-2]
figureId = chapterContent[lineNumber-3]
if book == "handbook":
toc += "* {0}.{1} link:{2}#{3}[{4}]\n".format(chapterCounter, figureCounter, chapter, figureId.replace("[[", "").replace("]]", ""), figureTitle[1:])
else:
toc += "* {0}.{1} link:{2}#{3}[{4}]\n".format(chapterCounter, figureCounter, "", figureId.replace("[[", "").replace("]]", ""), figureTitle[1:])
figureCounter += 1
else:
figureId = ""
figureTitle = ""
if checkIsChapter(chapter, chapterContent):
chapterCounter += 1
figureCounter = 1 # Reset figure counter
toc += "--\n"
with open('./content/{0}/books/{1}/toc-figures.adoc'.format(language, book), 'w', encoding = 'utf-8') as tocFile:
tocFile.write(toc)
if __name__ == "__main__":
main(sys.argv[1:])

181
documentation/tools/books-toc-parts-creator.py Normal file
View file

@ -0,0 +1,181 @@
# -*- coding: utf-8 -*-
"""
BSD 2-Clause License
Copyright (c) 2020-2021, The FreeBSD Project
Copyright (c) 2020-2021, Sergio Carlavilla <carlavilla@FreeBSD.org>
This script will generate the Table of Contents of the Handbook
"""
#!/usr/bin/env python3
import sys, getopt
import re
import os.path
languages = []
def cleanTocFile(language, tocCounter):
path = './content/{0}/books/handbook/toc-{1}.adoc'.format(language, tocCounter)
if os.path.exists(path):
tocFile = open(path, 'r+', encoding = 'utf-8')
tocFile.truncate(0)
def appendCommendAndTitle(language, tocCounter, tocContent):
toc = "// Code generated by the FreeBSD Documentation toolchain. DO NOT EDIT.\n"
toc += "// Please don't change this file manually but run `make` to update it.\n"
toc += "// For more information, please read the FreeBSD Documentation Project Primer\n\n"
toc += "[.toc]\n"
toc += "--\n"
toc += '[.toc-title]\n'
toc += setTOCTitle(language) + '\n\n'
toc += tocContent
toc += "--\n"
tocFile = open('./content/{0}/books/handbook/toc-{1}.adoc'.format(language, tocCounter), 'a+', encoding = 'utf-8')
tocFile.write(toc)
def checkIsPart(chapter):
if "part" in chapter:
return True
return False
def checkIsPreface(chapterContent):
if "[preface]" in chapterContent:
return True
return False
def checkIsAppendix(chapterContent):
if "[appendix]" in chapterContent:
return True
return False
def setAppendixTitle(language):
languages = {
'en': 'Appendix',
'de': 'Anhang',
'el': 'Παράρτημα',
'es': 'Apéndice',
'fr': 'Annexe',
'hu': 'függelék',
'it': 'Appendice',
'ja': '付録',
'mn': 'Хавсралт',
'nl': 'Bijlage',
'pl': 'Dodatek',
'pt-br': 'Apêndice',
'ru': 'Приложение',
'zh-cn': '附录',
'zh-tw': '附錄'
}
return languages.get(language)
def setChapterTitle(language):
languages = {
'en': 'Chapter',
'de': 'Kapitel',
'el': 'Κεφάλαιο',
'es': 'Capítulo',
'fr': 'Chapitre',
'hu': 'Fejezet',
'it': 'Capitolo',
'ja': '章',
'mn': 'Бүлэг',
'nl': 'Hoofdstuk',
'pl': 'Rozdział',
'pt-br': 'Capítulo',
'ru': 'Глава',
'zh-cn': '章',
'zh-tw': '章'
}
return languages.get(language)
def setTOCTitle(language):
languages = {
'en': 'Table of Contents',
'de': 'Inhaltsverzeichnis',
'el': 'Πίνακας Περιεχομένων',
'es': 'Tabla de contenidos',
'fr': 'Table des matières',
'hu': 'Tartalom',
'it': 'Indice',
'ja': '目次',
'mn': 'Гарчиг',
'nl': 'Inhoudsopgave',
'pl': 'Spis treści',
'pt-br': 'Índice',
'ru': 'Содержание',
'zh-cn': '目录',
'zh-tw': '內容目錄'
}
return languages.get(language)
def main(argv):
try:
opts, args = getopt.getopt(argv,"hl:",["language="])
except getopt.GetoptError:
print('handbook-toc-creator.py -l <language>')
sys.exit(2)
for opt, arg in opts:
if opt == '-h':
print('handbook-toc-creator.py -l <language>')
sys.exit()
elif opt in ("-l", "--language"):
languages = arg.split(',')
for language in languages:
with open('./content/{}/books/handbook/chapters-order.adoc'.format(language), 'r', encoding = 'utf-8') as chaptersFile:
chapters = [line.strip() for line in chaptersFile]
chapterCounter = 1
subChapterCounter = 1
partCounter = 0
toc = ""
for chapter in chapters:
with open('./content/{0}/books/handbook/{1}'.format(language, chapter), 'r', encoding = 'utf-8') as chapterFile:
chapterContent = chapterFile.read().splitlines()
chapterFile.close()
chapter = chapter.replace("/_index.adoc", "").replace(".adoc", "")
if checkIsPart(chapter):
if partCounter > 0:
cleanTocFile(language, partCounter)
appendCommendAndTitle(language, partCounter, toc)
toc = "" # Clean toc content
partCounter += 1
elif checkIsPreface(chapterContent):
pass
elif checkIsAppendix(chapterContent):
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if (re.match(r"^={1} [^\n]+", chapterLine)):
toc += "* link:../{0}[{1} {2}]\n".format(chapter, setAppendixTitle(language), chapterLine.replace("=", "").strip())
elif (re.match(r"^={2} [^\n]+", chapterLine)):
toc += "** link:../{0}/#{1}[{2}]\n".format(chapter, chapterContent[lineNumber-2].replace("[[", "").replace("]]", ""), chapterLine.replace("==", "").lstrip())
else:
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^={1} [^\n]+", chapterLine):
toc += "* link:../{0}[{1} {2}. {3}]\n".format(chapter, setChapterTitle(language), chapterCounter, chapterLine.replace("=", "").strip())
elif re.match(r"^={2} [^\n]+", chapterLine):
toc += "** link:../{0}/#{1}[{2}.{3}. {4}]\n".format(chapter, chapterContent[lineNumber-2].replace("[[", "").replace("]]", ""), chapterCounter, subChapterCounter, chapterLine.replace("==", "").lstrip())
subChapterCounter += 1
chapterCounter += 1
subChapterCounter = 1 # Reset subChapterCounter
cleanTocFile(language, partCounter)
appendCommendAndTitle(language, partCounter, toc) # For the last part
if __name__ == "__main__":
main(sys.argv[1:])

121
documentation/tools/books-toc-tables-creator.py Normal file
View file

@ -0,0 +1,121 @@
# -*- coding: utf-8 -*-
"""
BSD 2-Clause License
Copyright (c) 2020-2021, The FreeBSD Project
Copyright (c) 2020-2021, Sergio Carlavilla <carlavilla@FreeBSD.org>
This script will generate the Table of Contents of the Handbook
"""
#!/usr/bin/env python3
import sys, getopt
import re
languages = []
"""
To determine if a chapter is a chapter we are going to check if it is
anything else, an appendix, a part, the preface ... and if it is not
any of those, it will be a chapter.
It may not be the best option, but it works :)
"""
def checkIsChapter(chapter, chapterContent):
if "part" in chapter:
return False
elif "[preface]" in chapterContent:
return False
elif "[appendix]" in chapterContent:
return False
else:
return True
def setTOCTitle(language):
languages = {
'en': 'List of Tables',
'de': 'Tabellenverzeichnis',
'el': 'Κατάλογος Πινάκων',
'es': 'Lista de tablas',
'fr': 'Liste des tableaux',
'hu': 'A táblázatok listája',
'it': 'Lista delle tabelle',
'ja': '表の一覧',
'mn': 'Хүснэгтийн жагсаалт',
'nl': 'Lijst van tabellen',
'pl': 'Spis tabel',
'pt-br': 'Lista de Tabelas',
'ru': 'Список таблиц',
'zh-cn': '表格清单',
'zh-tw': '附表目錄'
}
return languages.get(language)
def main(argv):
try:
opts, args = getopt.getopt(argv,"hl:",["language="])
except getopt.GetoptError:
print('books-toc-tables-creator.py -l <language>')
sys.exit(2)
for opt, arg in opts:
if opt == '-h':
print('books-toc-tables-creator.py -l <language>')
sys.exit()
elif opt in ("-l", "--language"):
languages = arg.split(',')
for language in languages:
with open('./content/{}/books/books.adoc'.format(language), 'r', encoding = 'utf-8') as booksFile:
books = [line.strip() for line in booksFile]
for book in books:
with open('./content/{0}/books/{1}/chapters-order.adoc'.format(language, book), 'r', encoding = 'utf-8') as chaptersFile:
chapters = [line.strip() for line in chaptersFile]
toc = "// Code generated by the FreeBSD Documentation toolchain. DO NOT EDIT.\n"
toc += "// Please don't change this file manually but run `make` to update it.\n"
toc += "// For more information, please read the FreeBSD Documentation Project Primer\n\n"
toc += "[.toc]\n"
toc += "--\n"
toc += '[.toc-title]\n'
toc += setTOCTitle(language) + '\n\n'
chapterCounter = 1
tableCounter = 1
for chapter in chapters:
with open('./content/{0}/books/{1}/{2}'.format(language, book, chapter), 'r', encoding = 'utf-8') as chapterFile:
chapterContent = chapterFile.read().splitlines()
chapterFile.close()
chapter = chapter.replace("/_index.adoc", "").replace(".adoc", "").replace("/chapter.adoc", "")
tableId = ""
tableTitle = ""
for lineNumber, chapterLine in enumerate(chapterContent, 1):
if re.match(r"^\|\=\=\=+", chapterLine) and re.match(r"^[.]{1}[^\n]+", chapterContent[lineNumber-3]) and re.match(r"^\[\[[^\n]+\]\]", chapterContent[lineNumber-4]):
tableTitle = chapterContent[lineNumber-3]
tableId = chapterContent[lineNumber-4]
if book == "handbook":
toc += "* {0}.{1} link:{2}#{3}[{4}]\n".format(chapterCounter, tableCounter, chapter, tableId.replace("[[", "").replace("]]", ""), tableTitle[1:])
else:
toc += "* {0}.{1} link:{2}#{3}[{4}]\n".format(chapterCounter, tableCounter, "", tableId.replace("[[", "").replace("]]", ""), tableTitle[1:])
tableCounter += 1
else:
tableId = ""
tableTitle = ""
if checkIsChapter(chapter, chapterContent):
chapterCounter += 1
tableCounter = 1 # Reset table counter
toc += "--\n"
with open('./content/{0}/books/{1}/toc-tables.adoc'.format(language, book), 'w', encoding = 'utf-8') as tocFile:
tocFile.write(toc)
if __name__ == "__main__":
main(sys.argv[1:])

264
documentation/tools/checkkey.sh Executable file
View file

@ -0,0 +1,264 @@
#!/bin/sh
#
# $FreeBSD$
#
#
# This script is intended to sanity-check PGP keys used by folks with
# @FreeBSD.org email addresses. The checks are intended to be derived
# from the information at
# <https://riseup.net/en/security/message-security/openpgp/gpg-best-practices#openpgp-key-checks>
#
# make sure we are in a well known language
LANG=C
\unalias -a
progname=${0##*/}
# Print an informational message
info() {
echo "$@" >&2
}
# Print a warning message
warning() {
echo "WARNING: $@" >&2
}
# Print an error message and exit
error() {
echo "ERROR: $@" >&2
exit 1
}
# Print usage message and exit
usage() {
echo "usage: ${progname} [user] [keyid ...]\n" >&2
exit 1
}
# Look for gpg
gpg=$(which gpg)
if [ $? -gt 0 -o -z "${gpg}" -o ! -x "${gpg}" ] ; then
error "Cannot find gpg"
fi
# Set up our internal default gpg invocation options
_gpg() {
${gpg} \
--display-charset utf-8 \
--no-greeting \
--no-secmem-warning \
--keyid-format long \
--list-options no-show-uid-validity \
"$@"
}
# Look up key by key ID
getkeybyid() {
_gpg --with-colons --list-keys "$1" 2>/dev/null | awk -F: \
'$5 ~ /^\([0-9A-F]{8}\)?'"$1"'$/i && $12 ~ /ESC/ { print $5 }'
}
# Look up key by email
getkeybyemail() {
_gpg --with-colons --list-keys "$1" 2>/dev/null | awk -F: \
'$10 ~ /<'"$1"'>/i && $12 ~ /ESC/ { print $5 }'
}
# The first command-line argument can be a user name or a key ID.
if [ $# -gt 0 ] ; then
id="$1"
shift
else
id=$(id -nu)
warning "No argument specified, calculating user ID"
fi
# Now let's try to figure out what kind of thing we have as an ID.
# We'll check for a keyid first, as it's readily distinguishable
# from other things, but if we see that we have one, we push it back
# onto the argument list for later processing (becasue we may have
# been given a list of keyods).
if echo "${id}" | egrep -q '^[0-9A-F]{16}$'; then
id_type="keyid"
set -- "${id}" $@
elif echo "${id}" | egrep -q '^[0-9A-F]{8}$'; then
id_type="keyid"
set -- "${id}" $@
elif echo "${id}" | egrep -iq '^[a-z][-0-9a-z_]*@([-0-9a-z]+\.)[-0-9a-z]+$'; then
id_type="email"
email="${id}"
elif echo "${id}" | egrep -iq '^[a-z][-0-9a-z_]*$'; then
id_type="login"
login="${id}"
email="${id}@FreeBSD.org"
else
error "Cannot recognize type of ${id} (keyid, login, or email)"
fi
if [ $# -ne 0 ] ; then
# Verify the keys that were specified on the command line
for arg ; do
case $(expr "${arg}" : '^[0-9A-Fa-f]\{8,16\}$') in
8)
warning "${arg}: recommend using 16-digit keyid"
;&
16)
keyid=$(getkeybyid "${arg}")
if [ -n "${keyid}" ] ; then
keyids="${keyids} ${keyid}"
else
warning "${arg} not found"
fi
;;
*)
warning "${arg} does not appear to be a valid key ID"
;;
esac
shift
done
else
# Search for keys by freebsd.org email
keyids=$(getkeybyemail "${email}")
case $(echo "${keyids}" | wc -w) in
0)
error "no keys found for ${email}"
;;
1)
;;
*)
warning "Multiple keys found for <${email}>; checking all."
warning "If this is not what you want, specify a key ID" \
"on the command line."
;;
esac
fi
# :(
if [ -z "${keyids}" ] ; then
error "no valid keys were found"
fi
# add a problem report to the list of problems with this key
badkey() {
key_problems=" ${key_problems}$@
"
}
exitstatus=0
# Check the keys
for key in ${keyids} ; do
# no problems found yet
key_problems=""
IFS_save="${IFS}"
key_info=$( ${gpg} --no-secmem-warning --export-options export-minimal --export ${key} \
| ${gpg} --no-secmem-warning --list-packets )
# primary keys should be RSA or DSA-2
IFS=""
version=$( echo $key_info | \
awk '$1 == "version" && $3 == "algo" {sub(",", "", $2); print $2; exit 0}' )
IFS="${IFS_save}"
if [ $version -lt 4 ]; then
badkey "This key is a deprecated version $version key!"
fi
IFS=""
algonum=$( echo $key_info | \
awk '$1 == "version" && $3 == "algo" {sub(",", "", $4); print $4; exit 0}' )
IFS="${IFS_save}"
case ${algonum} in
"1") algo="RSA" ;;
"17") algo="DSA" ;;
"18") algo="ECC" ;;
"19") algo="ECDSA" ;;
*) algo="*UNKNOWN*" ;;
esac
IFS=""
bitlen=$( echo $key_info | \
awk -F : '$1 ~ "pkey" { gsub("[^0-9]*","", $2); print $2; exit 0}' )
IFS="${IFS_save}"
echo "key ${key}: ${algo}, ${bitlen} bits"
case ${algo} in
RSA) ;;
DSA) if [ "${bitlen}" -le 1024 ]; then \
badkey "DSA, but not DSA-2"; \
fi ;;
*) badkey "non-preferred algorithm"
esac
# self-signatures must not use MD5 or SHA1
IFS=""
sig_algonum=$( echo $key_info | \
awk '$1 == "digest" && $2 == "algo" {sub(",", "", $3); print $3; exit 0}' )
IFS="${IFS_save}"
case sig_algonum in
1) sigs="MD5";;
2) sigs="SHA1";;
3) sigs="RIPEMD160";;
8) sigs="SHA256";;
9) sigs="SHA384";;
10) sigs="SHA512";;
11) sigs="SHA224";;
*)
esac
for sig in ${sigs}; do
if [ "${sig}" = "MD5" -o "${sig}" = "SHA1" ]; then
badkey "self-signature ${sig}"
fi
done
# digest algo pref must include at least one member of SHA-2
# at a higher priority than both MD5 and SHA1
IFS=""
algopref=$( echo $key_info | \
awk -F : '$1 ~ "pref-hash-algos" {gsub("[^ 0-9]", "", $2); print $2; exit 0}' )
IFS="${IFS_save}"
# if 3, 2, or 1 are before 11, 10, 9, or 8, then
set -- ${algopref}
if [ $1 -lt 4 ]; then
badkey "algorithm prefs do not have SHA-2 higher than MD5 or SHA1"
fi
# primary keys should have an expiration date at least a year
# in the future to make them worth committing, but no more
# than three years in the future
expires=$( _gpg --list-keys ${key} | \
awk "/$keyid .*expires:/ {sub(\"[^-0-9]\", \"\", \$NF); print \$NF; exit 0}" )
if [ -z "${expires}" ]; then
badkey "this key does not expire"
else
expires_s=$( date -jf "%F" "+%s" "${expires}" )
now_s=$( date "+%s" )
# 86400 == # seconds in a normal day
expire_int_d=$(( ( ${expires_s} - ${now_s} ) / 86400 ))
exp_min=$(( 1 * 365 )) # Min expiry time is 1 year
exp_max=$(( 3 * 365 + 1 )) # Max expiry time is 3 years
# We add 1 day because in a 3-year
# period, probability of a leap day
# is 297/400, about 0.74250
if [ ${expire_int_d} -lt ${exp_min} ]; then
badkey "Key $key expires in less than 1 year ($expire_int_d days)"
fi
if [ ${expire_int_d} -gt ${exp_max} ]; then
badkey "Key $key expires in more than 3 years ($expire_int_d days)"
fi
fi
# report problems
if [ -z "${key_problems}" ]; then
echo " key okay, ${key} meets minimal requirements" >&2
else
exitstatus=1
echo " ** problems found:" >&2
echo "${key_problems}" >&2
echo " ** key ${key} should not be used!"
fi
echo
done
exit ${exitstatus}

196
documentation/tools/pgpkeyreport Executable file
View file

@ -0,0 +1,196 @@
#!/usr/bin/env perl
use strict;
use warnings;
# $FreeBSD$
#
# Generate a report showing all of the active FreeBSD developers and
# all of the PGP keys registered, and whether those keys are still in
# date.
use File::Temp qw{tempdir};
use POSIX qw{strftime};
use constant {
SVNCOMMAND => '/usr/local/bin/svn',
SVNREPOURL => 'svn://svn.freebsd.org/',
REPOS => [qw{base ports doc}],
SVNCONF => 'svnadmin/conf',
PGPKEYPATH => 'head/share/pgpkeys',
};
$0 =~ s@.*/@@;
sub svn_checkout($$$)
{
my $repo = shift;
my $path = shift;
my $dest = shift;
my $output;
open SVN, "-|", SVNCOMMAND . " co " . SVNREPOURL . "$repo/$path $dest"
or die "$0: can't checkout $repo/$path -- $!\n";
while (<SVN>) {
$output .= $_;
}
close SVN;
return;
}
sub read_keys($)
{
my $keyfile = shift;
my $pgp = [];
open PGPKEY, "<$keyfile"
or die "$0: can't open $keyfile for reading -- $!";
while (<PGPKEY>) {
m@^pub\s+@
&& do {
my @fields = split /\s+/;
my $thiskey = {};
$thiskey->{keyid} = $fields[1];
$thiskey->{created} = $fields[2];
# Remove the first three fields -- some times, that is all
# there are, and we don't want to read the creation date
# in place of the expiry date.
splice @fields, 0, 3;
if ( @fields && $fields[-1] =~ m/^([0-9-]{10})/ ) {
$thiskey->{expiry} = $1;
} else {
$thiskey->{expiry} = '';
}
push @{$pgp}, $thiskey;
};
}
close PGPKEY;
return $pgp;
}
sub scan_for_keys($$)
{
my $developers = shift;
my $pgpkeydir = shift;
my $name;
opendir( my $dh, $pgpkeydir )
or die "$0: couldn't open directory $pgpkeydir -- $!\n";
while ( my $f = readdir $dh ) {
next
unless $f =~ m/.key\Z/;
chomp $f;
( $name = $f ) =~ s/.key//;
$developers->{$name}->{keys} = read_keys("$pgpkeydir/$f");
}
closedir $dh;
return $developers;
}
sub active_committers($$$)
{
my $developers = shift;
my $repo = shift;
my $path = shift;
my $n;
my $r;
$repo =~ m/^(.)/;
$r = $1;
open ACCESS, "<$path" or die "$0: can't open access file for $repo -- $!\n";
while ( my $name = <ACCESS> ) {
next
if $name =~ m/^#/;
($n) = split( /\s+/, $name );
chomp $n;
$developers->{$n}->{$repo} = $r;
}
return $developers;
}
sub is_expired($)
{
my $date = shift;
my $year;
my $month;
my $day;
my $unixtime;
my $expired;
# Tri-state logic: we answer one of "yes", "no" or "dunno"
#
# Date is typically a string of form YYYY-MM-DD but we will accept
# any punctuation character as the field separator.
( $year, $month, $day ) =
( $date =~ m/^(\d{4})[[:punct:]](\d{2})[[:punct:]](\d{2})/ );
return "unknown"
unless $year && $month && $day;
$unixtime = strftime( "%s", 0, 0, 0, $day, $month - 1, $year - 1900 );
if ( $unixtime < $^T ) {
$expired = "expired";
} else {
$expired = "";
}
return $expired;
}
MAIN:
{
my $workspace;
my $developers = {};
$workspace = tempdir( ".$0.XXXXXX", TMPDIR => 1, CLEANUP => 1 )
or die "$0: can't create temporary directory -- $!\n";
svn_checkout( 'doc', PGPKEYPATH, "$workspace/pgpkeys" );
$developers = scan_for_keys( $developers, "$workspace/pgpkeys" );
for my $repo ( @{&REPOS} ) {
svn_checkout( $repo, SVNCONF, "$workspace/${repo}-conf" );
$developers = active_committers( $developers, $repo,
"$workspace/${repo}-conf/access" );
}
printf "#%18s %-5s %-26s %-10s %-10s %s\n", 'username', 'bits', 'keyid',
'created', 'expired', 'state';
for my $d ( sort keys %{$developers} ) {
if ( !defined $developers->{$d}->{keys} ) {
printf "%19s %1s %1s %1s No PGP key\n", $d,
$developers->{$d}->{base} // '-',
$developers->{$d}->{ports} // '-',
$developers->{$d}->{doc} // '-';
}
for my $k ( @{ $developers->{$d}->{keys} } ) {
my $expired = is_expired( $k->{expiry} );
printf "%19s %1s %1s %1s %-26s %-10s %-10s %s\n", $d,
$developers->{$d}->{base} // '-',
$developers->{$d}->{ports} // '-',
$developers->{$d}->{doc} // '-',
$k->{keyid} // '', $k->{created} // '',
$k->{expiry} // '', $expired;
}
}
}