os/doc
3
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Start to bring this up-to-date:

Browse source 
- Drop all references to providing CVSup mirrors
 - Drop all references to providing anonymous CVS mirrors
 - Drop all references to mirroring using CVSup, instead use rsync
 - Update the recommended arguments to rsync in the examples
 - Updatre size estimates for the individual parts
 - Change one reference to CVS to instead mention SVN

Note that we are still not accepting applications for new mirrors
of this style.
This commit is contained in:
Gavin Atkinson 2014-05-17 02:50:17 +00:00
parent 8e37c216d3
commit f3bb41af94
Notes: svn2git 2020-12-08 03:00:23 +00:00 
svn path=/head/; revision=44849
1 changed files with 21 additions and 376 deletions
Download patch file Download diff file Expand all files Collapse all files

397
en_US.ISO8859-1/articles/hubs/article.xml
View file

@ -57,7 +57,7 @@
and degree of completeness you want to mirror, a huge
amount of disk space may be consumed. Also keep in mind
that <emphasis>official</emphasis> mirrors are probably required to be
complete. The CVS repository and the web pages should
complete. The web pages should
always be mirrored completely. Also note that the
numbers stated here are reflecting the current
state (at &rel2.current;-RELEASE/&rel.current;-RELEASE). Further development and
@ -67,10 +67,9 @@
Here are some approximate figures:
</para>
<itemizedlist>
<listitem><para>Full FTP Distribution: 1.1 TB</para></listitem>
<listitem><para>CVS repository: 5.4 GB</para></listitem>
<listitem><para>CTM deltas: 3.2 GB</para></listitem>
<listitem><para>Web pages: 463 MB</para></listitem>
<listitem><para>Full FTP Distribution: 1.4 TB</para></listitem>
<listitem><para>CTM deltas: 10 GB</para></listitem>
<listitem><para>Web pages: 1GB</para></listitem>
</itemizedlist>
<para>
The current disk usage of FTP Distribution can be found at
@ -113,7 +112,7 @@
</para>
<para>
For a moderately visited site that offers
<application>Rsync</application>, you might
<application>rsync</application>, you might
consider a current CPU with around 800MHz - 1 GHz,
and at least 512MB RAM. This is probably the
minimum you want for an <emphasis>official</emphasis>
@ -127,7 +126,7 @@
</para>
<para>
You also want to consider a fast disk subsystem.
Operations on the CVS repository require a fast
Operations on the SVN repository require a fast
disk subsystem (RAID is highly advised). A SCSI
controller that has a cache of its own can also
speed up things since most of these services incur a
@ -181,17 +180,9 @@
</listitem>
<listitem><para><package>ftp/twoftpd</package>: As above.</para></listitem>
<listitem><para><package>ftp/vsftpd</package>: The <quote>very secure</quote> ftpd.</para></listitem>
<listitem>
<para><package>ftp/wu-ftpd</package>: The ftpd from Washington
University. It has become infamous, because of the huge
amount of security issues that have been found in it.
If you do choose to use this software be sure to
keep it up to date.
</para>
</listitem>
</itemizedlist>
<para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
<application>wu-ftpd</application> and maybe <application>ncftpd</application>
<para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>
and maybe <application>ncftpd</application>
are among the most commonly used FTPds.
The others do not have a large userbase among mirror sites. One
thing to consider is that you may need flexibility in limiting
@ -260,109 +251,6 @@
</listitem>
</itemizedlist>
</sect3>
<sect3 xml:id="mirror-serv-cvsup">
<title>CVSup (desired for CVS repository)</title>
<para>
<application>CVSup</application> is a very efficient way of distributing files.
It works similar to <application>rsync</application>, but was specially designed for
use with CVS repositories. If you want to offer the
FreeBSD CVS repository, you really want to consider
offering it via <application>CVSup</application>. It is possible to offer
the CVS repository via <application>AnonCVS</application>, FTP,
<application>rsync</application> or HTTP, but
people would benefit much more from <application>CVSup</application> access.
<application>CVSup</application> was developed by &a.jdp.email;.
It is a bit tricky to install on non-FreeBSD platforms,
since it is written in Modula-3 and therefore requires
a Modula-3 environment. &a.jdp; has built a
stripped down version of M3 that is sufficient to
run <application>CVSup</application>, and can be installed much easier.
See <link xlink:href="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</link>
for details. Related ports are:</para>
<itemizedlist>
<listitem>
<para><package>net/cvsup</package>: The native CVSup port (client and server)
which requires <package>lang/ezm3</package> now.</para>
</listitem>
<listitem>
<para><package>net/cvsup-mirror</package>: The CVSup mirror kit, which requires
<package>net/cvsup-without-gui</package>, and configures it mirror-ready. Some
site administrators may want a different setup though.
</para>
</listitem>
</itemizedlist>
<para>There are a few more like
<package>net/cvsup-without-gui</package> you might want to have
a look at. If you prefer a static binary package, take a look
<link xlink:href="http://people.FreeBSD.org/~jdp/s1g/">here</link>.
This page still refers to the S1G bug that was present
in <application>CVSup</application>. Maybe
John will set up a generic download-site to get
static binaries for various platforms.
</para>
<para>
It is possible to use <application>CVSup</application> to offer
any kind of fileset, not just CVS repositories,
but configuration can be complex.
<application>CVSup</application> is known to eat some CPU on both the server and the
client, since it needs to compare lots of files.
</para>
</sect3>
<sect3 xml:id="mirror-anoncvs">
<title>AnonCVS (optional for CVS repository)</title>
<para>
If you have the CVS repository, you may want to offer
anonymous CVS access. A short warning first:
There is not much demand for it,
it requires some experience, and you need to know
what you are doing.
</para>
<para>
Generally there are two ways
to access a CVS repository remotely: via
<emphasis>pserver</emphasis> or via <command>ssh</command>
(we do not consider <command>rsh</command>).
For anonymous access, <emphasis>pserver</emphasis> is
very well suited, but some still offer <command>ssh</command>
access as well. There is a custom crafted
<link xlink:href="ftp://ftp.FreeBSD.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</link>
in the CVS repository, to be used as a login-shell for the
anonymous ssh account. It does a chroot, and therefore
requires the CVS repository to be available under the
anonymous user's home-directory. This may not be possible
for all sites. If you just offer <emphasis>pserver</emphasis>
this restriction does not apply, but you may run with
more security risks. You do not need to install any special
software, since &man.cvs.1; comes with
FreeBSD. You need to enable access via <command>inetd</command>,
so add an entry into your <filename>/etc/inetd.conf</filename>
like this:</para>
<programlisting>
cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
</programlisting>
<para>See the manpage for details of the options. Also see the CVS <emphasis>info</emphasis>
page about additional ways to make sure access is read-only.
It is advised that you create an unprivileged account,
preferably called <systemitem class="username">anoncvs</systemitem>.
Also you need to create a file <filename>passwd</filename>
in your <filename>/home/ncvs/CVSROOT</filename> and assign a
CVS password (empty or <literal>anoncvs</literal>) to that user.
The directory <filename>/anoncvstmp</filename> is a special
purpose memory based file system. It is not required but
advised since &man.cvs.1; creates a shadow directory
structure in your <filename>/tmp</filename> which is
not used after the operation but slows things
dramatically if real disk operations are required.
Here is an excerpt from <filename>/etc/fstab</filename>,
how to set up such a MFS:</para>
<programlisting>
/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid 0 0
</programlisting>
<para>This is (of course) tuned a lot, and was suggested by &a.jdp.email;.
</para>
</sect3>
</sect2>
</sect1>
<sect1 xml:id="mirror-howto">
@ -374,8 +262,8 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
the various parts of FreeBSD, what tools to use,
and where to mirror from.
</para>
<sect2 xml:id="mirror-ftp">
<title>FTP</title>
<sect2 xml:id="mirror-ftp-rsync">
<title>Mirroring the FTP site</title>
<para>
The FTP area is the largest amount of data that
needs to be mirrored. It includes the <emphasis>distribution
@ -388,39 +276,8 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
for various FreeBSD versions,
and various architectures.
</para>
<sect3 xml:id="mirror-ftp-ftp">
<title>With FTP mirror</title>
<para>
You can use a <application>FTP mirror</application>
program to get the files. Some of the most commonly used are:</para>
<itemizedlist>
<listitem><para><package>ftp/mirror</package></para></listitem>
<listitem><para><package>ftp/ftpmirror</package></para></listitem>
<listitem><para><package>ftp/emirror</package></para></listitem>
<listitem><para><package>ftp/spegla</package></para></listitem>
<listitem><para><package>ftp/omi</package></para></listitem>
<listitem><para><package>ftp/wget</package></para></listitem>
</itemizedlist>
<para><package>ftp/mirror</package> was very popular, but seemed
to have some drawbacks, as it is written in &man.perl.1;,
and had real problems with mirroring large
directories like a FreeBSD site. There are rumors that
the current version has fixed this by allowing
a different algorithm for comparing
the directory structure to be specified.
</para>
<para>
In general FTP is not really good for mirroring. It transfers
the whole file if it has changed, and does
not create a single data stream which would benefit from
a large TCP congestion window.
</para>
</sect3>
<sect3 xml:id="mirror-ftp-rsync">
<title>With rsync</title>
<para>
A better way to mirror the FTP area is <application>rsync</application>.
The best way to mirror the FTP area is <application>rsync</application>.
You can install the port <package>net/rsync</package> and then use
rsync to sync with your upstream host.
<application>rsync</application> is already mentioned
@ -440,7 +297,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
</para>
</note>
<para>A command line to mirror FreeBSD might look like:</para>
<screen>&prompt.user; <userinput>rsync -vaz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/</userinput>
<screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/</userinput>
</screen>
<para>Consult the documentation for <application>rsync</application>,
which is also available at
@ -453,188 +310,15 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
want to set up a script framework that calls such a command
via &man.cron.8;.
</para>
</sect3>
<sect3 xml:id="mirror-ftp-cvsup">
<title>With CVSup</title>
<para>
A few sites, including the one-and-only <systemitem class="fqdomainname">ftp-master.FreeBSD.org</systemitem>
even offer <application>CVSup</application> to mirror the contents of
the FTP space. You need to install a <application>CVSup</application>
client, preferably from the port <package>net/cvsup</package>.
(Also reread <xref linkend="mirror-serv-cvsup"/>.)
A sample <filename>supfile</filename> suitable for <systemitem class="fqdomainname">ftp-master.FreeBSD.org</systemitem>
looks like this:</para>
<programlisting>
#
# FreeBSD archive supfile from master server
#
*default host=ftp-master.FreeBSD.org
*default base=/usr
*default prefix=/pub
#*default release=all
*default delete use-rel-suffix
*default umask=002
# If your network link is a T1 or faster, comment out the following line.
#*default compress
FreeBSD-archive release=all preserve
</programlisting>
<para>It seems <application>CVSup</application> would be the best
way to mirror the archive in terms of efficiency, but
it is only available from few sites.</para>
<note xml:id="mirror-cvsup-s-option">
<para>
Please have look at the <application>CVSup</application> documentation
like &man.cvsup.1; and consider using the <option>-s</option>
option. This reduces I/O operations by assuming the
recorded information about each file is correct.</para>
</note>
</sect3>
</sect2>
<sect2 xml:id="mirror-cvs">
<title>Mirroring the CVS repository</title>
<para>There are various ways to mirror the CVS repository.
<application>CVSup</application> is the most common method.</para>
<sect3 xml:id="mirror-cvs-cvsup">
<title>Using CVSup</title>
<para>
<application>CVSup</application> is described in some
detail in <xref linkend="mirror-serv-cvsup"/> and <xref linkend="mirror-ftp-cvsup"/>.
</para>
<para>It is very easy to setup a
<application>CVSup</application> mirror. Installing
<package>net/cvsup-mirror</package> will
make sure all of the needed programs are installed and then
gather all the needed information to configure the mirror.</para>
<note>
<para>
Please do not forget to consider the hint
mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
above.
</para>
</note>
</sect3>
<sect3 xml:id="mirror-cvs-other">
<title>Using other methods</title>
<para>
Using other methods than <application>CVSup</application> is
generally not recommended. We describe them in short here
anyway. Since most sites offer the CVS repository as
part of the FTP fileset under the path
<filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
the following methods could be used.</para>
<itemizedlist>
<listitem><para><application>FTP</application></para></listitem>
<listitem><para><application>Rsync</application></para></listitem>
<listitem><para><application>HTTP</application></para></listitem>
</itemizedlist>
<important>
<para>AnonCVS cannot be used to mirror the CVS repository
since CVS does not allow you to access the repository
itself, only checked out versions of the modules.</para>
</important>
</sect3>
</sect2>
<sect2 xml:id="mirror-www">
<title>Mirroring the WWW pages</title>
<para>
The best way is to check out the <emphasis>www</emphasis>
distribution from CVS. If you have a local mirror of the
CVS repository, it is as easy as:</para>
<screen>&prompt.user; <userinput>cvs -d /home/ncvs co www</userinput></screen>
<para>and a <emphasis>cronjob</emphasis>, that calls <command>cvs up -d -P</command>
on a regular basis, maybe just after your repository was updated.
Of course, the files need to remain in a directory available
for public WWW access. The installation and configuration of a
web server is not discussed here.
</para>
<para>
If you do not have a local repository, you can use
<application>CVSup</application> to maintain an <quote>up to date copy</quote>
of the www pages. A sample supfile can be found in
<filename>/usr/share/examples/cvsup/www-supfile</filename> and
could look like this:</para>
<programlisting>
#
# WWW module supfile for FreeBSD
#
*default host=cvsup3.de.FreeBSD.org
*default base=/usr
*default prefix=/usr/local
*default release=cvs tag=.
*default delete use-rel-suffix
# If your network link is a T1 or faster, comment out the following line.
*default compress
# This collection retrieves the www/ tree of the FreeBSD repository
www
</programlisting>
<para>
Using <package>ftp/wget</package> or other web-mirror tools is
not recommended.
</para>
<sect3 xml:id="mirror-www-doc">
<title>Mirroring the FreeBSD documentation</title>
<para>
Since the documentation is referenced a lot from the
web pages, it is recommended that you mirror the
FreeBSD documentation as well. However, this is not
as trivial as the www-pages alone.
</para>
<para>
First of all, you should get the doc sources,
again preferably via <application>CVSup</application>.
Here is a corresponding sample supfile:</para>
<programlisting>
#
# FreeBSD documentation supfile
#
*default host=cvsup3.de.FreeBSD.org
*default base=/usr
*default prefix=/usr/share
*default release=cvs tag=.
*default delete use-rel-suffix
# If your network link is a T1 or faster, comment out the following line.
#*default compress
# This will retrieve the entire doc branch of the FreeBSD repository.
# This includes the handbook, FAQ, and translations thereof.
doc-all
</programlisting>
<para>
Then you need to install a couple of ports.
You are lucky, there is a meta-port:
<package>textproc/docproj</package> to do the work
for you. You need to set up some
environment variables, like
<literal>SGML_CATALOG_FILES</literal>.
Also have a look at your <filename>/etc/make.conf</filename>
(copy <filename>/usr/share/examples/etc/make.conf</filename> if
you do not have one), and look at the
<literal>DOC_LANG</literal> variable.
Now you are probably ready to run <command>make</command>
in your doc directory (<filename>/usr/share/doc</filename>
by default) and build the documentation.
Again you need to make it accessible for your web server
and make sure the links point to the right location.</para>
<important>
<para>
The building of the documentation, as well as lots
of side issues, is documented itself in the
<link xlink:href="&url.books.fdp-primer;">&os; Documentation
Project Primer</link>.
Please read this piece of documentation, especially if you
have problems building the documentation.
</para>
</important>
</sect3>
The FreeBSD website should only be mirrored via
<application>rsync</application>.</para>
<para>A command line to mirror the FreeBSD web site might look like:</para>
<screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/</userinput>
</screen>
</sect2>
<sect2 xml:id="mirror-how-often">
<title>How often should I mirror?</title>
@ -683,7 +367,6 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
Here are some recommended schedules:</para>
<itemizedlist>
<listitem><para>FTP fileset: daily</para></listitem>
<listitem><para>CVS repository: hourly</para></listitem>
<listitem><para>WWW pages: daily</para></listitem>
</itemizedlist>
</sect2>
@ -708,7 +391,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<emphasis>N</emphasis> is a number,
telling that the host would be the <emphasis>Nth</emphasis>
mirror in that country.
(Same applies to <systemitem>cvsupN.CC.FreeBSD.org</systemitem>,
(Same applies to
<systemitem>wwwN.CC.FreeBSD.org</systemitem>, etc.)
There are mirrors with no <emphasis>CC</emphasis> part.
These are the mirror sites that are very well connected and
@ -812,9 +495,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
that.
</para>
<para>
There are three master sites for the FTP fileset and
one for the CVS repository (the web pages and docs are
obtained from CVS, so there is no need for master).
There is one master site for the FTP fileset.
</para>
<sect4 xml:id="mirror-where-master-ftp">
<title>ftp-master.FreeBSD.org</title>
@ -823,11 +504,9 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
</para>
<para>
<systemitem>ftp-master.FreeBSD.org</systemitem> provides
<application>rsync</application> and <application>CVSup</application>
<application>rsync</application>
access, in addition to FTP.
Refer to <xref linkend="mirror-ftp-rsync"/> and
<xref linkend="mirror-ftp-cvsup"/> how to access
via these protocols.
Refer to <xref linkend="mirror-ftp-rsync"/>.
</para>
<para>
Mirrors are also encouraged to allow <application>rsync</application>
@ -835,32 +514,6 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<emphasis>Tier-1</emphasis>-mirrors.
</para>
</sect4>
<sect4 xml:id="mirror-where-master-cvsup">
<title>cvsup-master.FreeBSD.org</title>
<para>
This is the master site for the CVS repository.
</para>
<para>
<systemitem>cvsup-master.FreeBSD.org</systemitem> provides
<application>CVSup</application> access only.
See <xref linkend="mirror-cvs-cvsup"/> for details.
</para>
<para>
To get access, you need to contact the &a.cvsup-master;.
Make sure you read the
<link xlink:href="http://people.FreeBSD.org/~jdp/cvsup-access/">FreeBSD CVSup Access Policy</link>
first!
</para>
<para>
Set up the required authentication by following
<link xlink:href="http://people.FreeBSD.org/~jdp/cvpasswd/">these
instructions</link>. Make sure you specify the server as
<systemitem>freefall.FreeBSD.org</systemitem> on the <command>cvpasswd</command>
command line, as described in this document,
even when you are contacting
<systemitem>cvsup-master.FreeBSD.org</systemitem>
</para>
</sect4>
</sect3>
</sect2>
</sect1>
@ -897,14 +550,6 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
what <emphasis>official tier-1 mirrors</emphasis>
are required to. All other official mirrors
can consider this a big <emphasis>should</emphasis>.</para>
<note>
<para>
The following applies mainly to the FTP fileset,
since a CVS repository should always be mirrored
completely, and the web pages are a case of
its own.
</para>
</note>
<para>
Tier-1 mirrors are required to:</para>
<itemizedlist>