os/doc
3
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

As per pr #37353:

Browse source 
- Updates to hubs article.

Submitted by:	Daniel Lang <dl@leo.org>
PR:		37353
This commit is contained in:
Valentino Vaschetto 2002-04-22 21:55:14 +00:00
parent 0379aee1ac
commit f61d605eab
Notes: svn2git 2020-12-08 03:00:23 +00:00 
svn path=/head/; revision=12853
1 changed files with 144 additions and 66 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

210
en_US.ISO8859-1/articles/hubs/article.sgml
View file

@ -50,7 +50,7 @@
Depending on the set of releases, architectures,
and degree of completeness you want to mirror, a huge
amount of disk space may be consumed. Also keep in mind,
that official mirrors are probably required to be
that <emphasis>official</emphasis> mirrors are probably required to be
complete. The CVS repository and the web pages should
always be mirrored completely. Also note, that the
numbers stated here, are reflecting the current
@ -80,8 +80,9 @@
estimates here:
</para>
<itemizedlist>
<listitem><para>Local site, no public access: basically no minimum.</para></listitem>
<listitem><para>Unofficial public site: 34 Mbps is a good start.</para></listitem>
<listitem><para>Local site, no public access: basically no minimum,
but &lt; 2 Mbps could make syncing a pain.</para></listitem>
<listitem><para>Unofficial public site: 34 Mbps is probably a good start.</para></listitem>
<listitem><para>Official site: &gt; 100 Mbps is recommended, also your host
should be connected as close as possible to your border router.</para></listitem>
</itemizedlist>
@ -100,7 +101,22 @@
indeed consume some CPU. For AnonCVS it might
be a nice idea to set up a memory resident filesystem (MFS) of at least
300 MB, so you need to take this into account
for your memory requirements.
for your memory requirements. The following
are just examples to give you a very rough hint.
</para>
<para>
For a moderately visited site, that offers
<application>rsync</application>, you might
consider a current CPU with around 800Mhz - 1 GHz,
and at least 512MB RAM. This is probably the
minimum you want for an <emphasis>official</emphasis>
site.
</para>
<para>
For a frequently used site you need definitely
more RAM (consider 2GB as a good start),
and possibly more CPU, which could also mean,
that you need to go for a SMP system.
</para>
<para>
You also want to consider a fast disk subsystem.
@ -120,11 +136,11 @@
<sect2 id="mirror-services">
<title>Services to offer</title>
<para>
Every mirror site is required to have a set of core services
available. In addition to these basic services, which mirrors are
required to provide, there is a number of optional services that
server administrators may choose to offer. This section explains
which services you can provide and how to go about implementing them.
Every mirror site is required to have a set of core services
available. In addition to these basic services, which mirrors are
required to provide, there is a number of optional services that
server administrators may choose to offer. This section explains
which services you can provide and how to go about implementing them.
</para>
<sect3 id="mirror-serv-ftp">
<title>FTP (required for FTP fileset)</title>
@ -140,19 +156,19 @@
the path <filename>/pub/FreeBSD</filename>.
</para>
<para>
There is lot of software available which
There is a lot of software available which
can be set up to allow anonymous FTP
(in alphabetical order).
<itemizedlist>
<listitem><para><filename>/usr/libexec/ftpd</filename>: FreeBSD's own <application>ftpd</application>,
<listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd
can be used. Be sure to read &man.ftpd.8;.</para>
</listitem>
<listitem>
<para><filename role="package">ftp/ncftpd</filename>: A commercial package, free for
educational use.</para>
<para><filename role="package">ftp/ncftpd</filename>: A commercial package,
free for educational use.</para>
</listitem>
<listitem>
<para><filename role="package">ftp/oftpd</filename>: an ftpd designed with
<para><filename role="package">ftp/oftpd</filename>: An ftpd designed with
security as a main focus.</para>
</listitem>
<listitem>
@ -166,29 +182,32 @@
<listitem><para><filename role="package">ftp/vsftpd</filename>: The <quote>very secure</quote> ftpd.</para></listitem>
<listitem>
<para><filename role="package">ftp/wu-ftpd</filename>: The ftpd from Washington
University. It has become infamous, because of the huge
amount of security issues that have been found in it.
If you do choose to use this software be sure to
keep it up to date.</para>
University. It has become infamous, because of the huge
amount of security issues that have been found in it.
If you do choose to use this software be sure to
keep it up to date.
</para>
</listitem>
</itemizedlist>
FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the
most commonly ones. The others do not have a large userbase among mirror sites.
FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
<application>wu-ftpd</application> and maybe <application>ncftpd</application>
are among the most commonly ones.
The others do not have a large userbase among mirror sites.
</para>
</sect3>
<sect3 id="mirror-serv-rsync">
<title>RSYNC (optional for FTP fileset)</title>
<para>
Rsync is often also offered for convenience, for the
<application>Rsync</application> is often also offered for convenience, for the
contents of the FTP area of FreeBSD. The
protocol is different from FTP in many ways, and
overall, it can be stated, that it is much more
bandwidth friendly, as only differences between files
are transferred, not whole files.
Rsync does require significant amount of memory for
<application>Rsync</application> does require significant amount of memory for
each instance. The size depends on the size of
the synced module in terms of number of directories and
files. Rsync can use <command>rsh</command> and
files. <application>Rsync</application> can use <command>rsh</command> and
<command>ssh</command> (now default) as a transport,
or use it's own protocol for stand-alone access
(this is the preferred method for public rsync servers).
@ -204,7 +223,8 @@
<title>HTTP (required for webpages, optional for FTP fileset)</title>
<para>
If you want to offer the FreeBSD webpages, you need
to install a webserver. You may optionally offer the FTP fileset via HTTP.
to install a webserver a.k.a <application>httpd</application>.
You may optionally offer the FTP fileset via HTTP.
The choice of Webserver software is left up to the mirror administrator.
Some of the most popular choices are:
@ -240,19 +260,20 @@
<sect3 id="mirror-serv-cvsup">
<title>CVSup (desired for CVS repository)</title>
<para>
CVSup is a very efficient way of distributing files.
<application>CVSup</application> is a very efficient way of distributing files.
It works similar as rsync, but was specially designed for
the use with CVS repositories. If you want to offer the
FreeBSD CVS repository, you really want to consider
offering it via CVSup. Still it is possible to offer
the CVS repository via AnonCVS, FTP, Rsync or HTTP, but
people would not benefit from CVSup access.
CVSup was developed by &a.jdp;.
offering it via <application>CVSup</application>. Still it is possible to offer
the CVS repository via <application>AnonCVS</application>, FTP,
<application>Rsync</application> or HTTP, but
people would benefit much more from <application>CVSup</application> access.
<application>CVSup</application> was developed by &a.jdp;.
It is a bit tricky to install on non-FreeBSD platforms,
since it is written in Modula-3 and therefore requires
a Modula-3 environment. John Polstra has built a
stripped down version of M3, that is sufficient to
run CVSup, and can be installed much easier.
run <application>CVSup</application>, and can be installed much easier.
See <ulink url="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</ulink>
for details. Related ports are:
@ -285,29 +306,26 @@
CVSup is known to eat some CPU on the server as on the
client, since it needs to compare lots of files.
</para>
<note>
<para>
Please have look at the <application>CVSup</application>
like &man.cvsup.1; and consider using the <option>-s</option>
option, as it can reduce the amount of work to be done
a lot.
</para>
</note>
</sect3>
<sect3 id="mirror-anoncvs">
<title>AnonCVS (optional for CVS repository)</title>
<para>
If you have the CVS repository, you may want to offer
anonymous CVS access. There is not that much demand for it,
anonymous CVS access. A short warning first:
There is not that much demand for it,
and it requires some experience and you need to know,
what you are doing. Generally there are two ways, how
what you are doing.
</para>
<para>
Generally there are two ways, how
to access a CVS repository remotely: via
<emphasis>pserver</emphasis> or via <command>ssh</command>
(we don't consider <command>rsh</command>).
For anonymous access, <emphasis>pserver</emphasis> is
very well suited, but some still offer <command>ssh</command>
access as well. There is some custom crafted wrapper
program around, to be used as a login-shell for the
access as well. There is a custom crafted
<ulink url="ftp://ftp.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</ulink>
in the CVS repository, to be used as a login-shell for the
anonymous ssh account. It does a chroot, and therefore
requires the CVS repository to be available under the
anonymous user's home-directory, which may not be possible
@ -322,7 +340,7 @@
cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
</programlisting>
See the manpage for details of the options. See also the cvs <emphasis>info</emphasis>
page, about how additional ways, to make sure, access is read-only.
page, about additional ways to make sure, access is read-only.
It is advisable, that you create an unprivileged account,
preferably called <username>anoncvs</username>.
Also you need to create a file <filename>passwd</filename>
@ -383,7 +401,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
</itemizedlist>
<filename role="package">ftp/mirror</filename> was very popular, but seemed
to have some drawbacks, as its written in perl,
to have some drawbacks, as it is written in &man.perl.1;,
and did have real problems on mirroring large
directories like a FreeBSD site. There are rumors, that
the current version has fixed this, by allowing
@ -406,8 +424,8 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<application>rsync</application> is already mentioned
in <xref linkend="mirror-serv-rsync">.
Since <application>rsync</application> access is not
required, your preferred upstream site may not allow it,
but it already widely used, so chances are small, that
required, your preferred upstream site may not allow it.
Since it is quite common, though, chances are small, that
you cannot use it. You can always consider using an
upstream server, that offers it, just for the benefits
of rsync.
@ -418,7 +436,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
machine, most admins impose limitations on their
server. For a mirror, you should ask the site maintainer
you are syncing from, about their policy, and maybe
an exception for your host, since you are a mirror.
an exception for your host (since you are a mirror).
</para>
</note>
A command line to mirror FreeBSD could look like that:
@ -464,6 +482,14 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
way to mirror the archive, in terms of efficiency, but
it is only available from few sites. In fact I just know
<hostid>ftp-master.freebsd.org</hostid> for sure.
<note id="mirror-cvsup-s-option">
<para>
Please have look at the <application>CVSup</application> documentation
like &man.cvsup.1; and consider using the <option>-s</option>
option, as it can reduce the amount of work to be done
a lot.
</para>
</note>
</para>
</sect3>
</sect2>
@ -500,6 +526,13 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
You should also have a look at <filename>/usr/share/examples/cvsup</filename>
</para>
<note>
<para>
Please don't forget to consider the hint,
mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
above.
</para>
</note>
</sect3>
<sect3 id="mirror-cvs-other">
<title>Using other methods</title>
@ -509,7 +542,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
anyway. Since most sites offer the CVS repository as
part of the FTP fileset under the path
<filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
the following methods can of course be used.
the following methods could be used.
<itemizedlist>
<listitem><para><application>FTP</application></para></listitem>
<listitem><para><application>RSYNC</application></para></listitem>
@ -524,7 +557,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<important>
<para>
AFAICT, you can <emphasis>NOT</emphasis> AnonCVS to
You can <emphasis>NOT</emphasis> use AnonCVS to
mirror the CVS repository, since CVS does not allow
you to access the repository itself, but only checked
out versions of the modules.
@ -622,7 +655,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<important>
<para>
The building of the documentation, as well as lots
of side issues is documented itself in:
of side issues, is documented itself in:
<ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/">fdp-primer</ulink>.
Please read this piece of documentation, especially if you
have problems, building the documentation.
@ -636,6 +669,52 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
</para>
</sect3>
</sect2>
<sect2 id="mirror-how-often">
<title>How often should I mirror?</title>
<para>
Every mirror should be updated on a regular
basis. You will certainly need some script
framework for it, that will be called by
&man.cron.8;. Since nearly every admin
does this his own way, we cannot give
specific instructions. It could work
like this:
</para>
<procedure>
<step>
<para>
Put the command to run your mirroring application
in a script. Use of a plain <command>/bin/sh</command>
script is recommended.
</para>
</step>
<step>
<para>
Add some output redirections, so diagnostic
messages are logged to a file.
</para>
</step>
<step>
<para>
Test, if your script works. Check the logs.
</para>
</step>
<step>
<para>
Use &man.crontab.1; to add the script the
appropriate user's &man.crontab.5;.
</para>
</step>
</procedure>
<para>
Here are some recommended schedules:
<itemizedlist>
<listitem><para>FTP fileset: daily</para></listitem>
<listitem><para>CVS repository: daily to hourly</para></listitem>
<listitem><para>WWW pages: daily</para></listitem>
</itemizedlist>
</para>
</sect2>
</sect1>
<sect1 id="mirror-where">
<title>Where to mirror from</title>
@ -648,15 +727,15 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<para>
Mirrors are organized by country. All
official mirrors have a DNS entry of the form
<hostid>ftpX.CC.freebsd.org</hostid>. Whith
<emphasis>CC</emphasis> (i.e. country code) being the
<hostid>ftpN.CC.freebsd.org</hostid>. Whith
<emphasis>CC</emphasis> (i.e. country code) beeing the
<emphasis>top level domain</emphasis>
of the country, where this mirror is located;
and <emphasis>X</emphasis> is a number,
telling that the host would be the <emphasis>Xth</emphasis>
and <emphasis>N</emphasis> is a number,
telling that the host would be the <emphasis>Nth</emphasis>
mirror in that country.
(Same applies to <hostid>cvsupX.CC.freebsd.org</hostid>,
<hostid>wwwX.CC.freebsd.org</hostid>, etc.)
(Same applies to <hostid>cvsupN.CC.freebsd.org</hostid>,
<hostid>wwwN.CC.freebsd.org</hostid>, etc.)
There are mirrors with no <emphasis>CC</emphasis> part.
These are usually located in the US, but don't need to.
<hostid>ftp.freebsd.org</hostid> is currently
@ -749,13 +828,12 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
for access. If you are already an <emphasis>official</emphasis>
mirror, this certainly helps you getting access.
In any other case make sure your country really needs another mirror.
If it already has three or more, ask
the &a.hubs; first.
If it already has three or more, ask the &a.hubs; first.
</para>
<para>
There are just two master sites, for the FTP fileset and
for the CVS repository (the webpages and docs are obtained from CVS,
so there is no need for it).
There are just two master sites, one for the FTP fileset and
one for the CVS repository (the webpages and docs are
obtained from CVS, so there is no need for it).
</para>
<sect4 id="mirror-where-master-ftp">
<title>ftp-master.freebsd.org</title>
@ -800,7 +878,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
Set up the required authentication by following
<ulink url="http://people.freebsd.org/~jdp/cvpasswd/">these
instructions</ulink>. Make sure you specify the server as
<hostid>freefall.freebsd.org</hostid> on the cvpasswd
<hostid>freefall.freebsd.org</hostid> on the <command>cvpasswd</command>
command line, as described in this document,
even when you are contacting
<hostid>cvsup-master.freebsd.org</hostid>
@ -840,7 +918,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
official mirrors, since the project is sort of
tolerant here. It is more easy to say,
what <emphasis>official tier-1 mirrors</emphasis>
are required to. All other official mirrors,
are required to. All other official mirrors
can consider this a big <emphasis>should</emphasis>.
<note>
<para>
@ -882,7 +960,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
<sect2 id="mirror-official-become">
<title>How to become official then?</title>
<para>
An interesting questions, especially, since the state
An interesting question, especially, since the state
of being official comes with some benefits, like a much
higher bill from your ISP, as more people will be using
your site. Also it may be a key requirement, to get access
@ -925,8 +1003,8 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all
</step>
<step>
<para>
Then you can ask the &a.doc;, or the &a.hubs; to add your
mirror site to the mirror list in the
Then you can ask the &a.doc; or the &a.hubs;
to add your mirror site to the mirror list in the
<ulink url="http://www.freebsd.org/handbook">FreeBSD
Handbook</ulink>. Make sure you tell them the email address,
to list as the maintainer of the site.