1085 lines
48 KiB
Text
1085 lines
48 KiB
Text
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook V4.1-Based Extension//EN" [
|
|
<!ENTITY % man PUBLIC "-//FreeBSD//ENTITIES DocBook Manual Page Entities//EN">
|
|
%man;
|
|
<!ENTITY % authors PUBLIC "-//FreeBSD//ENTITIES DocBook Author Entities//EN">
|
|
%authors;
|
|
<!ENTITY % mailing-lists PUBLIC "-//FreeBSD//ENTITIES DocBook Mailing List Entities//EN">
|
|
%mailing-lists;
|
|
<!ENTITY % not.published "IGNORE">
|
|
]>
|
|
|
|
<article>
|
|
<articleinfo>
|
|
<title>Mirroring FreeBSD</title>
|
|
<pubdate>$FreeBSD$</pubdate>
|
|
<authorgroup>
|
|
<author>
|
|
<firstname>Jun</firstname>
|
|
<surname>Kuriyama</surname>
|
|
<affiliation>
|
|
<address><email>kuriyama@FreeBSD.org</email></address>
|
|
</affiliation>
|
|
</author>
|
|
<author>
|
|
<firstname>Valentino</firstname>
|
|
<surname>Vaschetto</surname>
|
|
<affiliation>
|
|
<address><email>logo@FreeBSD.org</email></address>
|
|
</affiliation>
|
|
</author>
|
|
<author>
|
|
<firstname>Daniel</firstname>
|
|
<surname>Lang</surname>
|
|
<affiliation>
|
|
<address><email>dl@leo.org</email></address>
|
|
</affiliation>
|
|
</author>
|
|
</authorgroup>
|
|
<abstract>
|
|
<para>An in-progress article on how to mirror FreeBSD, aimed at
|
|
hub administrators</para>
|
|
</abstract>
|
|
</articleinfo>
|
|
|
|
<sect1 id="mirror-requirements">
|
|
<title>Requirements for FreeBSD mirrors</title>
|
|
<sect2 id="mirror-diskspace">
|
|
<title>Disk Space</title>
|
|
<para>
|
|
Disk space is one of the most important requirements.
|
|
Depending on the set of releases, architectures,
|
|
and degree of completeness you want to mirror, a huge
|
|
amount of disk space may be consumed. Also keep in mind,
|
|
that <emphasis>official</emphasis> mirrors are probably required to be
|
|
complete. The CVS repository and the web pages should
|
|
always be mirrored completely. Also note, that the
|
|
numbers stated here, are reflecting the current
|
|
state (at 4.5-RELEASE). Further development and
|
|
releases will only increase the required amount.
|
|
Also make sure, to keep some (ca. 10-20%) extra space
|
|
around, just to be sure.
|
|
Here are some approximate figures:
|
|
</para>
|
|
<itemizedlist>
|
|
<listitem><para>Full FTP Distribution: 60 GB</para></listitem>
|
|
<listitem><para>CVS repository: 1.4 GB</para></listitem>
|
|
<listitem><para>CTM deltas: 1.5 GB</para></listitem>
|
|
<listitem><para>Webpages: 150 MB</para></listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
<sect2 id="mirror-bandwidth">
|
|
<title>Network Connection/Bandwidth</title>
|
|
<para>
|
|
Of course, you need to be connected to the internet.
|
|
The required bandwidth depends on your intended use
|
|
of the mirror. If you just want to mirror some
|
|
parts of FreeBSD for local use at your site/intranet,
|
|
the demand may be much smaller, than if you want to
|
|
make the files publicly available or even if you intend
|
|
to become an official mirror. We can only give rough
|
|
estimates here:
|
|
</para>
|
|
<itemizedlist>
|
|
<listitem><para>Local site, no public access: basically no minimum,
|
|
but < 2 Mbps could make syncing a pain.</para></listitem>
|
|
<listitem><para>Unofficial public site: 34 Mbps is probably a good start.</para></listitem>
|
|
<listitem><para>Official site: > 100 Mbps is recommended, also your host
|
|
should be connected as close as possible to your border router.</para></listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
<sect2 id="mirror-system">
|
|
<title>System Requirements, CPU, RAM</title>
|
|
<para>
|
|
This also depends on the expected amount of clients,
|
|
which is determined by the servers policy. It is
|
|
also affected by the types of services you want to offer.
|
|
Plain FTP or HTTP services may not require a huge
|
|
amount of resources. Watch out, if you provide
|
|
CVSup, rsync or even AnonCVS. This can have a huge
|
|
impact on CPU and memory requirements. Especially
|
|
rsync is considered a memory hog, and CVSup does
|
|
indeed consume some CPU. For AnonCVS it might
|
|
be a nice idea to set up a memory resident filesystem (MFS) of at least
|
|
300 MB, so you need to take this into account
|
|
for your memory requirements. The following
|
|
are just examples to give you a very rough hint.
|
|
</para>
|
|
<para>
|
|
For a moderately visited site, that offers
|
|
<application>rsync</application>, you might
|
|
consider a current CPU with around 800Mhz - 1 GHz,
|
|
and at least 512MB RAM. This is probably the
|
|
minimum you want for an <emphasis>official</emphasis>
|
|
site.
|
|
</para>
|
|
<para>
|
|
For a frequently used site you need definitely
|
|
more RAM (consider 2GB as a good start),
|
|
and possibly more CPU, which could also mean,
|
|
that you need to go for a SMP system.
|
|
</para>
|
|
<para>
|
|
You also want to consider a fast disk subsystem.
|
|
Operations on the CVS repository require a fast
|
|
disk subsystem (RAID is greatly advised). A SCSI
|
|
controller that has a cache of its own can also
|
|
speed up things, since most of these services incur a
|
|
very large number of small modifications to the disk.
|
|
</para>
|
|
<para>You can also experiment with enlarging the portion
|
|
of system memory which is used for the filesystem buffer cache.
|
|
This will also help to reduce the quantity of disk access. This
|
|
can be done with the BUFCACHEPERCENT kernel option. The default is
|
|
to use 5% of system memory.
|
|
</para>
|
|
</sect2>
|
|
<sect2 id="mirror-services">
|
|
<title>Services to offer</title>
|
|
<para>
|
|
Every mirror site is required to have a set of core services
|
|
available. In addition to these basic services, which mirrors are
|
|
required to provide, there is a number of optional services that
|
|
server administrators may choose to offer. This section explains
|
|
which services you can provide and how to go about implementing them.
|
|
</para>
|
|
<sect3 id="mirror-serv-ftp">
|
|
<title>FTP (required for FTP fileset)</title>
|
|
<para>
|
|
This is one of the most basic services, and
|
|
it is required for each mirror, offering public
|
|
FTP distributions. FTP access must be
|
|
anonymous, and no upload/download ratios
|
|
are allowed (a ridiculous thing anyway).
|
|
Upload capability is not required (and <emphasis>must</emphasis>
|
|
never be allowed for the FreeBSD file space).
|
|
Also the FreeBSD archive should be available under
|
|
the path <filename>/pub/FreeBSD</filename>.
|
|
</para>
|
|
<para>
|
|
There is a lot of software available which
|
|
can be set up to allow anonymous FTP
|
|
(in alphabetical order).
|
|
<itemizedlist>
|
|
<listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd
|
|
can be used. Be sure to read &man.ftpd.8;.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename role="package">ftp/ncftpd</filename>: A commercial package,
|
|
free for educational use.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename role="package">ftp/oftpd</filename>: An ftpd designed with
|
|
security as a main focus.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename role="package">ftp/proftpd</filename>: A modular and very flexible ftpd.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename role="package">ftp/pure-ftpd</filename>: Another ftpd developed with
|
|
security in mind.</para>
|
|
</listitem>
|
|
<listitem><para><filename role="package">ftp/twoftpd</filename>: As above.</para></listitem>
|
|
<listitem><para><filename role="package">ftp/vsftpd</filename>: The <quote>very secure</quote> ftpd.</para></listitem>
|
|
<listitem>
|
|
<para><filename role="package">ftp/wu-ftpd</filename>: The ftpd from Washington
|
|
University. It has become infamous, because of the huge
|
|
amount of security issues that have been found in it.
|
|
If you do choose to use this software be sure to
|
|
keep it up to date.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
|
|
<application>wu-ftpd</application> and maybe <application>ncftpd</application>
|
|
are among the most commonly ones.
|
|
The others do not have a large userbase among mirror sites.
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-serv-rsync">
|
|
<title>RSYNC (optional for FTP fileset)</title>
|
|
<para>
|
|
<application>Rsync</application> is often also offered for convenience, for the
|
|
contents of the FTP area of FreeBSD. The
|
|
protocol is different from FTP in many ways, and
|
|
overall, it can be stated, that it is much more
|
|
bandwidth friendly, as only differences between files
|
|
are transferred, not whole files.
|
|
<application>Rsync</application> does require significant amount of memory for
|
|
each instance. The size depends on the size of
|
|
the synced module in terms of number of directories and
|
|
files. <application>Rsync</application> can use <command>rsh</command> and
|
|
<command>ssh</command> (now default) as a transport,
|
|
or use it's own protocol for stand-alone access
|
|
(this is the preferred method for public rsync servers).
|
|
Authentication, connection limits, and other restrictions
|
|
may be applied. There is just one software package
|
|
available:
|
|
<itemizedlist>
|
|
<listitem><para><filename role="package">net/rsync</filename></para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-serv-http">
|
|
<title>HTTP (required for webpages, optional for FTP fileset)</title>
|
|
<para>
|
|
If you want to offer the FreeBSD webpages, you need
|
|
to install a webserver a.k.a. <application>httpd</application>.
|
|
You may optionally offer the FTP fileset via HTTP.
|
|
The choice of Webserver software is left up to the mirror administrator.
|
|
Some of the most popular choices are:
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><filename role="package">www/apache13</filename>:
|
|
Apache is the most widely deployed Webserver on the Internet. It
|
|
is used extensively by the FreeBSD Project. You may also
|
|
wish to use the next generation of the Apache Webserver,
|
|
available in the ports collection as <filename
|
|
role="package">www/apache2</filename>.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><filename role="package">www/thttpd</filename>:
|
|
If you are going to be serving a lot amount of static content
|
|
you may find that using an application such as tHttpd is more
|
|
efficient than Apache. It is optimized for excellent performance
|
|
on FreeBSD.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><filename role="package">www/boa</filename>:
|
|
Boa is another alternative to tHttpd and Apache. It should
|
|
provide considerably better performance than Apache for purely
|
|
static content. It does not, at the time of writing, contain the
|
|
same set of optimizations for FreeBSD that are found in
|
|
tHttpd.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-serv-cvsup">
|
|
<title>CVSup (desired for CVS repository)</title>
|
|
<para>
|
|
<application>CVSup</application> is a very efficient way of distributing files.
|
|
It works similar as rsync, but was specially designed for
|
|
the use with CVS repositories. If you want to offer the
|
|
FreeBSD CVS repository, you really want to consider
|
|
offering it via <application>CVSup</application>. Still it is possible to offer
|
|
the CVS repository via <application>AnonCVS</application>, FTP,
|
|
<application>Rsync</application> or HTTP, but
|
|
people would benefit much more from <application>CVSup</application> access.
|
|
<application>CVSup</application> was developed by &a.jdp;.
|
|
It is a bit tricky to install on non-FreeBSD platforms,
|
|
since it is written in Modula-3 and therefore requires
|
|
a Modula-3 environment. John Polstra has built a
|
|
stripped down version of M3, that is sufficient to
|
|
run <application>CVSup</application>, and can be installed much easier.
|
|
See <ulink url="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</ulink>
|
|
for details. Related ports are:
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><filename role="package">net/cvsup</filename>: The native CVSup port (client and server)
|
|
which requires <filename role="package">lang/ezm3</filename> now.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename role="package">net/cvsup-mirror</filename>: The CVSup mirror kit, which requires
|
|
<filename role="package">net/cvsup</filename>, and configures it mirror-ready. Some
|
|
site administrators may want a different setup, though.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
There are a few more like <filename role="package">net/cvsupit</filename> and
|
|
<filename role="package">net/cvsup-without-gui</filename> you might want to have
|
|
a look at. If you prefer a static binary package, take a look
|
|
<ulink url="http://people.freebsd.org/~jdp/s1g/">here</ulink>.
|
|
This page stil refers to the S1G bug, that was present
|
|
in <application>CVSup</application>. Maybe
|
|
John will setup a generic download-site to get
|
|
static binaries for various platforms.
|
|
</para>
|
|
<para>
|
|
It is possible to use CVSup to offer
|
|
any kind of fileset, not just CVS repositories,
|
|
but configuration can be complex.
|
|
CVSup is known to eat some CPU on the server as on the
|
|
client, since it needs to compare lots of files.
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-anoncvs">
|
|
<title>AnonCVS (optional for CVS repository)</title>
|
|
<para>
|
|
If you have the CVS repository, you may want to offer
|
|
anonymous CVS access. A short warning first:
|
|
There is not that much demand for it,
|
|
and it requires some experience and you need to know,
|
|
what you are doing.
|
|
</para>
|
|
<para>
|
|
Generally there are two ways, how
|
|
to access a CVS repository remotely: via
|
|
<emphasis>pserver</emphasis> or via <command>ssh</command>
|
|
(we don't consider <command>rsh</command>).
|
|
For anonymous access, <emphasis>pserver</emphasis> is
|
|
very well suited, but some still offer <command>ssh</command>
|
|
access as well. There is a custom crafted
|
|
<ulink url="ftp://ftp.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</ulink>
|
|
in the CVS repository, to be used as a login-shell for the
|
|
anonymous ssh account. It does a chroot, and therefore
|
|
requires the CVS repository to be available under the
|
|
anonymous user's home-directory, which may not be possible
|
|
for all sites. If you just offer <emphasis>pserver</emphasis>
|
|
this restriction does not apply, but you may run with
|
|
more security risks. You don't need to install any special
|
|
software, since &man.cvs.1; comes with
|
|
FreeBSD. You need to enable access via <command>inetd</command>,
|
|
so add an entry into your <filename>/etc/inetd.conf</filename>
|
|
like this:
|
|
<programlisting>
|
|
cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
|
|
</programlisting>
|
|
See the manpage for details of the options. See also the cvs <emphasis>info</emphasis>
|
|
page, about additional ways to make sure, access is read-only.
|
|
It is advisable, that you create an unprivileged account,
|
|
preferably called <username>anoncvs</username>.
|
|
Also you need to create a file <filename>passwd</filename>
|
|
in your <filename>/home/ncvs/CVSROOT</filename> and assign a
|
|
CVS password (empty or <literal>anoncvs</literal>) to that user.
|
|
The directory <filename>/anoncvstmp</filename> is a special
|
|
purpose memory based filesystem. It is not required but
|
|
advised, since &man.cvs.1; creates a shadow directory
|
|
structure in your <filename>/tmp</filename> which is
|
|
not used after the operation, but slows things
|
|
dramatically, if real disk operations are required.
|
|
Here is an excerpt from <filename>/etc/fstab</filename>,
|
|
how to set up such a MFS:
|
|
<programlisting>
|
|
/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid,nodev 0 0
|
|
</programlisting>
|
|
This is (of course) tuned a lot, and was suggested by &a.jdp;.
|
|
</para>
|
|
</sect3>
|
|
</sect2>
|
|
</sect1>
|
|
<sect1 id="mirror-howto">
|
|
<title>How to mirror FreeBSD</title>
|
|
<para>
|
|
Ok now, you know the requirements, and how to offer
|
|
the services, but not how to get it. :-)
|
|
This section explains how to actually mirror
|
|
the various parts of FreeBSD, what tools to use,
|
|
and where to mirror from.
|
|
</para>
|
|
<sect2 id="mirror-ftp">
|
|
<title>FTP</title>
|
|
<para>
|
|
The FTP area is the largest amount of data, that
|
|
needs to be mirrored. It includes the <emphasis>distributions
|
|
sets</emphasis>, required for network installation, the
|
|
<emphasis>branches</emphasis>, that are actually snapshots
|
|
of checked-out source trees, the <emphasis>ISO Images</emphasis>
|
|
to write CD-Roms with the installation distribution,
|
|
a live filesystem, and lots of packages, the ports tree,
|
|
distfiles and a huge amount of packages. All of course
|
|
for various FreeBSD versions, and <emphasis>i386</emphasis>
|
|
and <emphasis>alpha</emphasis> architecture.
|
|
</para>
|
|
<sect3 id="mirror-ftp-ftp">
|
|
<title>With FTP mirror</title>
|
|
<para>
|
|
You can use a <application>FTP mirror</application>
|
|
program, to get the files. There are a lot around, and
|
|
widely used, like:
|
|
<itemizedlist>
|
|
<listitem><para><filename role="package">ftp/mirror</filename></para></listitem>
|
|
<listitem><para><filename role="package">ftp/ftpmirror</filename></para></listitem>
|
|
<listitem><para><filename role="package">ftp/emirror</filename></para></listitem>
|
|
<listitem><para><filename role="package">ftp/spegla</filename></para></listitem>
|
|
<listitem><para><filename role="package">ftp/omi</filename></para></listitem>
|
|
<listitem><para>some even use <filename role="package">ftp/wget</filename></para></listitem>
|
|
</itemizedlist>
|
|
|
|
<filename role="package">ftp/mirror</filename> was very popular, but seemed
|
|
to have some drawbacks, as it is written in &man.perl.1;,
|
|
and did have real problems on mirroring large
|
|
directories like a FreeBSD site. There are rumors, that
|
|
the current version has fixed this, by allowing
|
|
to specify a different algorithm for comparing
|
|
the directory structure.
|
|
</para>
|
|
<para>
|
|
In general FTP is not really good for mirroring, since it transfers
|
|
each whole file, if it has changed, and does
|
|
not create a single data stream, that will benefit from
|
|
a large TCP congestion window.
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-ftp-rsync">
|
|
<title>With RSYNC</title>
|
|
<para>
|
|
A better way, to mirror the FTP area is <application>rsync</application>.
|
|
You can install the port <filename role="package">net/rsync</filename> and then use
|
|
rsync to sync with your upstream host.
|
|
<application>rsync</application> is already mentioned
|
|
in <xref linkend="mirror-serv-rsync">.
|
|
Since <application>rsync</application> access is not
|
|
required, your preferred upstream site may not allow it.
|
|
Since it is quite common, though, chances are small, that
|
|
you cannot use it. You can always consider using an
|
|
upstream server, that offers it, just for the benefits
|
|
of rsync.
|
|
<note>
|
|
<para>
|
|
Since the number of <application>rsync</application>
|
|
clients will have a significant impact on the server
|
|
machine, most admins impose limitations on their
|
|
server. For a mirror, you should ask the site maintainer
|
|
you are syncing from, about their policy, and maybe
|
|
an exception for your host (since you are a mirror).
|
|
</para>
|
|
</note>
|
|
A command line to mirror FreeBSD could look like that:
|
|
<screen>&prompt.user; <userinput>rsync -vaz --delete ftp4.de.freebsd.org::FreeBSD/ /pub/FreeBSD/</userinput>
|
|
</screen>
|
|
Consult the documentation for <application>rsync</application>,
|
|
which is also available at
|
|
<ulink url="http://rsync.samba.org/">http://rsync.samba.org/</ulink>
|
|
about the various options to be used with rsync.
|
|
Also you might
|
|
want to set up a script framework, that calls such a command
|
|
via &man.cron.8;.
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-ftp-cvsup">
|
|
<title>With CVSup</title>
|
|
<para>
|
|
A few sites, including the one-and-only <hostid>ftp-master.freebsd.org</hostid>
|
|
even offer <application>CVSup</application> to mirror the contents of
|
|
the FTP space. You need to install a <application>cvsup</application>
|
|
client, preferably from the port: <filename role="package">net/cvsup</filename>.
|
|
(Also reread <xref linkend="mirror-serv-cvsup">.)
|
|
A sample supfile, suitable for <hostid>ftp-master.freebsd.org</hostid>
|
|
looks like this:
|
|
<programlisting>
|
|
#
|
|
# FreeBSD archive supfile from master server
|
|
#
|
|
*default host=ftp-master.FreeBSD.org
|
|
*default base=/usr
|
|
*default prefix=/pub
|
|
#*default release=all
|
|
*default delete use-rel-suffix
|
|
*default umask=002
|
|
|
|
# If your network link is a T1 or faster, comment out the following line.
|
|
#*default compress
|
|
|
|
FreeBSD-archive release=all preserve
|
|
</programlisting>
|
|
|
|
It seems <application>CVSup</application> would be the best
|
|
way to mirror the archive, in terms of efficiency, but
|
|
it is only available from few sites. In fact I just know
|
|
<hostid>ftp-master.freebsd.org</hostid> for sure.
|
|
<note id="mirror-cvsup-s-option">
|
|
<para>
|
|
Please have look at the <application>CVSup</application> documentation
|
|
like &man.cvsup.1; and consider using the <option>-s</option>
|
|
option, as it can reduce the amount of work to be done
|
|
a lot.
|
|
</para>
|
|
</note>
|
|
</para>
|
|
</sect3>
|
|
</sect2>
|
|
<sect2 id="mirror-cvs">
|
|
<title>Mirroring the CVS repository</title>
|
|
<para>
|
|
Again you have various possibilities, but the most
|
|
recommended one, is to use <link linkend="mirror-cvs-cvsup">CVSup</link>.
|
|
</para>
|
|
<sect3 id="mirror-cvs-cvsup">
|
|
<title>Using CVSup</title>
|
|
<para>
|
|
<application>CVSup</application> was already described to some
|
|
detail in <xref linkend="mirror-serv-cvsup"> and <xref linkend="mirror-ftp-cvsup">.
|
|
</para>
|
|
<para>
|
|
Here we just describe an example to set up the <filename>supfile</filename>:
|
|
<programlisting>
|
|
#
|
|
# FreeBSD CVS supfile from master server
|
|
#
|
|
*default host=cvsup-master.FreeBSD.org
|
|
*default base=/usr
|
|
*default prefix=/pub/FreeBSD/development/FreeBSD-CVS
|
|
*default release=cvs
|
|
*default delete use-rel-suffix
|
|
*default umask=002
|
|
|
|
# If your network link is a T1 or faster, comment out the following line.
|
|
#*default compress
|
|
|
|
cvs-all
|
|
</programlisting>
|
|
|
|
You should also have a look at <filename>/usr/share/examples/cvsup</filename>
|
|
</para>
|
|
<note>
|
|
<para>
|
|
Please don't forget to consider the hint,
|
|
mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
|
|
above.
|
|
</para>
|
|
</note>
|
|
</sect3>
|
|
<sect3 id="mirror-cvs-other">
|
|
<title>Using other methods</title>
|
|
<para>
|
|
Using other methods than <application>CVSup</application> is
|
|
generally not recommended. We describe them in short here
|
|
anyway. Since most sites offer the CVS repository as
|
|
part of the FTP fileset under the path
|
|
<filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
|
|
the following methods could be used.
|
|
<itemizedlist>
|
|
<listitem><para><application>FTP</application></para></listitem>
|
|
<listitem><para><application>RSYNC</application></para></listitem>
|
|
<listitem><para>maybe even <application>HTTP</application></para></listitem>
|
|
</itemizedlist>
|
|
|
|
If you find a site, that supports it, you could use
|
|
<filename role="package">net/sup</filename>, but it is inferior to <application>CVSup</application>
|
|
and it's deficiencies caused John Polstra to develop
|
|
<application>CVSup</application> in the first place, so
|
|
it is clearly not recommended.
|
|
|
|
<important>
|
|
<para>
|
|
You can <emphasis>NOT</emphasis> use AnonCVS to
|
|
mirror the CVS repository, since CVS does not allow
|
|
you to access the repository itself, but only checked
|
|
out versions of the modules.
|
|
</para>
|
|
</important>
|
|
</para>
|
|
</sect3>
|
|
</sect2>
|
|
<sect2 id="mirror-www">
|
|
<title>Mirroring the WWW pages</title>
|
|
<para>
|
|
The best way is, to check out the <emphasis>www</emphasis>
|
|
distribution from CVS. If you have a local mirror of the
|
|
CVS repository, it is probably as easy as:
|
|
<screen>&prompt.user; <userinput>cvs -d /home/ncvs co www</userinput></screen>
|
|
and a <emphasis>cronjob</emphasis>, that calls <command>cvs up -d -P</command>
|
|
on a regular basis, maybe just after your repository was updated.
|
|
Of course, the files need to remain in a directory, available
|
|
for public WWW access. The installation and configuration of a
|
|
webserver is not discussed here.
|
|
</para>
|
|
<para>
|
|
If you don't have a local repository, you can use
|
|
<application>CVSup</application> to maintain an <quote>up to date copy</quote>
|
|
of the www pages. A sample supfile can be found in
|
|
<filename>/usr/share/examples/cvsup/www-supfile</filename> and
|
|
could look like this:
|
|
<programlisting>
|
|
#
|
|
# WWW module supfile for FreeBSD
|
|
#
|
|
*default host=cvsup3.de.FreeBSD.org
|
|
*default base=/usr
|
|
*default prefix=/usr/local
|
|
*default release=cvs tag=.
|
|
*default delete use-rel-suffix
|
|
|
|
# If your network link is a T1 or faster, comment out the following line.
|
|
*default compress
|
|
|
|
# This collection retrieves the www/ tree of the FreeBSD repository
|
|
www
|
|
</programlisting>
|
|
</para>
|
|
<para>
|
|
Using <filename role="package">ftp/wget</filename> or other web-mirror tools, is
|
|
probably not recommended.
|
|
</para>
|
|
<sect3 id="mirror-www-doc">
|
|
<title>Mirroring the FreeBSD documentation</title>
|
|
<para>
|
|
As the documentation is referenced a lot from the
|
|
webpages, it is recommended, that you mirror the
|
|
FreeBSD documentation as well. However, this is not
|
|
so trivial as the www-pages alone.
|
|
</para>
|
|
<para>
|
|
First of all, you should get the doc sources,
|
|
again preferably via <application>CVSup</application>.
|
|
Here is a corresponding sample supfile:
|
|
<programlisting>
|
|
#
|
|
# FreeBSD documentation supfile
|
|
#
|
|
*default host=cvsup3.de.FreeBSD.org
|
|
*default base=/usr
|
|
*default prefix=/usr/share
|
|
*default release=cvs tag=.
|
|
*default delete use-rel-suffix
|
|
|
|
# If your network link is a T1 or faster, comment out the following line.
|
|
#*default compress
|
|
|
|
# This will retrieve the entire doc branch of the FreeBSD repository.
|
|
# This includes the handbook, FAQ, and translations thereof.
|
|
doc-all
|
|
</programlisting>
|
|
</para>
|
|
<para>
|
|
Then you need to install a couple of ports.
|
|
You are lucky, that there is a meta-port:
|
|
<filename role="package">textproc/docproj</filename> to do the work
|
|
for you. You need to setup some
|
|
environment variables, like
|
|
<literal>SGML_CATALOG_FILES</literal>,
|
|
also have a look at your <filename>/etc/make.conf</filename>
|
|
(copy <filename>/etc/defaults/make.conf</filename> if
|
|
you don't have one), and look at the
|
|
<literal>DOC_LANG</literal> variable.
|
|
Now you are probably ready to run <command>make</command>
|
|
in you doc directory (<filename>/usr/share/doc</filename>
|
|
by default) and build the documentation.
|
|
Again you need to make it accessible for your webserver
|
|
and make sure, the links point to the right location.
|
|
<important>
|
|
<para>
|
|
The building of the documentation, as well as lots
|
|
of side issues, is documented itself in:
|
|
<ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/">fdp-primer</ulink>.
|
|
Please read this piece of documentation, especially if you
|
|
have problems, building the documentation.
|
|
</para>
|
|
</important>
|
|
<note>
|
|
<para>
|
|
XXX MAYBE THIS CAN BE LINKED FROM WITHIN - NOT USING AN ABSOLUTE URL XXX
|
|
</para>
|
|
</note>
|
|
</para>
|
|
</sect3>
|
|
</sect2>
|
|
<sect2 id="mirror-how-often">
|
|
<title>How often should I mirror?</title>
|
|
<para>
|
|
Every mirror should be updated on a regular
|
|
basis. You will certainly need some script
|
|
framework for it, that will be called by
|
|
&man.cron.8;. Since nearly every admin
|
|
does this his own way, we cannot give
|
|
specific instructions. It could work
|
|
like this:
|
|
</para>
|
|
<procedure>
|
|
<step>
|
|
<para>
|
|
Put the command to run your mirroring application
|
|
in a script. Use of a plain <command>/bin/sh</command>
|
|
script is recommended.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Add some output redirections, so diagnostic
|
|
messages are logged to a file.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Test, if your script works. Check the logs.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Use &man.crontab.1; to add the script the
|
|
appropriate user's &man.crontab.5;.
|
|
</para>
|
|
</step>
|
|
</procedure>
|
|
<para>
|
|
Here are some recommended schedules:
|
|
<itemizedlist>
|
|
<listitem><para>FTP fileset: daily</para></listitem>
|
|
<listitem><para>CVS repository: daily to hourly</para></listitem>
|
|
<listitem><para>WWW pages: daily</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
<sect1 id="mirror-where">
|
|
<title>Where to mirror from</title>
|
|
<para>
|
|
This is an important issue. So this section will
|
|
spend some effort to explain the backgrounds.
|
|
</para>
|
|
<sect2 id="mirror-where-organization">
|
|
<title>A few words about the organization</title>
|
|
<para>
|
|
Mirrors are organized by country. All
|
|
official mirrors have a DNS entry of the form
|
|
<hostid>ftpN.CC.freebsd.org</hostid>. With
|
|
<emphasis>CC</emphasis> (i.e. country code) being the
|
|
<emphasis>top level domain</emphasis>
|
|
of the country, where this mirror is located;
|
|
and <emphasis>N</emphasis> is a number,
|
|
telling that the host would be the <emphasis>Nth</emphasis>
|
|
mirror in that country.
|
|
(Same applies to <hostid>cvsupN.CC.freebsd.org</hostid>,
|
|
<hostid>wwwN.CC.freebsd.org</hostid>, etc.)
|
|
There are mirrors with no <emphasis>CC</emphasis> part.
|
|
These are usually located in the US, but don't need to.
|
|
<hostid>ftp.freebsd.org</hostid> is currently
|
|
located in Denmark and just another mirror
|
|
(i.e. it is <emphasis>NO</emphasis> master site).
|
|
</para>
|
|
<para>
|
|
Additionally there exists a hierarchy of mirrors, which
|
|
is described terms of <emphasis>tiers</emphasis>.
|
|
The master sites are not referred to, but can be
|
|
described as <emphasis>Tier-0</emphasis>. Mirrors
|
|
that mirror from these sites can be considered
|
|
<emphasis>Tier-1</emphasis>, mirrors of <emphasis>Tier-1</emphasis>-mirrors,
|
|
are <emphasis>Tier-2</emphasis>, etc.
|
|
Official sites are encouraged to be of a low <emphasis>tier</emphasis>,
|
|
but the lower the tier, the higher the requirements in
|
|
terms as described in <xref linkend="mirror-requirements">.
|
|
Also access to low-tier-mirrors may be restricted, and
|
|
access to master sites is definitely restricted.
|
|
The <emphasis>tier</emphasis>-hierarchy is not reflected
|
|
by DNS and generally not documented anywhere, except
|
|
for the master sites. However, official mirrors with low numbers,
|
|
like 1-4, are usually <emphasis>Tier-1</emphasis>
|
|
(this is just a rough hint, and there's no rule).
|
|
</para>
|
|
</sect2>
|
|
<sect2 id="mirror-where-where">
|
|
<title>Ok, but where should I get the stuff now?</title>
|
|
<para>
|
|
The short answer is: from the
|
|
site, that is closest to you in internet terms, or gives you
|
|
the fastest access.
|
|
</para>
|
|
<sect3 id="mirror-where-simple">
|
|
<title>I just want to mirror from somewhere!</title>
|
|
<para>
|
|
If you have no special intentions or
|
|
requirements, the statement in <xref linkend="mirror-where-where">
|
|
applies. This means:
|
|
</para>
|
|
<procedure>
|
|
<step>
|
|
<para>
|
|
Look at available mirrors in your country.
|
|
The <ulink url="http://www.freebsdmirrors.org/FBSDsites.php3">FreeBSD
|
|
Mirror Database</ulink> can help you with this.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Check roughly those, which provide fastest access
|
|
(number of hops, round-trip-times)
|
|
and offer the services you intend to
|
|
use (like <application>rsync</application>
|
|
or <application>CVSup</application>).
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Contact the admins of your chosen site, stating your
|
|
request, and asking about their terms and
|
|
policies.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Setup your mirror as described above.
|
|
</para>
|
|
</step>
|
|
</procedure>
|
|
</sect3>
|
|
<sect3 id="mirror-where-official">
|
|
<title>I'm an official mirror, what is the right site for me?</title>
|
|
<para>
|
|
In general the description in <xref linkend="mirror-where-simple">
|
|
still applies. Of course you may want to put some
|
|
weight on the fact, that your upstream should be of
|
|
a low tier.
|
|
There are some other considerations about <emphasis>official</emphasis>
|
|
mirrors, that are described in <xref linkend="mirror-official">.
|
|
</para>
|
|
</sect3>
|
|
<sect3 id="mirror-where-master">
|
|
<title>I want to access the master sites!</title>
|
|
<para>
|
|
If you have good reasons, and good prerequisites,
|
|
you may want and get access to one of the
|
|
master sites. Access to these sites is
|
|
generally restricted, and there are special policies
|
|
for access. If you are already an <emphasis>official</emphasis>
|
|
mirror, this certainly helps you getting access.
|
|
In any other case make sure your country really needs another mirror.
|
|
If it already has three or more, ask the &a.hubs; first.
|
|
</para>
|
|
<para>
|
|
There are just two master sites, one for the FTP fileset and
|
|
one for the CVS repository (the webpages and docs are
|
|
obtained from CVS, so there is no need for it).
|
|
</para>
|
|
<sect4 id="mirror-where-master-ftp">
|
|
<title>ftp-master.freebsd.org</title>
|
|
<para>
|
|
This is the master site for the FTP fileset.
|
|
</para>
|
|
<para>
|
|
<hostid>ftp-master.FreeBSD.org</hostid> provides
|
|
<application>rsync</application> and <application>CVSup</application>
|
|
access, rather in addition to ftp protocol.
|
|
Refer to <xref linkend="mirror-ftp-rsync"> and
|
|
<xref linkend="mirror-ftp-cvsup"> how to access
|
|
via these protocols.
|
|
</para>
|
|
<para>
|
|
Mirrors should be encouraged to also allow <application>rsync</application>
|
|
access for the FTP contents, since they are
|
|
<emphasis>Tier-1</emphasis>-mirrors.
|
|
</para>
|
|
<para>
|
|
To get access to <hostid>ftp-master.FreeBSD.org</hostid>,
|
|
you need to contact &a.peter;.
|
|
</para>
|
|
</sect4>
|
|
<sect4 id="mirror-where-master-cvsup">
|
|
<title>cvsup-master.freebsd.org</title>
|
|
<para>
|
|
This is the master site for the CVS repository.
|
|
</para>
|
|
<para>
|
|
<hostid>cvsup-master.FreeBSD.org</hostid> provides
|
|
<application>CVSup</application> access only.
|
|
See <xref linkend="mirror-cvs-cvsup"> for details.
|
|
</para>
|
|
<para>
|
|
To get access, you need to contact &a.jdp;.
|
|
Make sure you read
|
|
<ulink url="http://people.freebsd.org/~jdp/cvsup-access/">FreeBSD CVSup Access Policy</ulink>
|
|
first!
|
|
</para>
|
|
<para>
|
|
Set up the required authentication by following
|
|
<ulink url="http://people.freebsd.org/~jdp/cvpasswd/">these
|
|
instructions</ulink>. Make sure you specify the server as
|
|
<hostid>freefall.freebsd.org</hostid> on the <command>cvpasswd</command>
|
|
command line, as described in this document,
|
|
even when you are contacting
|
|
<hostid>cvsup-master.freebsd.org</hostid>
|
|
</para>
|
|
</sect4>
|
|
</sect3>
|
|
</sect2>
|
|
</sect1>
|
|
<sect1 id="mirror-official">
|
|
<title>Official Mirrors</title>
|
|
<para>
|
|
Official mirrors are mirrors that
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
a) have a <hostid>freebsd.org</hostid> DNS entry
|
|
(usually a CNAME).
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
b) are listed as an official mirror in the FreeBSD
|
|
documentation (like handbook).
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
So far to distinguish official mirrors.
|
|
Official mirrors are not necessarily <emphasis>Tier-1</emphasis>-mirrors.
|
|
However you probably won't find a <emphasis>Tier-1</emphasis>-mirror,
|
|
that is not also official.
|
|
</para>
|
|
<sect2 id="mirror-official-requirements">
|
|
<title>Special Requirements for official (tier-1) mirrors</title>
|
|
<para>
|
|
It is not so easy to state requirements for all
|
|
official mirrors, since the project is sort of
|
|
tolerant here. It is more easy to say,
|
|
what <emphasis>official tier-1 mirrors</emphasis>
|
|
are required to. All other official mirrors
|
|
can consider this a big <emphasis>should</emphasis>.
|
|
<note>
|
|
<para>
|
|
The following applies mainly to the FTP fileset,
|
|
since a CVS repository should always be mirrored
|
|
completely, and the webpages are a case of
|
|
its own.
|
|
</para>
|
|
</note>
|
|
</para>
|
|
<para>
|
|
Tier-1 mirrors are required to:
|
|
<itemizedlist>
|
|
<listitem><para>carry the complete fileset</para></listitem>
|
|
<listitem><para>allow access to other mirror sites</para></listitem>
|
|
<listitem><para>provide <application>FTP</application> and
|
|
<application>RSYNC</application> access</para></listitem>
|
|
</itemizedlist>
|
|
|
|
Furthermore, admins should be subscribed to the &a.hubs;.
|
|
See <ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/eresources.html#ERESOURCES-MAIL">this link</ulink> for details, how to subscribe.
|
|
</para>
|
|
<important>
|
|
<para>It is <emphasis>very</emphasis> important for a hub administrator, especially
|
|
Tier-1 hub admins, to check the
|
|
<ulink url="http://www.freebsd.org/releases/4.6R/schedule.html">release schedule</ulink>
|
|
for the next FreeBSD release. This is important because it will tell you when the
|
|
next release is scheduled
|
|
to come out, and thus giving you time to prepare for the big spike of traffic which follows it.
|
|
</para>
|
|
<para>
|
|
It is also eminent that hub administrators try to keep their mirrors as up-to-date as
|
|
possible (again, even more crucial for Tier-1 mirrors). If Mirror1 doesn't update for a
|
|
while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins
|
|
a downward spiral... Keep your mirrors up to date!
|
|
</para>
|
|
</important>
|
|
</sect2>
|
|
<sect2 id="mirror-official-become">
|
|
<title>How to become official then?</title>
|
|
<para>
|
|
An interesting question, especially, since the state
|
|
of being official comes with some benefits, like a much
|
|
higher bill from your ISP, as more people will be using
|
|
your site. Also it may be a key requirement, to get access
|
|
to a master site.
|
|
</para>
|
|
<para>
|
|
Before applying, please consider (again) if
|
|
another official mirror is really needed for
|
|
your region. Ask on the &a.hubs;, if in doubt.
|
|
</para>
|
|
<para>Ok, here is how to do it:</para>
|
|
<procedure>
|
|
<step>
|
|
<para>
|
|
Get the mirror running in first place (maybe not
|
|
using a master site, yet).
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
<ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/eresources.html#ERESOURCES-MAIL">Subscribe</ulink> to the &a.hubs;.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
If everything works so far, contact the DNS admin, responsible
|
|
for your region/country, and ask for a DNS entry for your
|
|
site. The admin should able to be contacted via
|
|
<email>hostmaster@cc.freebsd.org</email>, which
|
|
<emphasis>cc</emphasis> being your country code/TLD
|
|
again. Your DNS entry will look like described
|
|
in <xref linkend="mirror-where-organization">.
|
|
</para>
|
|
<para>
|
|
If there is no subdomain delegated, yet, for your
|
|
country, you probably need to contact
|
|
<email>hostmaster@freebsd.org</email>,
|
|
however, you can try the &a.hubs; first.
|
|
</para>
|
|
</step>
|
|
<step>
|
|
<para>
|
|
Then you can ask the &a.doc; or the &a.hubs;
|
|
to add your mirror site to the mirror list in the
|
|
<ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook">FreeBSD
|
|
Handbook</ulink>. Make sure you tell them the email address,
|
|
to list as the maintainer of the site.
|
|
</para>
|
|
</step>
|
|
</procedure>
|
|
<para>This is it.</para>
|
|
</sect2>
|
|
</sect1>
|
|
<sect1 id="mirror-statpages">
|
|
<title>Some statistics from mirror sites</title>
|
|
<para>
|
|
Here are links to the stat pages of your favorite mirrors
|
|
(a.k.a. the only ones who feel like providing stats).
|
|
</para>
|
|
<sect2 id="mirror-statpagesftp">
|
|
<title>FTP site statistics</title>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>ftp2.FreeBSD.org - <email>grisha@ispol.com</email> -
|
|
<ulink url="http://people.freebsd.org/~logo/ftp2/">(Bandwidth)</ulink>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>ftp.is.FreeBSD.org - <email>hostmaster@is.FreeBSD.org</email> -
|
|
<ulink url="http://www.rhnet.is/status/draupnir-total/draupnir-total.html">
|
|
(Bandwidth)</ulink> <ulink url="http://www.rhnet.is/status/ftp/ftp-notendur.html">(FTP
|
|
processes)</ulink> <ulink url="http://www.rhnet.is/status/ftp/http-notendur.html">(HTTP processes)
|
|
</ulink>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>ftp.cz.FreeBSD.org - <email>cejkar@fit.vutbr.cz</email> -
|
|
<ulink url="http://www.cz.FreeBSD.org/stats/mrtg/net.html">(Bandwidth)</ulink>
|
|
<ulink url="http://www.freebsd.cz/stats/mrtg/ftpd.html">(FTP processes)</ulink>
|
|
<ulink url="http://www.freebsd.cz/stats/mrtg/rsyncd.html">(Rsync processes)</ulink>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>ftp4.de.FreeBSD.org - <email>dl@leo.org</email> -
|
|
<ulink url="http://admin.leo.org/mrtg/ftpusers/ftpusers.html">(FTP users)</ulink>
|
|
<ulink url="http://admin.leo.org/mrtg/rsync/rsync.html">(RSYNC users)</ulink>
|
|
<ulink url="http://admin.leo.org/mrtg/traffic/atleo5/traffic_atleo5.html">(Bandwidth)</ulink>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
<sect2 id="mirror-statpagescvsup">
|
|
<title>CVSup site stats</title>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>cvsup5.FreeBSD.org - <email>staff@blackened.com</email> - <ulink
|
|
url="http://cvsup.freebsd.blackened.com/cvsup.freebsd.html">(CVSup processes)</ulink></para>
|
|
<listitem>
|
|
<para>cvsup[23456].jp.FreeBSD.org - <email>kuriyama@FreeBSD.org</email> - <ulink
|
|
url="http://home.jp.freebsd.org/stats/mrtg/cvsup/">(CVSup processes)</ulink></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>cvsup.cz.FreeBSD.org - <email>cejkar@fit.vutbr.cz</email> -
|
|
<ulink url="http://www.freebsd.cz/stats/mrtg/cvsupd.html">(CVSup processes)</ulink></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>[cvsup3|anoncvs].de.FreeBSD.org - <email>dl@leo.org</email> -
|
|
<ulink url="http://admin.leo.org/mrtg/services/cvsup/cvsup.html">(CVSup processes)</ulink></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</sect2>
|
|
</sect1>
|
|
</article>
|
|
|
|
<!--
|
|
Local Variables:
|
|
mode: sgml
|
|
sgml-indent-data: t
|
|
sgml-omittag: nil
|
|
sgml-always-quote-attributes: t
|
|
End:
|
|
-->
|