<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook V4.1-Based Extension//EN" [
<!ENTITY % man PUBLIC "-//FreeBSD//ENTITIES DocBook Manual Page Entities//EN">
%man;
<!ENTITY % authors PUBLIC "-//FreeBSD//ENTITIES DocBook Author Entities//EN">
%authors;
<!ENTITY % mailing-lists PUBLIC "-//FreeBSD//ENTITIES DocBook Mailing List Entities//EN">
%mailing-lists;
<!ENTITY % not.published "IGNORE">
]>

<article>
  <articleinfo>
    <title>Mirroring FreeBSD</title>
    <pubdate>$FreeBSD$</pubdate>
    <authorgroup>
      <author>
        <firstname>Jun</firstname>
        <surname>Kuriyama</surname>
        <affiliation>
          <address><email>kuriyama@FreeBSD.org</email></address>
        </affiliation>
      </author>
      <author>
        <firstname>Valentino</firstname>  
        <surname>Vaschetto</surname>
        <affiliation>
          <address><email>logo@FreeBSD.org</email></address>                             
        </affiliation>
      </author>
      <author>
        <firstname>Daniel</firstname>
        <surname>Lang</surname>
        <affiliation>
          <address><email>dl@leo.org</email></address>
        </affiliation>
      </author>
    </authorgroup>
    <abstract>
      <para>An in-progress article on how to mirror FreeBSD, aimed at
        hub administrators</para>
    </abstract>
  </articleinfo>

  <sect1 id="mirror-requirements">
    <title>Requirements for FreeBSD mirrors</title>
    <sect2 id="mirror-diskspace">
      <title>Disk Space</title>
      <para>
        Disk space is one of the most important requirements.
        Depending on the set of releases, architectures,
        and degree of completeness you want to mirror, a huge
        amount of disk space may be consumed. Also keep in mind,
        that <emphasis>official</emphasis> mirrors are probably required to be
        complete. The CVS repository and the web pages should
        always be mirrored completely. Also note, that the
        numbers stated here, are reflecting the current
        state (at 4.5-RELEASE). Further development and
        releases will only increase the required amount.
        Also make sure, to keep some (ca. 10-20%) extra space
        around, just to be sure.
        Here are some approximate figures:
      </para>
      <itemizedlist>
        <listitem><para>Full FTP Distribution: 60 GB</para></listitem>
        <listitem><para>CVS repository: 1.4 GB</para></listitem>
        <listitem><para>CTM deltas: 1.5 GB</para></listitem>
        <listitem><para>Webpages: 150 MB</para></listitem>
      </itemizedlist>
    </sect2>
    <sect2 id="mirror-bandwidth">
      <title>Network Connection/Bandwidth</title>
      <para>
        Of course, you need to be connected to the internet.
        The required bandwidth depends on your intended use
        of the mirror. If you just want to mirror some
        parts of FreeBSD for local use at your site/intranet,
        the demand may be much smaller, than if you want to
        make the files publicly available or even if you intend
        to become an official mirror. We can only give rough
        estimates here:
      </para>
      <itemizedlist>
         <listitem><para>Local site, no public access: basically no minimum,
           but &lt; 2 Mbps could make syncing a pain.</para></listitem>
         <listitem><para>Unofficial public site: 34 Mbps is probably a good start.</para></listitem>
         <listitem><para>Official site: &gt; 100 Mbps is recommended, also your host
           should be connected as close as possible to your border router.</para></listitem>
      </itemizedlist>
    </sect2>
    <sect2 id="mirror-system">
      <title>System Requirements, CPU, RAM</title>
      <para>
        This also depends on the expected amount of clients,
        which is determined by the servers policy. It is
        also affected by the types of services you want to offer.
        Plain FTP or HTTP services may not require a huge
        amount of resources. Watch out, if you provide 
        CVSup, rsync or even AnonCVS. This can have a huge
        impact on CPU and memory requirements. Especially
        rsync is considered a memory hog, and CVSup does 
        indeed consume some CPU. For AnonCVS it might
        be a nice idea to set up a memory resident filesystem (MFS) of at least
        300 MB, so you need to take this into account
        for your memory requirements. The following
        are just examples to give you a very rough hint.
      </para>
      <para>
        For a moderately visited site, that offers
        <application>rsync</application>, you might
        consider a current CPU with around 800Mhz - 1 GHz,
        and at least 512MB RAM. This is probably the
        minimum you want for an <emphasis>official</emphasis>
        site.
      </para>
      <para>
        For a frequently used site you need definitely
        more RAM (consider 2GB as a good start),
        and possibly more CPU, which could also mean,
        that you need to go for a SMP system.
      </para>
      <para>
        You also want to consider a fast disk subsystem.
        Operations on the CVS repository require a fast
        disk subsystem (RAID is greatly advised). A SCSI
        controller that has a cache of its own can also
        speed up things, since most of these services incur a
        very large number of small modifications to the disk.
      </para>
      <para>You can also experiment with enlarging the portion
        of system memory which is used for the filesystem buffer cache.
        This will also help to reduce the quantity of disk access. This
        can be done with the BUFCACHEPERCENT kernel option. The default is
        to use 5% of system memory.
      </para>
    </sect2>
    <sect2 id="mirror-services">
      <title>Services to offer</title>
      <para>
        Every mirror site is required to have a set of core services
        available. In addition to these basic services, which mirrors are
        required to provide, there is a number of optional services that
        server administrators may choose to offer.  This section explains
        which services you can provide and how to go about implementing them.
      </para>
      <sect3 id="mirror-serv-ftp">
        <title>FTP (required for FTP fileset)</title>
        <para>
          This is one of the most basic services, and
          it is required for each mirror, offering public
          FTP distributions. FTP access must be
          anonymous, and no upload/download ratios
          are allowed (a ridiculous thing anyway).
          Upload capability is not required (and <emphasis>must</emphasis>
          never be allowed for the FreeBSD file space).
          Also the FreeBSD archive should be available under
          the path <filename>/pub/FreeBSD</filename>.
        </para>
        <para>
          There is a lot of software available which
          can be set up to allow anonymous FTP
          (in alphabetical order).
          <itemizedlist>
            <listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd
              can be used. Be sure to read &man.ftpd.8;.</para>
            </listitem>
            <listitem>
              <para><filename role="package">ftp/ncftpd</filename>: A commercial package,
              free for educational use.</para>
            </listitem>
            <listitem>
              <para><filename role="package">ftp/oftpd</filename>: An ftpd designed with
              security as a main focus.</para>
            </listitem>
            <listitem>
              <para><filename role="package">ftp/proftpd</filename>: A modular and very flexible ftpd.</para>
            </listitem>
            <listitem>
              <para><filename role="package">ftp/pure-ftpd</filename>: Another ftpd developed with
                security in mind.</para>
            </listitem>
            <listitem><para><filename role="package">ftp/twoftpd</filename>: As above.</para></listitem>
            <listitem><para><filename role="package">ftp/vsftpd</filename>: The <quote>very secure</quote> ftpd.</para></listitem>
            <listitem>
              <para><filename role="package">ftp/wu-ftpd</filename>: The ftpd from Washington
                University. It has become infamous, because of the huge
                amount of security issues that have been found in it.
                If you do choose to use this software be sure to
                keep it up to date.
              </para>
            </listitem>
          </itemizedlist>
          FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
          <application>wu-ftpd</application> and maybe <application>ncftpd</application>
          are among the most commonly ones.
          The others do not have a large userbase among mirror sites.
        </para>
      </sect3>
      <sect3 id="mirror-serv-rsync">
        <title>RSYNC (optional for FTP fileset)</title>
        <para>
          <application>Rsync</application> is often also offered for convenience, for the
          contents of the FTP area of FreeBSD. The 
          protocol is different from FTP in many ways, and
          overall, it can be stated, that it is much more
          bandwidth friendly, as only differences between files
          are transferred, not whole files. 
          <application>Rsync</application> does require significant amount of memory for
          each instance. The size depends on the size of
          the synced module in terms of number of directories and 
          files. <application>Rsync</application> can use <command>rsh</command> and
          <command>ssh</command> (now default) as a transport,
          or use it's own protocol for stand-alone access
          (this is the preferred method for public rsync servers).
          Authentication, connection limits, and other restrictions
          may be applied. There is just one software package
          available:
          <itemizedlist>
            <listitem><para><filename role="package">net/rsync</filename></para></listitem>
          </itemizedlist>
        </para>
      </sect3>
      <sect3 id="mirror-serv-http">
        <title>HTTP (required for webpages, optional for FTP fileset)</title>
        <para>
          If you want to offer the FreeBSD webpages, you need
          to install a webserver a.k.a <application>httpd</application>.
          You may optionally offer the FTP fileset via HTTP.
          The choice of Webserver software is left up to the mirror administrator.
          Some of the most popular choices are:

          <itemizedlist>
            <listitem>
              <para><filename role="package">www/apache13</filename>:
                Apache is the most widely deployed Webserver on the Internet. It
                is used extensively by the FreeBSD Project. You may also
                wish to use the next generation of the Apache Webserver,
                available in the ports collection as <filename
                role="package">www/apache2</filename>.</para>
            </listitem>

            <listitem>
              <para><filename role="package">www/thttpd</filename>:
                If you are going to be serving a lot amount of static content
                you may find that using an application such as tHttpd is more
                efficient than Apache. It is optimized for excellent performance
                on FreeBSD.</para>
            </listitem>

            <listitem>
              <para><filename role="package">www/boa</filename>:
                Boa is another alternative to tHttpd and Apache. It should
                provide considerably better performance than Apache for purely
                static content. It does not, at the time of writing, contain the
                same set of optimizations for FreeBSD that are found in
                tHttpd.</para>
            </listitem>
          </itemizedlist>
        </para>
      </sect3>
      <sect3 id="mirror-serv-cvsup">
        <title>CVSup (desired for CVS repository)</title>
        <para>
          <application>CVSup</application> is a very efficient way of distributing files.
          It works similar as rsync, but was specially designed for
          the use with CVS repositories. If you want to offer the
          FreeBSD CVS repository, you really want to consider
          offering it via <application>CVSup</application>. Still it is possible to offer
          the CVS repository via <application>AnonCVS</application>, FTP,
          <application>Rsync</application> or HTTP, but
          people would benefit much more from <application>CVSup</application> access. 
          <application>CVSup</application> was developed by &a.jdp;.
          It is a bit tricky to install on non-FreeBSD platforms, 
          since it is written in Modula-3 and therefore requires
          a Modula-3 environment. John Polstra has built a 
          stripped down version of M3, that is sufficient to
          run <application>CVSup</application>, and can be installed much easier. 
          See <ulink url="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</ulink>
          for details. Related ports are:

          <itemizedlist>
            <listitem>
              <para><filename role="package">net/cvsup</filename>: The native CVSup port (client and server)
                which requires <filename role="package">lang/ezm3</filename> now.</para>
            </listitem>
            <listitem>
              <para><filename role="package">net/cvsup-mirror</filename>: The CVSup mirror kit, which requires
                 <filename role="package">net/cvsup</filename>, and configures it mirror-ready. Some
                 site administrators may want a different setup, though.
              </para>
            </listitem>
          </itemizedlist>

          There are a few more like <filename role="package">net/cvsupit</filename> and 
          <filename role="package">net/cvsup-without-gui</filename> you might want to have
          a look at. If you prefer a static binary package, take a look 
          <ulink url="http://people.freebsd.org/~jdp/s1g/">here</ulink>.
          This page stil refers to the S1G bug, that was present
          in <application>CVSup</application>. Maybe
          John will setup a generic download-site to get
          static binaries for various platforms.
        </para>
        <para> 
          It is possible to use CVSup to offer
          any kind of fileset, not just CVS repositories, 
          but configuration can be complex.
          CVSup is known to eat some CPU on the server as on the 
          client, since it needs to compare lots of files.
        </para>
      </sect3>
      <sect3 id="mirror-anoncvs">
        <title>AnonCVS (optional for CVS repository)</title>
        <para>
          If you have the CVS repository, you may want to offer
          anonymous CVS access. A short warning first:
          There is not that much demand for it,
          and it requires some experience and you need to know,
          what you are doing.
        </para>
        <para>
          Generally there are two ways, how
          to access a CVS repository remotely: via
          <emphasis>pserver</emphasis> or via <command>ssh</command>
          (we don't consider <command>rsh</command>).
          For anonymous access, <emphasis>pserver</emphasis> is
          very well suited, but some still offer <command>ssh</command>
          access as well. There is a custom crafted
          <ulink url="ftp://ftp.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</ulink>
          in the CVS repository, to be used as a login-shell for the
          anonymous ssh account. It does a chroot, and therefore
          requires the CVS repository to be available under the
          anonymous user's home-directory, which may not be possible
          for all sites. If you just offer <emphasis>pserver</emphasis>
          this restriction does not apply, but you may run with
          more security risks. You don't need to install any special
          software, since &man.cvs.1; comes with
          FreeBSD. You need to enable access via <command>inetd</command>,
          so add an entry into your <filename>/etc/inetd.conf</filename>
          like this:
          <programlisting>
cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
          </programlisting>
          See the manpage for details of the options. See also the cvs <emphasis>info</emphasis>
          page, about additional ways to make sure, access is read-only.
          It is advisable, that you create an unprivileged account, 
          preferably called <username>anoncvs</username>.
          Also you need to create a file <filename>passwd</filename>
          in your <filename>/home/ncvs/CVSROOT</filename> and assign a 
          CVS password (empty or <literal>anoncvs</literal>) to that user.
          The directory <filename>/anoncvstmp</filename> is a special
          purpose memory based filesystem. It is not required but
          advised, since &man.cvs.1; creates a shadow directory 
          structure in your <filename>/tmp</filename> which is
          not used after the operation, but slows things
          dramatically, if real disk operations are required.
          Here is an excerpt from <filename>/etc/fstab</filename>,
          how to set up such a MFS:
          <programlisting>
/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid,nodev 0 0
          </programlisting>
          This is (of course) tuned a lot, and was suggested by &a.jdp;.
        </para>
      </sect3>
   </sect2>
  </sect1>
  <sect1 id="mirror-howto">
    <title>How to mirror FreeBSD</title>
    <para>
      Ok now, you know the requirements, and how to offer
      the services, but not how to get it. :-)
      This section explains how to actually mirror
      the various parts of FreeBSD, what tools to use,
      and where to mirror from.
    </para>
    <sect2 id="mirror-ftp">
    <title>FTP</title>
      <para>
        The FTP area is the largest amount of data, that
        needs to be mirrored. It includes the <emphasis>distributions
        sets</emphasis>, required for network installation, the 
        <emphasis>branches</emphasis>, that are actually snapshots
        of checked-out source trees, the <emphasis>ISO Images</emphasis>
        to write CD-Roms with the installation distribution,
        a live filesystem, and lots of packages, the ports tree,
        distfiles and a huge amount of packages. All of course
        for various FreeBSD versions, and <emphasis>i386</emphasis>
        and <emphasis>alpha</emphasis> architecture.
      </para>
      <sect3 id="mirror-ftp-ftp">
        <title>With FTP mirror</title>
        <para>
          You can use a <application>FTP mirror</application>
          program, to get the files. There are a lot around, and
          widely used, like:
          <itemizedlist>
            <listitem><para><filename role="package">ftp/mirror</filename></para></listitem>
            <listitem><para><filename role="package">ftp/ftpmirror</filename></para></listitem>
            <listitem><para><filename role="package">ftp/emirror</filename></para></listitem>
            <listitem><para><filename role="package">ftp/spegla</filename></para></listitem>
            <listitem><para><filename role="package">ftp/omi</filename></para></listitem>
            <listitem><para>some even use <filename role="package">ftp/wget</filename></para></listitem>
          </itemizedlist>

          <filename role="package">ftp/mirror</filename> was very popular, but seemed
          to have some drawbacks, as it is written in &man.perl.1;,
          and did have real problems on mirroring large
          directories like a FreeBSD site. There are rumors, that
          the current version has fixed this, by allowing
          to specify a different algorithm for comparing
          the directory structure.
        </para>
        <para>
          In general FTP is not really good for mirroring, since it transfers
          each whole file, if it has changed, and does
          not create a single data stream, that will benefit from
          a large TCP congestion window. 
        </para>
      </sect3>
      <sect3 id="mirror-ftp-rsync">
        <title>With RSYNC</title>
        <para>
          A better way, to mirror the FTP area is <application>rsync</application>.
          You can install the port <filename role="package">net/rsync</filename> and then use
          rsync to sync with your upstream host. 
          <application>rsync</application> is already mentioned
          in <xref linkend="mirror-serv-rsync">.
          Since <application>rsync</application> access is not 
          required, your preferred upstream site may not allow it.
          Since it is quite common, though, chances are small, that
          you cannot use it. You can always consider using an
          upstream server, that offers it, just for the benefits
          of rsync. 
          <note>
            <para>
              Since the number of <application>rsync</application> 
              clients will have a significant impact on the server
              machine, most admins impose limitations on their
              server. For a mirror, you should ask the site maintainer
              you are syncing from, about their policy, and maybe
              an exception for your host (since you are a mirror).
            </para>
          </note>
          A command line to mirror FreeBSD could look like that:
          <screen>&prompt.user; <userinput>rsync -vaz --delete ftp4.de.freebsd.org::FreeBSD/ /pub/FreeBSD/</userinput>
          </screen>
          Consult the documentation for <application>rsync</application>,
          which is also available at
          <ulink url="http://rsync.samba.org/">http://rsync.samba.org/</ulink>
          about the various options to be used with rsync.
          Also you might
          want to set up a script framework, that calls such a command
          via &man.cron.8;.
        </para>
      </sect3>
      <sect3 id="mirror-ftp-cvsup">
        <title>With CVSup</title>
        <para>
          A few sites, including the one-and-only <hostid>ftp-master.freebsd.org</hostid>
          even offer <application>CVSup</application> to mirror the contents of 
          the FTP space. You need to install a <application>cvsup</application>
          client, preferably from the port: <filename role="package">net/cvsup</filename>.
          (Also reread <xref linkend="mirror-serv-cvsup">.)
          A sample supfile, suitable for <hostid>ftp-master.freebsd.org</hostid>
          looks like this:
          <programlisting>
          #
          # FreeBSD archive supfile from master server
          #
          *default host=ftp-master.FreeBSD.org
          *default base=/usr
          *default prefix=/pub
          #*default release=all
          *default delete use-rel-suffix
          *default umask=002

          # If your network link is a T1 or faster, comment out the following line.
          #*default compress

          FreeBSD-archive release=all preserve
          </programlisting>

          It seems <application>CVSup</application> would be the best
          way to mirror the archive, in terms of efficiency, but
          it is only available from few sites. In fact I just know
          <hostid>ftp-master.freebsd.org</hostid> for sure.
          <note id="mirror-cvsup-s-option">
            <para>
              Please have look at the <application>CVSup</application> documentation
              like &man.cvsup.1; and consider using the <option>-s</option>
              option, as it can reduce the amount of work to be done
              a lot.
            </para>
          </note>
        </para>
      </sect3>
    </sect2>
    <sect2 id="mirror-cvs">
      <title>Mirroring the CVS repository</title>
      <para>
        Again you have various possibilities, but the most
        recommended one, is to use <link linkend="mirror-cvs-cvsup">CVSup</link>.
      </para>
      <sect3 id="mirror-cvs-cvsup">
        <title>Using CVSup</title>
        <para>
          <application>CVSup</application> was already described to some
          detail in <xref linkend="mirror-serv-cvsup"> and <xref linkend="mirror-ftp-cvsup">.
        </para>
        <para>
          Here we just describe an example to set up the <filename>supfile</filename>:
          <programlisting>
          #
          # FreeBSD CVS supfile from master server
          #
          *default host=cvsup-master.FreeBSD.org
          *default base=/usr
          *default prefix=/pub/FreeBSD/development/FreeBSD-CVS
          *default release=cvs
          *default delete use-rel-suffix
          *default umask=002

          # If your network link is a T1 or faster, comment out the following line.
          #*default compress

          cvs-all
          </programlisting>

          You should also have a look at <filename>/usr/share/examples/cvsup</filename>
        </para>
        <note>
          <para>
            Please don't forget to consider the hint,
            mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
            above.
          </para>
        </note>
      </sect3>
      <sect3 id="mirror-cvs-other">
        <title>Using other methods</title>
        <para>
          Using other methods than <application>CVSup</application> is
          generally not recommended. We describe them in short here 
          anyway. Since most sites offer the CVS repository as
          part of the FTP fileset under the path
          <filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
          the following methods could be used.
          <itemizedlist>
            <listitem><para><application>FTP</application></para></listitem>
            <listitem><para><application>RSYNC</application></para></listitem>
            <listitem><para>maybe even <application>HTTP</application></para></listitem>
          </itemizedlist>

          If you find a site, that supports it, you could use
          <filename role="package">net/sup</filename>, but it is inferior to <application>CVSup</application>
          and it's deficiencies caused John Polstra to develop
          <application>CVSup</application> in the first place, so
          it is clearly not recommended.

          <important>
            <para>
              You can <emphasis>NOT</emphasis> use AnonCVS to 
              mirror the CVS repository, since CVS does not allow
              you to access the repository itself, but only checked
              out versions of the modules.
            </para>
          </important>
        </para>
      </sect3>
    </sect2>
    <sect2 id="mirror-www">
      <title>Mirroring the WWW pages</title>
      <para>
        The best way is, to check out the <emphasis>www</emphasis>
        distribution from CVS. If you have a local mirror of the
        CVS repository, it is probably as easy as:
        <screen>&prompt.user; <userinput>cvs -d /home/ncvs co www</userinput></screen>
        and a <emphasis>cronjob</emphasis>, that calls <command>cvs up -d -P</command>
        on a regular basis, maybe just after your repository was updated.
        Of course, the files need to remain in a directory, available
        for public WWW access. The installation and configuration of a 
        webserver is not discussed here.
      </para>
      <para>
        If you don't have a local repository, you can use 
        <application>CVSup</application> to maintain an <quote>up to date copy</quote>
        of the www pages. A sample supfile can be found in
        <filename>/usr/share/examples/cvsup/www-supfile</filename> and
        could look like this:
        <programlisting>
        #
        # WWW module supfile for FreeBSD
        #
        *default host=cvsup3.de.FreeBSD.org
        *default base=/usr
        *default prefix=/usr/local
        *default release=cvs tag=.
        *default delete use-rel-suffix

        # If your network link is a T1 or faster, comment out the following line.
        *default compress

        # This collection retrieves the www/ tree of the FreeBSD repository
        www
        </programlisting>
     </para>
     <para>
       Using <filename role="package">ftp/wget</filename> or other web-mirror tools, is
       probably not recommended.
     </para>
     <sect3 id="mirror-www-doc">
       <title>Mirroring the FreeBSD documentation</title>
       <para>
         As the documentation is referenced a lot from the 
         webpages, it is recommended, that you mirror the 
         FreeBSD documentation as well. However, this is not
         so trivial as the www-pages alone.
       </para>
       <para>
         First of all, you should get the doc sources,
         again preferably via <application>CVSup</application>.
         Here is a corresponding sample supfile:
         <programlisting>
         #
         # FreeBSD documentation supfile
         #
         *default host=cvsup3.de.FreeBSD.org
         *default base=/usr
         *default prefix=/usr/share
         *default release=cvs tag=.
         *default delete use-rel-suffix

         # If your network link is a T1 or faster, comment out the following line.
         #*default compress

         # This will retrieve the entire doc branch of the FreeBSD repository.
         # This includes the handbook, FAQ, and translations thereof.
         doc-all
         </programlisting>
       </para>
       <para>
         Then you need to install a couple of ports.
         You are lucky, that there is a meta-port:
         <filename role="package">textproc/docproj</filename> to do the work
         for you. You need to setup some 
         environment variables, like
         <literal>SGML_CATALOG_FILES</literal>,
         also have a look at your <filename>/etc/make.conf</filename>
         (copy <filename>/etc/defaults/make.conf</filename> if
         you don't have one), and look at the
         <literal>DOC_LANG</literal> variable.
         Now you are probably ready to run <command>make</command>
         in you doc directory (<filename>/usr/share/doc</filename>
         by default) and build the documentation.
         Again you need to make it accessible for your webserver
         and make sure, the links point to the right location.
         <important>
           <para>
             The building of the documentation, as well as lots
             of side issues, is documented itself in:
             <ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/">fdp-primer</ulink>.
             Please read this piece of documentation, especially if you
             have problems, building the documentation.
           </para>
         </important>
         <note>
           <para>
           XXX MAYBE THIS CAN BE LINKED FROM WITHIN - NOT USING AN ABSOLUTE URL XXX
           </para>
         </note>
       </para>
     </sect3>
   </sect2>
   <sect2 id="mirror-how-often">
     <title>How often should I mirror?</title>
     <para>
       Every mirror should be updated on a regular
       basis. You will certainly need some script
       framework for it, that will be called by 
       &man.cron.8;. Since nearly every admin
       does this his own way, we cannot give
       specific instructions. It could work
       like this:
     </para>
     <procedure>
       <step>
         <para>
           Put the command to run your mirroring application
           in a script. Use of a plain <command>/bin/sh</command>
           script is recommended.
         </para>
       </step>
       <step>
         <para>
           Add some output redirections, so diagnostic
           messages are logged to a file.
         </para>
       </step>
       <step>
         <para>
           Test, if your script works. Check the logs.
         </para>
       </step>
       <step>
         <para>
           Use &man.crontab.1; to add the script the
           appropriate user's &man.crontab.5;.
         </para>
       </step>
     </procedure>
     <para>
       Here are some recommended schedules:
       <itemizedlist>
         <listitem><para>FTP fileset: daily</para></listitem>
         <listitem><para>CVS repository: daily to hourly</para></listitem>
         <listitem><para>WWW pages: daily</para></listitem>
       </itemizedlist>
     </para>
   </sect2>
  </sect1>
  <sect1 id="mirror-where">
    <title>Where to mirror from</title>
    <para>
      This is an important issue. So this section will
      spend some effort to explain the backgrounds.
    </para>
    <sect2 id="mirror-where-organization">
      <title>A few words about the organization</title>
      <para>
        Mirrors are organized by country. All
        official mirrors have a DNS entry of the form
        <hostid>ftpN.CC.freebsd.org</hostid>. Whith 
        <emphasis>CC</emphasis> (i.e. country code) beeing the
        <emphasis>top level domain</emphasis>
        of the country, where this mirror is located;
        and <emphasis>N</emphasis> is a number,
        telling that the host would be the <emphasis>Nth</emphasis>
        mirror in that country.
        (Same applies to <hostid>cvsupN.CC.freebsd.org</hostid>,
        <hostid>wwwN.CC.freebsd.org</hostid>, etc.)
        There are mirrors with no <emphasis>CC</emphasis> part.
        These are usually located in the US, but don't need to.
        <hostid>ftp.freebsd.org</hostid> is currently
        located in Denmark and just another mirror
        (i.e. it is <emphasis>NO</emphasis> master site).
      </para>
      <para>
        Additionally there exists a hierarchy of mirrors, which
        is described terms of <emphasis>tiers</emphasis>. 
        The master sites are not referred to, but can be
        described as <emphasis>Tier-0</emphasis>. Mirrors
        that mirror from these sites can be considered
        <emphasis>Tier-1</emphasis>, mirrors of <emphasis>Tier-1</emphasis>-mirrors,
        are <emphasis>Tier-2</emphasis>, etc. 
        Official sites are encouraged to be of a low <emphasis>tier</emphasis>,
        but the lower the tier, the higher the requirements in 
        terms as described in <xref linkend="mirror-requirements">.
        Also access to low-tier-mirrors may be restricted, and
        access to master sites is definitely restricted.
        The <emphasis>tier</emphasis>-hierarchy is not reflected
        by DNS and generally not documented anywhere, except
        for the master sites. However, official mirrors with low numbers,
        like 1-4, are usually <emphasis>Tier-1</emphasis>
        (this is just a rough hint, and there's no rule).
      </para>
    </sect2>
    <sect2 id="mirror-where-where">
      <title>Ok, but where should I get the stuff now?</title>
      <para>
        The short answer is: from the
        site, that is closest to you in internet terms, or gives you
        the fastest access. 
      </para>
      <sect3 id="mirror-where-simple">
        <title>I just want to mirror from somewhere!</title>
        <para>
          If you have no special intentions or
          requirements, the statement in <xref linkend="mirror-where-where">
          applies.  This means:
        </para>
        <procedure>
          <step>
            <para>
              Look at available mirrors in your country.
              The <ulink url="http://www.freebsdmirrors.org/FBSDsites.php3">FreeBSD
              Mirror Database</ulink> can help you with this.
            </para>
          </step>
          <step>
            <para>
              Check roughly those, which provide fastest access
              (number of hops, round-trip-times)
              and offer the services you intend to
              use (like <application>rsync</application>
              or <application>CVSup</application>).
            </para>
          </step>
          <step>
            <para>
              Contact the admins of your chosen site, stating your
              request, and asking about their terms and
              policies.
            </para>
          </step>
          <step>
            <para>
              Setup your mirror as described above.
            </para>
          </step>
        </procedure>
      </sect3>
      <sect3 id="mirror-where-official">
        <title>I'm an official mirror, what is the right site for me?</title>
        <para>
          In general the description in <xref linkend="mirror-where-simple">
          still applies. Of course you may want to put some
          weight on the fact, that your upstream should be of
          a low tier.
          There are some other considerations about <emphasis>official</emphasis>
          mirrors, that are described in <xref linkend="mirror-official">.
        </para>
      </sect3>
      <sect3 id="mirror-where-master">
        <title>I want to access the master sites!</title>
        <para>
          If you have good reasons, and good prerequisites,
          you may want and get access to one of the 
          master sites. Access to these sites is 
          generally restricted, and there are special policies
          for access. If you are already an <emphasis>official</emphasis>
          mirror, this certainly helps you getting access.
          In any other case make sure your country really needs another mirror.
          If it already has three or more, ask the &a.hubs; first.
        </para>
        <para>
          There are just two master sites, one for the FTP fileset and
          one for the CVS repository (the webpages and docs are
          obtained from CVS, so there is no need for it).
        </para>
        <sect4 id="mirror-where-master-ftp">
          <title>ftp-master.freebsd.org</title>
          <para>
            This is the master site for the FTP fileset. 
          </para>
          <para>
            <hostid>ftp-master.FreeBSD.org</hostid> provides
            <application>rsync</application> and <application>CVSup</application>
            access, rather in addition to ftp protocol.
            Refer to <xref linkend="mirror-ftp-rsync"> and
            <xref linkend="mirror-ftp-cvsup"> how to access
            via these protocols.
          </para>
          <para>
            Mirrors should be encouraged to also allow <application>rsync</application>
            access for the FTP contents, since they are
            <emphasis>Tier-1</emphasis>-mirrors.
          </para>
          <para>
            To get access to <hostid>ftp-master.FreeBSD.org</hostid>,
            you need to contact &a.peter;.
          </para>
        </sect4>
        <sect4 id="mirror-where-master-cvsup">
          <title>cvsup-master.freebsd.org</title>
          <para>
            This is the master site for the CVS repository.
          </para>
          <para>
            <hostid>cvsup-master.FreeBSD.org</hostid> provides
            <application>CVSup</application> access only.
            See <xref linkend="mirror-cvs-cvsup"> for details.
          </para>
          <para>
            To get access, you need to contact &a.jdp;.
            Make sure you read
            <ulink url="http://people.freebsd.org/~jdp/cvsup-access/">FreeBSD CVSup Access Policy</ulink>
            first!
          </para>
          <para>
            Set up the required authentication by following
            <ulink url="http://people.freebsd.org/~jdp/cvpasswd/">these
            instructions</ulink>. Make sure you specify the server as
            <hostid>freefall.freebsd.org</hostid> on the <command>cvpasswd</command>
            command line, as described in this document,
            even when you are contacting
            <hostid>cvsup-master.freebsd.org</hostid>
          </para>
        </sect4>
      </sect3>
    </sect2>
  </sect1>
  <sect1 id="mirror-official">
    <title>Official Mirrors</title>
    <para>
      Official mirrors are mirrors that
      <itemizedlist>
        <listitem>
          <para>
            a) have a <hostid>freebsd.org</hostid> DNS entry
            (usually a CNAME).
          </para>
        </listitem>
        <listitem>
          <para>
            b) are listed as an official mirror in the FreeBSD
               documentation (like handbook).
          </para>
        </listitem>
      </itemizedlist>

      So far to distinguish official mirrors.
      Official mirrors are not necessarily <emphasis>Tier-1</emphasis>-mirrors.
      However you probably won't find a <emphasis>Tier-1</emphasis>-mirror,
      that is not also official.
    </para>
    <sect2 id="mirror-official-requirements">
      <title>Special Requirements for official (tier-1) mirrors</title>
      <para>
        It is not so easy to state requirements for all
        official mirrors, since the project is sort of
        tolerant here. It is more easy to say,
        what <emphasis>official tier-1 mirrors</emphasis>
        are required to. All other official mirrors
        can consider this a big <emphasis>should</emphasis>.
        <note>
          <para>
            The following applies mainly to the FTP fileset,
            since a CVS repository should always be mirrored 
            completely, and the webpages are a case of 
            its own.
          </para>
        </note>
      </para>
      <para>
        Tier-1 mirrors are required to:
        <itemizedlist>
          <listitem><para>to carry the complete fileset</para></listitem>
          <listitem><para>allow access to other mirror sites</para></listitem>
          <listitem><para>provide <application>FTP</application> and 
            <application>RSYNC</application> access</para></listitem>
        </itemizedlist>

        Furthermore, admins should be subscribed to the &a.hubs;.
        See <ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/eresources.html#ERESOURCES-MAIL">this link</ulink> for details, how to subscribe.
      </para>
      <important>
        <para>It is <emphasis>very</emphasis> important for a hub administrator, especially
          Tier-1 hub admins, to check the
          <ulink url="http://www.freebsd.org/releases/4.6R/schedule.html">release schedule</ulink>
          for the next FreeBSD release. This is important because it will tell you when the
          next release is scheduled
          to come out, and thus giving you time to prepare for the big spike of traffic which follows it.
        </para>
        <para>
          It is also eminent that hub administrators try to keep their mirrors as up-to-date as
          possible (again, even more crucial for Tier-1 mirrors). If Mirror1 doesn't update for a
          while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins
          a downward spiral... Keep your mirrors up to date!
        </para>
      </important>
    </sect2>
    <sect2 id="mirror-official-become">
      <title>How to become official then?</title>
      <para>
        An interesting question, especially, since the state
        of being official comes with some benefits, like a much
        higher bill from your ISP, as more people will be using
        your site. Also it may be a key requirement, to get access
        to a master site.
      </para>
      <para>
        Before applying, please consider (again) if
        another official mirror is really needed for
        your region. Ask on the &a.hubs;, if in doubt.
      </para>
      <para>Ok, here is how to do it:</para>
      <procedure>
        <step>
          <para>
            Get the mirror running in first place (maybe not
            using a master site, yet).
          </para>
        </step>
        <step>
          <para>
            <ulink url="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/eresources.html#ERESOURCES-MAIL">Subscribe</ulink> to the &a.hubs;.
          </para>
        </step>
        <step>
          <para>
            If everything works so far, contact the DNS admin, responsible
            for your region/country, and ask for a DNS entry for your
            site. The admin should able to be contacted via
            <email>hostmaster@cc.freebsd.org</email>, which 
            <emphasis>cc</emphasis> being your country code/TLD
            again. Your DNS entry will look like described
            in <xref linkend="mirror-where-organization">.
          </para>
          <para>
            If there is no subdomain delegated, yet, for your
            country, you probably need to contact
            <email>hostmaster@freebsd.org</email>,
            however, you can try the &a.hubs; first.
          </para>
        </step>
        <step>
          <para>
            Then you can ask the &a.doc; or the &a.hubs;  
            to add your mirror site to the mirror list in the
            <ulink url="http://www.freebsd.org/handbook">FreeBSD
            Handbook</ulink>. Make sure you tell them the email address,
            to list as the maintainer of the site.
          </para>
        </step> 
      </procedure>
      <para>This is it.</para>
    </sect2>
  </sect1>
  <sect1 id="mirror-statpages">
    <title>Some statistics from mirror sites</title>
    <para>
      Here are links to the stat pages of your favorite mirrors
      (a.k.a. the only ones who feel like providing stats).
    </para>
    <sect2 id="mirror-statpagesftp">
      <title>FTP site statistics</title>
      <itemizedlist>
        <listitem>
          <para>ftp2.FreeBSD.org - <email>grisha@ispol.com</email> -
            <ulink url="http://people.freebsd.org/~logo/ftp2/">(Bandwidth)</ulink>
          </para>
        </listitem>
        <listitem>
          <para>ftp.is.FreeBSD.org - <email>oli@isnic.is</email> - 
            <ulink url="http://www.rhnet.is/status/draupnir/draupnir.html">
            (Bandwidth)</ulink> <ulink url="http://www.rhnet.is/status/ftp/ftp-notendur.html">(FTP
            processes)</ulink> <ulink url="http://www.rhnet.is/status/ftp/http-notendur.html">(HTTP processes)
            </ulink>
          </para>
        </listitem>
        <listitem>
          <para>ftp.cz.FreeBSD.org - <email>cejkar@fit.vutbr.cz</email> -
            <ulink url="http://www.cz.FreeBSD.org/stats/mrtg/net.html">(Bandwidth)</ulink>
            <ulink url="http://www.freebsd.cz/stats/mrtg/ftpd.html">(FTP processes)</ulink>
            <ulink url="http://www.freebsd.cz/stats/mrtg/rsyncd.html">(Rsync processes)</ulink>
          </para>
        </listitem>
        <listitem>
          <para>ftp4.de.FreeBSD.org - <email>dl@leo.org</email> -
            <ulink url="http://admin.leo.org/mrtg/ftpusers/ftpusers.html">(FTP users)</ulink>
            <ulink url="http://admin.leo.org/mrtg/rsync/rsync.html">(RSYNC users)</ulink>
            <ulink url="http://admin.leo.org/mrtg/traffic/atleo5/traffic_atleo5.html">(Bandwidth)</ulink>
          </para>
        </listitem>
      </itemizedlist>
    </sect2>
    <sect2 id="mirror-statpagescvsup">
      <title>CVSup site stats</title>
      <itemizedlist>
        <listitem>
          <para>cvsup5.FreeBSD.org - <email>staff@blackened.com</email> - <ulink
            url="http://cvsup.freebsd.blackened.com/cvsup.freebsd.html">(CVSup processes)</ulink></para>
        <listitem>
          <para>cvsup[23456].jp.FreeBSD.org - <email>kuriyama@FreeBSD.org</email> - <ulink
            url="http://home.jp.freebsd.org/stats/mrtg/cvsup/">(CVSup processes)</ulink></para>
        </listitem>
        <listitem>
          <para>cvsup.cz.FreeBSD.org - <email>cejkar@fit.vutbr.cz</email> -
            <ulink url="http://www.freebsd.cz/stats/mrtg/cvsupd.html">(CVSup processes)</ulink></para>
        </listitem>
        <listitem>
          <para>[cvsup3|anoncvs].de.FreeBSD.org - <email>dl@leo.org</email> -
            <ulink url="http://admin.leo.org/mrtg/cvsup/cvsup.html">(CVSup processes)</ulink></para>
        </listitem>
      </itemizedlist>
    </sect2>
  </sect1>
</article>

<!-- 
     Local Variables:
     mode: sgml
     sgml-indent-data: t
     sgml-omittag: nil
     sgml-always-quote-attributes: t
     End:
-->
