From f61d605eab3a996d304c62574a441c01d3c6cb2a Mon Sep 17 00:00:00 2001 From: Valentino Vaschetto Date: Mon, 22 Apr 2002 21:55:14 +0000 Subject: [PATCH] As per pr #37353: - Updates to hubs article. Submitted by: Daniel Lang PR: 37353 --- en_US.ISO8859-1/articles/hubs/article.sgml | 210 ++++++++++++++------- 1 file changed, 144 insertions(+), 66 deletions(-) diff --git a/en_US.ISO8859-1/articles/hubs/article.sgml b/en_US.ISO8859-1/articles/hubs/article.sgml index ef2e364d9f..20b4742d75 100644 --- a/en_US.ISO8859-1/articles/hubs/article.sgml +++ b/en_US.ISO8859-1/articles/hubs/article.sgml @@ -50,7 +50,7 @@ Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind, - that official mirrors are probably required to be + that official mirrors are probably required to be complete. The CVS repository and the web pages should always be mirrored completely. Also note, that the numbers stated here, are reflecting the current @@ -80,8 +80,9 @@ estimates here: - Local site, no public access: basically no minimum. - Unofficial public site: 34 Mbps is a good start. + Local site, no public access: basically no minimum, + but < 2 Mbps could make syncing a pain. + Unofficial public site: 34 Mbps is probably a good start. Official site: > 100 Mbps is recommended, also your host should be connected as close as possible to your border router. @@ -100,7 +101,22 @@ indeed consume some CPU. For AnonCVS it might be a nice idea to set up a memory resident filesystem (MFS) of at least 300 MB, so you need to take this into account - for your memory requirements. + for your memory requirements. The following + are just examples to give you a very rough hint. + + + For a moderately visited site, that offers + rsync, you might + consider a current CPU with around 800Mhz - 1 GHz, + and at least 512MB RAM. This is probably the + minimum you want for an official + site. + + + For a frequently used site you need definitely + more RAM (consider 2GB as a good start), + and possibly more CPU, which could also mean, + that you need to go for a SMP system. You also want to consider a fast disk subsystem. @@ -120,11 +136,11 @@ Services to offer - Every mirror site is required to have a set of core services - available. In addition to these basic services, which mirrors are - required to provide, there is a number of optional services that - server administrators may choose to offer. This section explains - which services you can provide and how to go about implementing them. + Every mirror site is required to have a set of core services + available. In addition to these basic services, which mirrors are + required to provide, there is a number of optional services that + server administrators may choose to offer. This section explains + which services you can provide and how to go about implementing them. FTP (required for FTP fileset) @@ -140,19 +156,19 @@ the path /pub/FreeBSD. - There is lot of software available which + There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order). - /usr/libexec/ftpd: FreeBSD's own ftpd, + /usr/libexec/ftpd: FreeBSD's own ftpd can be used. Be sure to read &man.ftpd.8;. - ftp/ncftpd: A commercial package, free for - educational use. + ftp/ncftpd: A commercial package, + free for educational use. - ftp/oftpd: an ftpd designed with + ftp/oftpd: An ftpd designed with security as a main focus. @@ -166,29 +182,32 @@ ftp/vsftpd: The very secure ftpd. ftp/wu-ftpd: The ftpd from Washington - University. It has become infamous, because of the huge - amount of security issues that have been found in it. - If you do choose to use this software be sure to - keep it up to date. + University. It has become infamous, because of the huge + amount of security issues that have been found in it. + If you do choose to use this software be sure to + keep it up to date. + - FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the - most commonly ones. The others do not have a large userbase among mirror sites. + FreeBSD's ftpd, proftpd, + wu-ftpd and maybe ncftpd + are among the most commonly ones. + The others do not have a large userbase among mirror sites. RSYNC (optional for FTP fileset) - Rsync is often also offered for convenience, for the + Rsync is often also offered for convenience, for the contents of the FTP area of FreeBSD. The protocol is different from FTP in many ways, and overall, it can be stated, that it is much more bandwidth friendly, as only differences between files are transferred, not whole files. - Rsync does require significant amount of memory for + Rsync does require significant amount of memory for each instance. The size depends on the size of the synced module in terms of number of directories and - files. Rsync can use rsh and + files. Rsync can use rsh and ssh (now default) as a transport, or use it's own protocol for stand-alone access (this is the preferred method for public rsync servers). @@ -204,7 +223,8 @@ HTTP (required for webpages, optional for FTP fileset) If you want to offer the FreeBSD webpages, you need - to install a webserver. You may optionally offer the FTP fileset via HTTP. + to install a webserver a.k.a httpd. + You may optionally offer the FTP fileset via HTTP. The choice of Webserver software is left up to the mirror administrator. Some of the most popular choices are: @@ -240,19 +260,20 @@ CVSup (desired for CVS repository) - CVSup is a very efficient way of distributing files. + CVSup is a very efficient way of distributing files. It works similar as rsync, but was specially designed for the use with CVS repositories. If you want to offer the FreeBSD CVS repository, you really want to consider - offering it via CVSup. Still it is possible to offer - the CVS repository via AnonCVS, FTP, Rsync or HTTP, but - people would not benefit from CVSup access. - CVSup was developed by &a.jdp;. + offering it via CVSup. Still it is possible to offer + the CVS repository via AnonCVS, FTP, + Rsync or HTTP, but + people would benefit much more from CVSup access. + CVSup was developed by &a.jdp;. It is a bit tricky to install on non-FreeBSD platforms, since it is written in Modula-3 and therefore requires a Modula-3 environment. John Polstra has built a stripped down version of M3, that is sufficient to - run CVSup, and can be installed much easier. + run CVSup, and can be installed much easier. See Ezm3 for details. Related ports are: @@ -285,29 +306,26 @@ CVSup is known to eat some CPU on the server as on the client, since it needs to compare lots of files. - - - Please have look at the CVSup - like &man.cvsup.1; and consider using the - option, as it can reduce the amount of work to be done - a lot. - - AnonCVS (optional for CVS repository) If you have the CVS repository, you may want to offer - anonymous CVS access. There is not that much demand for it, + anonymous CVS access. A short warning first: + There is not that much demand for it, and it requires some experience and you need to know, - what you are doing. Generally there are two ways, how + what you are doing. + + + Generally there are two ways, how to access a CVS repository remotely: via pserver or via ssh (we don't consider rsh). For anonymous access, pserver is very well suited, but some still offer ssh - access as well. There is some custom crafted wrapper - program around, to be used as a login-shell for the + access as well. There is a custom crafted + wrapper + in the CVS repository, to be used as a login-shell for the anonymous ssh account. It does a chroot, and therefore requires the CVS repository to be available under the anonymous user's home-directory, which may not be possible @@ -322,7 +340,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver See the manpage for details of the options. See also the cvs info - page, about how additional ways, to make sure, access is read-only. + page, about additional ways to make sure, access is read-only. It is advisable, that you create an unprivileged account, preferably called anoncvs. Also you need to create a file passwd @@ -383,7 +401,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all ftp/mirror was very popular, but seemed - to have some drawbacks, as its written in perl, + to have some drawbacks, as it is written in &man.perl.1;, and did have real problems on mirroring large directories like a FreeBSD site. There are rumors, that the current version has fixed this, by allowing @@ -406,8 +424,8 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all rsync is already mentioned in . Since rsync access is not - required, your preferred upstream site may not allow it, - but it already widely used, so chances are small, that + required, your preferred upstream site may not allow it. + Since it is quite common, though, chances are small, that you cannot use it. You can always consider using an upstream server, that offers it, just for the benefits of rsync. @@ -418,7 +436,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from, about their policy, and maybe - an exception for your host, since you are a mirror. + an exception for your host (since you are a mirror). A command line to mirror FreeBSD could look like that: @@ -464,6 +482,14 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all way to mirror the archive, in terms of efficiency, but it is only available from few sites. In fact I just know ftp-master.freebsd.org for sure. + + + Please have look at the CVSup documentation + like &man.cvsup.1; and consider using the + option, as it can reduce the amount of work to be done + a lot. + + @@ -500,6 +526,13 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all You should also have a look at /usr/share/examples/cvsup + + + Please don't forget to consider the hint, + mentioned in this note + above. + + Using other methods @@ -509,7 +542,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all anyway. Since most sites offer the CVS repository as part of the FTP fileset under the path /pub/FreeBSD/development/FreeBSD-CVS, - the following methods can of course be used. + the following methods could be used. FTP RSYNC @@ -524,7 +557,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all - AFAICT, you can NOT AnonCVS to + You can NOT use AnonCVS to mirror the CVS repository, since CVS does not allow you to access the repository itself, but only checked out versions of the modules. @@ -622,7 +655,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all The building of the documentation, as well as lots - of side issues is documented itself in: + of side issues, is documented itself in: fdp-primer. Please read this piece of documentation, especially if you have problems, building the documentation. @@ -636,6 +669,52 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all + + How often should I mirror? + + Every mirror should be updated on a regular + basis. You will certainly need some script + framework for it, that will be called by + &man.cron.8;. Since nearly every admin + does this his own way, we cannot give + specific instructions. It could work + like this: + + + + + Put the command to run your mirroring application + in a script. Use of a plain /bin/sh + script is recommended. + + + + + Add some output redirections, so diagnostic + messages are logged to a file. + + + + + Test, if your script works. Check the logs. + + + + + Use &man.crontab.1; to add the script the + appropriate user's &man.crontab.5;. + + + + + Here are some recommended schedules: + + FTP fileset: daily + CVS repository: daily to hourly + WWW pages: daily + + + Where to mirror from @@ -648,15 +727,15 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all Mirrors are organized by country. All official mirrors have a DNS entry of the form - ftpX.CC.freebsd.org. Whith - CC (i.e. country code) being the + ftpN.CC.freebsd.org. Whith + CC (i.e. country code) beeing the top level domain of the country, where this mirror is located; - and X is a number, - telling that the host would be the Xth + and N is a number, + telling that the host would be the Nth mirror in that country. - (Same applies to cvsupX.CC.freebsd.org, - wwwX.CC.freebsd.org, etc.) + (Same applies to cvsupN.CC.freebsd.org, + wwwN.CC.freebsd.org, etc.) There are mirrors with no CC part. These are usually located in the US, but don't need to. ftp.freebsd.org is currently @@ -749,13 +828,12 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all for access. If you are already an official mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. - If it already has three or more, ask - the &a.hubs; first. + If it already has three or more, ask the &a.hubs; first. - There are just two master sites, for the FTP fileset and - for the CVS repository (the webpages and docs are obtained from CVS, - so there is no need for it). + There are just two master sites, one for the FTP fileset and + one for the CVS repository (the webpages and docs are + obtained from CVS, so there is no need for it). ftp-master.freebsd.org @@ -800,7 +878,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all Set up the required authentication by following these instructions. Make sure you specify the server as - freefall.freebsd.org on the cvpasswd + freefall.freebsd.org on the cvpasswd command line, as described in this document, even when you are contacting cvsup-master.freebsd.org @@ -840,7 +918,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all official mirrors, since the project is sort of tolerant here. It is more easy to say, what official tier-1 mirrors - are required to. All other official mirrors, + are required to. All other official mirrors can consider this a big should. @@ -882,7 +960,7 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all How to become official then? - An interesting questions, especially, since the state + An interesting question, especially, since the state of being official comes with some benefits, like a much higher bill from your ISP, as more people will be using your site. Also it may be a key requirement, to get access @@ -925,8 +1003,8 @@ cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --all - Then you can ask the &a.doc;, or the &a.hubs; to add your - mirror site to the mirror list in the + Then you can ask the &a.doc; or the &a.hubs; + to add your mirror site to the mirror list in the FreeBSD Handbook. Make sure you tell them the email address, to list as the maintainer of the site.