Glen Barber
32324aabdf
Remove the restriction of indexing man.cgi.
...
Discussed with: emaste, simon
Sponsored by: The FreeBSD Foundation
2013-10-04 19:09:47 +00:00
Glen Barber
1d975503b8
Explicitly list the files within cgi/, since not all crawlers will
...
honor the following syntax:
Allow: /cgi/foo
Disallow: /cgi/
2013-10-04 18:57:43 +00:00
Glen Barber
449060a3f2
Remove a stray blank line from robots.txt.
...
Add a commented Crawl-delay of 10 seconds, which might be useful
after the commit that is to follow.
2013-10-04 18:55:55 +00:00
Eitan Adler
f80e6a070a
Due to the time sensitive nature of some of the testing I'm doing, readd /sites/downloads.sourceforge.net/ which is a path that exists on some mirrors outside of our control.
2013-06-01 10:19:14 +00:00
Glen Barber
39b20f2e5b
Sigh...
...
Remove a path that does not even exist.
2013-06-01 00:23:11 +00:00
Eitan Adler
2b9e11b84c
The folder blocked previously seems to be a special case of a broader problem which clusteradm@ is looking into.
...
However, /sites/downloads.sourceforge.net/ should be blocked as
- it is not our code
- some mirrors are serving malware from it which we don't want to be associated with us
2013-05-31 23:10:53 +00:00
Eitan Adler
183eca6912
There appear to be configuration errors causing an absurd amount of links to be generated.
...
For now, block access to search engines until the problem is properly resolved.
While here, add $FreeBSD$
2013-05-31 22:54:45 +00:00
Hiroki Sato
282a032540
- Remove junk directories.
...
- Repocopy from www/<lang> to head/<lang>/htdocs to eliminate duplicate
information in the www and the doc directory.
- Add various administration files to svnadmin.
Approved by: doceng (implicit)
2012-05-17 02:51:08 +00:00
