Wolfram Schneider
50431e5205
disallow man.cgi tarball downloads
2019-05-04 21:11:19 +00:00
Edward Tomasz Napierala
7f15a538c5
Prepare for moving the Quarterly to Markdown. This includes removing
...
the monthly.cgi and updating various pieces of documentation. Notably
missing is the French translation.
Reviewed by: bcr
Approved by: bcr (mentor)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D17300
2018-09-26 15:00:07 +00:00
Eitan Adler
08cf89c19c
remove testing robots.txt entry: it didn't help :(
2015-04-03 19:11:54 +00:00
Gavin Atkinson
ab4ca5d942
Re-add /cgi/url.cgi to the disallowed list, this is now served by Varnish.
2014-07-23 11:38:56 +00:00
Gavin Atkinson
14f2f1e1dd
As of r45331, /cgi/pds.cgi and /cgi/url.cgi no longer exist.
2014-07-22 20:21:08 +00:00
Glen Barber
0a8db54c12
Re-enable indexing manual pages, jet is fixed.
...
Sponsored by: The FreeBSD Foundation
2013-12-29 16:05:35 +00:00
Glen Barber
9681847d47
Block man.cgi lookups until jet stops overheating.
...
Approved by: clusteradm (implicit)
Sponsored by: The FreeBSD Foundation
2013-12-29 04:23:48 +00:00
Glen Barber
32324aabdf
Remove the restriction of indexing man.cgi.
...
Discussed with: emaste, simon
Sponsored by: The FreeBSD Foundation
2013-10-04 19:09:47 +00:00
Glen Barber
1d975503b8
Explicitly list the files within cgi/, since not all crawlers will
...
honor the following syntax:
Allow: /cgi/foo
Disallow: /cgi/
2013-10-04 18:57:43 +00:00
Glen Barber
449060a3f2
Remove a stray blank line from robots.txt.
...
Add a commented Crawl-delay of 10 seconds, which might be useful
after the commit that is to follow.
2013-10-04 18:55:55 +00:00
Eitan Adler
f80e6a070a
Due to the time sensitive nature of some of the testing I'm doing, readd /sites/downloads.sourceforge.net/ which is a path that exists on some mirrors outside of our control.
2013-06-01 10:19:14 +00:00
Glen Barber
39b20f2e5b
Sigh...
...
Remove a path that does not even exist.
2013-06-01 00:23:11 +00:00
Eitan Adler
2b9e11b84c
The folder blocked previously seems to be a special case of a broader problem which clusteradm@ is looking into.
...
However, /sites/downloads.sourceforge.net/ should be blocked as
- it is not our code
- some mirrors are serving malware from it which we don't want to be associated with us
2013-05-31 23:10:53 +00:00
Eitan Adler
183eca6912
There appear to be configuration errors causing an absurd amount of links to be generated.
...
For now, block access to search engines until the problem is properly resolved.
While here, add $FreeBSD$
2013-05-31 22:54:45 +00:00
Hiroki Sato
282a032540
- Remove junk directories.
...
- Repocopy from www/<lang> to head/<lang>/htdocs to eliminate duplicate
information in the www and the doc directory.
- Add various administration files to svnadmin.
Approved by: doceng (implicit)
2012-05-17 02:51:08 +00:00
