os/doc
3
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Explicitly list the files within cgi/, since not all crawlers will

Browse source 
honor the following syntax:

	Allow:    /cgi/foo
	Disallow: /cgi/
This commit is contained in:
Glen Barber 2013-10-04 18:57:43 +00:00
parent 449060a3f2
commit 1d975503b8
Notes: svn2git 2020-12-08 03:00:23 +00:00 
svn path=/head/; revision=42831
1 changed files with 16 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
en_US.ISO8859-1/htdocs/robots.txt
View file

@ -2,6 +2,21 @@
User-agent: *
#Crawl-delay: 10
Disallow: /cgi/
Disallow: /cgi/confirm-code.cgi
Disallow: /cgi/cvsweb.cgi
Disallow: /cgi/dosendpr.cgi
Disallow: /cgi/getmsg.cgi
Disallow: /cgi/mailindex.cgi
Disallow: /cgi/man.cgi
Disallow: /cgi/mid.cgi
Disallow: /cgi/mirror.cgi
Disallow: /cgi/missing_handler.cgi
Disallow: /cgi/monthly.cgi
Disallow: /cgi/pds.cgi
Disallow: /cgi/ports.cgi
Disallow: /cgi/query-pr-summary.cgi
Disallow: /cgi/query-pr.cgi
Disallow: /cgi/search.cgi
Disallow: /cgi/url.cgi
Disallow: /statistic
Disallow: /sites/downloads.sourceforge.net/