svnno****@sourc*****
svnno****@sourc*****
2008年 5月 8日 (木) 21:22:02 JST
スラドRevision: 629
http://svn.sourceforge.jp/cgi-bin/viewcvs.cgi?root=slashdotjp&view=rev&rev=629
Author: tach
Date: 2008-05-08 21:22:02 +0900 (Thu, 08 May 2008)
Log Message:
-----------
add robots.txt
Added Paths:
-----------
images/robots.txt
-------------- next part --------------
Added: images/robots.txt
===================================================================
--- images/robots.txt (rev 0)
+++ images/robots.txt 2008-05-08 12:22:02 UTC (rev 629)
@@ -0,0 +1,64 @@
+# robots.txt for slashdot.jp
+# $Id: robots.txt,v 1.5 2003/08/29 15:21:08 jamie Exp $
+#
+# "Any empty [Disallow] value, indicates that all URLs can be retrieved.
+# At least one Disallow field needs to be present in a record."
+
+User-agent: Mediapartners-Google
+Disallow:
+
+User-agent: Googlebot
+Crawl-delay: 100
+Disallow: /firehose.pl
+Disallow: /submit.pl
+Disallow: /users.pl
+Disallow: /zoo.pl
+Disallow: firehose.pl
+Disallow: submit.pl
+Disallow: users.pl
+Disallow: zoo.pl
+
+User-agent: Slurp
+Crawl-delay: 100
+Disallow:
+
+User-agent: Yahoo-NewsCrawler
+Disallow:
+
+User-Agent: msnbot
+Crawl-delay: 100
+Disallow:
+
+User-agent: *
+Crawl-delay: 100
+Disallow: /authors.pl
+Disallow: /index.pl
+Disallow: /article.pl
+Disallow: /comments.pl
+Disallow: /firehose.pl
+Disallow: /journal.pl
+Disallow: /messages.pl
+Disallow: /metamod.pl
+Disallow: /users.pl
+Disallow: /search.pl
+Disallow: /submit.pl
+Disallow: /pollBooth.pl
+Disallow: /pubkey.pl
+Disallow: /topics.pl
+Disallow: /zoo.pl
+Disallow: authors.pl
+Disallow: index.pl
+Disallow: article.pl
+Disallow: comments.pl
+Disallow: firehose.pl
+Disallow: journal.pl
+Disallow: messages.pl
+Disallow: metamod.pl
+Disallow: users.pl
+Disallow: search.pl
+Disallow: submit.pl
+Disallow: pollBooth.pl
+Disallow: pubkey.pl
+Disallow: topics.pl
+Disallow: zoo.pl
+Disallow: /i