svnno****@sourc*****
svnno****@sourc*****
2008年 5月 8日 (木) 21:28:11 JST
スラドRevision: 631
http://svn.sourceforge.jp/cgi-bin/viewcvs.cgi?root=slashdotjp&view=rev&rev=631
Author: tach
Date: 2008-05-08 21:28:11 +0900 (Thu, 08 May 2008)
Log Message:
-----------
add some disallow
Modified Paths:
--------------
images/robots.txt
Property Changed:
----------------
images/robots.txt
-------------- next part --------------
Modified: images/robots.txt
===================================================================
--- images/robots.txt 2008-05-08 12:22:21 UTC (rev 630)
+++ images/robots.txt 2008-05-08 12:28:11 UTC (rev 631)
@@ -1,5 +1,5 @@
# robots.txt for slashdot.jp
-# $Id: robots.txt,v 1.5 2003/08/29 15:21:08 jamie Exp $
+# $Id$
#
# "Any empty [Disallow] value, indicates that all URLs can be retrieved.
# At least one Disallow field needs to be present in a record."
@@ -20,14 +20,35 @@
User-agent: Slurp
Crawl-delay: 100
-Disallow:
+Disallow: /firehose.pl
+Disallow: /submit.pl
+Disallow: /users.pl
+Disallow: /zoo.pl
+Disallow: firehose.pl
+Disallow: submit.pl
+Disallow: users.pl
+Disallow: zoo.pl
User-agent: Yahoo-NewsCrawler
-Disallow:
+Disallow: /firehose.pl
+Disallow: /submit.pl
+Disallow: /users.pl
+Disallow: /zoo.pl
+Disallow: firehose.pl
+Disallow: submit.pl
+Disallow: users.pl
+Disallow: zoo.pl
User-Agent: msnbot
Crawl-delay: 100
-Disallow:
+Disallow: /firehose.pl
+Disallow: /submit.pl
+Disallow: /users.pl
+Disallow: /zoo.pl
+Disallow: firehose.pl
+Disallow: submit.pl
+Disallow: users.pl
+Disallow: zoo.pl
User-agent: *
Crawl-delay: 100
Property changes on: images/robots.txt
___________________________________________________________________
Name: svn:keywords
+ Author Date Id Revision