[interchange-cvs] interchange - kwalsh modified dist/robots.cfg

[interchange-cvs at icdevgroup.org]

User:      kwalsh
Date:      2007-08-09 11:11:47 GMT
Modified:  dist     robots.cfg
Log:
    * Twiceler experimental web crawler.

    * Various Wikis, such as ShopWiki.

Revision  Changes    Path
2.6       +2 -0      interchange/dist/robots.cfg


rev 2.6, prev_rev 2.5
Index: robots.cfg
===================================================================
RCS file: /var/cvs/interchange/dist/robots.cfg,v
retrieving revision 2.5
retrieving revision 2.6
diff -u -r2.5 -r2.6
--- robots.cfg	11 Jul 2007 19:34:41 -0000	2.5
+++ robots.cfg	9 Aug 2007 11:11:47 -0000	2.6
@@ -123,6 +123,7 @@
     Toutatis,
     TurnitinBot,
     Tv*Merc,
+    Twiceler,
     urllib,
     VB Project,
     Valkyrie,
@@ -132,6 +133,7 @@
     wget,
     WhizBang,
     whowhere,
+    Wiki,
     WinInet,
     winona,
     Wire,