[interchange-cvs] interchange - kwalsh modified dist/robots.cfg

interchange-cvs at icdevgroup.org interchange-cvs at icdevgroup.org
Sun Dec 23 13:38:19 EST 2007


User:      kwalsh
Date:      2007-12-23 18:38:18 GMT
Modified:  dist     robots.cfg
Log:
    * More robots recognised.

Revision  Changes    Path
2.10      +4 -0      interchange/dist/robots.cfg


rev 2.10, prev_rev 2.9
Index: robots.cfg
===================================================================
RCS file: /var/cvs/interchange/dist/robots.cfg,v
retrieving revision 2.9
retrieving revision 2.10
diff -u -r2.9 -r2.10
--- robots.cfg	27 Sep 2007 21:43:30 -0000	2.9
+++ robots.cfg	23 Dec 2007 18:38:18 -0000	2.10
@@ -13,6 +13,7 @@
     asterias,
     ATN_Worldwide,
     Atomz,
+    AurNet,
     Awasu,
     BackRub,
     bender,
@@ -52,6 +53,7 @@
     Gromit,
     Gulliver,
     H?m?h?kki,
+    heritrix,
     HTTrack,
     Harvest,
     holmes,
@@ -149,6 +151,7 @@
     xtreme,
     Yahoo,
     Yandex,
+    Zeus,
     ZyBorg,
 EOR
 
@@ -190,6 +193,7 @@
     *.lycos.com,
     *.pa-x.dec.com,
     *.phx.gbl,
+    *.search.live.com,
     add-url.altavista.com,
     msnbot.msn.com,
     westinghouse-rsl-com-usa.NorthRoyalton.cw.net,








More information about the interchange-cvs mailing list