[interchange-cvs] interchange - kwalsh modified dist/robots.cfg
interchange-cvs at icdevgroup.org
interchange-cvs at icdevgroup.org
Sun Dec 23 13:38:19 EST 2007
User: kwalsh
Date: 2007-12-23 18:38:18 GMT
Modified: dist robots.cfg
Log:
* More robots recognised.
Revision Changes Path
2.10 +4 -0 interchange/dist/robots.cfg
rev 2.10, prev_rev 2.9
Index: robots.cfg
===================================================================
RCS file: /var/cvs/interchange/dist/robots.cfg,v
retrieving revision 2.9
retrieving revision 2.10
diff -u -r2.9 -r2.10
--- robots.cfg 27 Sep 2007 21:43:30 -0000 2.9
+++ robots.cfg 23 Dec 2007 18:38:18 -0000 2.10
@@ -13,6 +13,7 @@
asterias,
ATN_Worldwide,
Atomz,
+ AurNet,
Awasu,
BackRub,
bender,
@@ -52,6 +53,7 @@
Gromit,
Gulliver,
H?m?h?kki,
+ heritrix,
HTTrack,
Harvest,
holmes,
@@ -149,6 +151,7 @@
xtreme,
Yahoo,
Yandex,
+ Zeus,
ZyBorg,
EOR
@@ -190,6 +193,7 @@
*.lycos.com,
*.pa-x.dec.com,
*.phx.gbl,
+ *.search.live.com,
add-url.altavista.com,
msnbot.msn.com,
westinghouse-rsl-com-usa.NorthRoyalton.cw.net,
More information about the interchange-cvs
mailing list