[interchange-cvs] interchange - racke modified debian/robots.cfg
interchange-core@icdevgroup.org
interchange-core@icdevgroup.org
Mon Apr 7 08:02:04 2003
User: racke
Date: 2003-04-07 12:00:22 GMT
Added: debian robots.cfg
Log:
robots configuration file added
Revision Changes Path
1.1 interchange/debian/robots.cfg
rev 1.1, prev_rev 1.0
Index: robots.cfg
===================================================================
RobotUA <<EOR
ATN_Worldwide, AltaVista, Arachnoidea, Aranha, Architext, Ask, Atomz,
BackRub, Builder, CMC, Contact, Digital*Integrity, Directory, EZResult,
Excite, Ferret, Fireball, Google, Gromit, Gulliver, Harvest, Hubater,
H?m?h?kki, INGRID, IncyWincy, Jack, KIT*Fireball, Kototoi, LWP, Lycos,
MegaSheep, Mercator, Nazilla, NetMechanic, NetResearchServer, NetScoop,
ParaSite, Refiner, RoboDude, Rover, Rutgers, Scooter, Slurp, Spyder,
T-H-U-N-D-E-R-S-T-O-N-E, Toutatis, Tv*Merc, Valkyrie, Voyager, WIRE,
Walker, Wget, WhizBang, Wire, Wombat, Yahoo, Yandex, ZyBorg, appie,
asterias, bot, contact, crawl, collector, fido, find, gazz, grabber,
griffon, archiver, legs, marvin, mirago, moget, newscan, seek, speedy,
spider, suke, tarantula, agent, topiclink, whowhere, winona, worm, xtreme,
EOR
RobotIP <<EOR
202.9.155.123, 204.152.191.41, 208.146.26.19,
208.146.26.233, 209.185.141.209, 209.185.141.211,
209.202.148.36, 209.202.148.41, 216.200.130.207,
216.35.103.6?, 216.35.103.70,
EOR
RobotHost <<EOR
*.crawler*.com, *.excite.com, *.googlebot.com,
*.infoseek.com, *.inktomi.com, *.inktomisearch.com,
*.lycos.com, *.pa-x.dec.com, add-url.altavista.com,
westinghouse-rsl-com-usa.NorthRoyalton.cw.net,
EOR