[ic] Webcrawler 80legs vs NotRobotUA

Gert van der Spoel gert at 3edge.com
Wed Mar 2 13:38:27 UTC 2011



See also : http://rt.icdevgroup.org/Ticket/Display.html?id=344


Because in NotRobotUA is found 'Gecko' this crawler will not be stopped by
the current Robot setup of Interchange.


The following is a solution to fence off this crawler . Side effect, of
course, is not being crawled.

But saves a lot of sessions being created for nothing.


User-agent: 008
Disallow: /





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.icdevgroup.org/pipermail/interchange-users/attachments/20110302/7d3e3b7f/attachment.htm>

More information about the interchange-users mailing list