[ic] Webcrawler 80legs vs NotRobotUA
Gert van der Spoel
gert at 3edge.com
Wed Mar 2 13:38:27 UTC 2011
Hi,
See also : http://rt.icdevgroup.org/Ticket/Display.html?id=344
Because in NotRobotUA is found 'Gecko' this crawler will not be stopped by
the current Robot setup of Interchange.
The following is a solution to fence off this crawler . Side effect, of
course, is not being crawled.
But saves a lot of sessions being created for nothing.
User-agent: 008
Disallow: /
CU,
Gert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.icdevgroup.org/pipermail/interchange-users/attachments/20110302/7d3e3b7f/attachment.htm>
More information about the interchange-users
mailing list