[ic] Webcrawler 80legs vs NotRobotUA
Gert van der Spoel
gert at 3edge.com
Wed Mar 2 13:38:27 UTC 2011
See also : http://rt.icdevgroup.org/Ticket/Display.html?id=344
Because in NotRobotUA is found 'Gecko' this crawler will not be stopped by
the current Robot setup of Interchange.
The following is a solution to fence off this crawler . Side effect, of
course, is not being crawled.
But saves a lot of sessions being created for nothing.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the interchange-users