[ic] Webcrawler 80legs vs NotRobotUA

Gert van der Spoel gert at 3edge.com
Wed Mar 2 13:38:27 UTC 2011


Hi,

 

See also : http://rt.icdevgroup.org/Ticket/Display.html?id=344

 

Because in NotRobotUA is found 'Gecko' this crawler will not be stopped by
the current Robot setup of Interchange.

 

The following is a solution to fence off this crawler . Side effect, of
course, is not being crawled.

But saves a lot of sessions being created for nothing.

 

User-agent: 008
Disallow: /

 

CU,

 

Gert

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.icdevgroup.org/pipermail/interchange-users/attachments/20110302/7d3e3b7f/attachment.htm>


More information about the interchange-users mailing list