[ic] AOL spiders

[Kevin Walsh]

Grant [listbox@email.com] wrote:
> 
> AOL spiders don't have anything unique in their User Agent, but their
> hostname could be used to trap them with something like "spider*aol".  Is
> there any way to set functionality like RobotUA's using the spider's
> hostname info?
> 
This should do the trick for 4.9.3+:

    RobotHost spider*.proxy.aol.com

The RobotHost directive goes in your interchange.cfg file along with
the RobotUA and RobotIP directives.

-- 
   _/   _/  _/_/_/_/  _/    _/  _/_/_/  _/    _/
  _/_/_/   _/_/      _/    _/    _/    _/_/  _/   K e v i n   W a l s h
 _/ _/    _/          _/ _/     _/    _/  _/_/    kevin@cursor.biz
_/   _/  _/_/_/_/      _/    _/_/_/  _/    _/