[ic] RobotIPs

[Kevin Walsh]

Grant [emailgrant123b at yahoo.com] wrote:
> > A more focused list of addresses may make it into the default set,
> > as long as a proper justification is presented.  You're free to
> > add any addresses you like into your own local configuration,
> > of course.
> >
> Why would you want to use IPs instead of UAs?
> 
For spiders who don't identify themselves using a UA, or set the UA
to pretend to be MSIE etc.  Common examples are spambots.  There are
also "IP" spiders, such as those you may have seen from the
63.148.99.224/27 range.  Spambots and IP spiders don't follow the
established rules, such as honouring robots.txt, so are best blocked
from the firewall.  Other spiders may be more friendly, despite not
being reliably identifiable using the UA.

-- 
   _/   _/  _/_/_/_/  _/    _/  _/_/_/  _/    _/
  _/_/_/   _/_/      _/    _/    _/    _/_/  _/   K e v i n   W a l s h
 _/ _/    _/          _/ _/     _/    _/  _/_/    kevin at cursor.biz
_/   _/  _/_/_/_/      _/    _/_/_/  _/    _/