[ic] RobotIPs

[Kevin Walsh]

Grant [emailgrant123b at yahoo.com] wrote:
> > For spiders who don't identify themselves using a
> > UA, or set the UA
> > to pretend to be MSIE etc.  Common examples are
> > spambots.  There are
> > also "IP" spiders, such as those you may have seen
> > from the
> > 63.148.99.224/27 range.  Spambots and IP spiders
> > don't follow the
> > established rules, such as honouring robots.txt, so are best blocked
> > from the firewall.  Other spiders may be more
> > friendly, despite not
> > being reliably identifiable using the UA.
> >
> I see, but RobotIP doesn't do any blocking at the
> firewall, right?  I thought all RobotIP/UA did was
> suppress sessions...
>
Correct - that's all it does.

>
> ...and keep other forms of "URL variables" out of the URL.
>
No - if you want to use URI arguments then it's up to you to
consider the impact that decision may have on some spiders.
Interchange won't interfere.

>
> How is that useful for a spambot or other rule-not-follower?
> 
It's no use at all if you want to block a spambot entirely.  "Spambots
and IP spiders don't follow the established rules, such as honouring
robots.txt, so are best blocked from the firewall."

-- 
   _/   _/  _/_/_/_/  _/    _/  _/_/_/  _/    _/
  _/_/_/   _/_/      _/    _/    _/    _/_/  _/   K e v i n   W a l s h
 _/ _/    _/          _/ _/     _/    _/  _/_/    kevin at cursor.biz
_/   _/  _/_/_/_/      _/    _/_/_/  _/    _/