kevin at cursor.biz
Sat Feb 21 13:34:06 EST 2004
Grant [emailgrant123b at yahoo.com] wrote:
> > For spiders who don't identify themselves using a
> > UA, or set the UA
> > to pretend to be MSIE etc. Common examples are
> > spambots. There are
> > also "IP" spiders, such as those you may have seen
> > from the
> > 18.104.22.168/27 range. Spambots and IP spiders
> > don't follow the
> > established rules, such as honouring robots.txt, so are best blocked
> > from the firewall. Other spiders may be more
> > friendly, despite not
> > being reliably identifiable using the UA.
> I see, but RobotIP doesn't do any blocking at the
> firewall, right? I thought all RobotIP/UA did was
> suppress sessions...
Correct - that's all it does.
> ...and keep other forms of "URL variables" out of the URL.
No - if you want to use URI arguments then it's up to you to
consider the impact that decision may have on some spiders.
Interchange won't interfere.
> How is that useful for a spambot or other rule-not-follower?
It's no use at all if you want to block a spambot entirely. "Spambots
and IP spiders don't follow the established rules, such as honouring
robots.txt, so are best blocked from the firewall."
_/ _/ _/_/_/_/ _/ _/ _/_/_/ _/ _/
_/_/_/ _/_/ _/ _/ _/ _/_/ _/ K e v i n W a l s h
_/ _/ _/ _/ _/ _/ _/ _/_/ kevin at cursor.biz
_/ _/ _/_/_/_/ _/ _/_/_/ _/ _/
More information about the interchange-users