[ic] HELP - Inktomisearch stuck on ord/basket.html page

Bryan tquark at gmail.com
Tue Oct 26 18:49:40 EDT 2004


> Banning spiders from the basket is your best bet.

While robots.txt is great not all robots follow it and as far as I'm
concerned I'd rather have more control by banning them directly rather
than hoping the bots will do the work for me after they read
robots.txt

To do it with rewrites you can use something like (in httpd.conf
<VirtualHost ..> directive or in .htaccess):

RewriteEngine  On
RewriteCond %{HTTP_USER_AGENT}     
^.*(bot|pider|Googlebot|FAST|Sidewinder|T-Rex|Architext|Backrub|Gulliver|Slurp|ZyBorg|Yahoo|Scooter|msnbot|mozDex|psbot|ia_archiver|Wotbox|Teoma|Gigablast|Testingbot).*
RewriteRule ^(/ord/basket.html.*)$ -  [F,L]

Beware that this is a broad net ... anything with bot or pider in the
name will be banned.  To just do inktomi's spider put only Slurp in
the second line like
RewriteCond %{HTTP_USER_AGENT}      ^.*(Slurp).*

**Disclaimer**  Using rewrites is great, but may break all sorts of
things so test and re-test and do research before you implement.

Bryan


More information about the interchange-users mailing list