[ic] HELP - Inktomisearch stuck on ord/basket.html page
Andrew Rich
andrew at mapsdownunder.com.au
Wed Oct 27 22:04:11 EDT 2004
> > Banning spiders from the basket is your best bet.
>
> While robots.txt is great not all robots follow it and as far as I'm
> concerned I'd rather have more control by banning them directly rather
> than hoping the bots will do the work for me after they read
> robots.txt
>
> To do it with rewrites you can use something like (in httpd.conf
> <VirtualHost ..> directive or in .htaccess):
>
> RewriteEngine On
> RewriteCond %{HTTP_USER_AGENT}
> ^.*(bot|pider|Googlebot|FAST|Sidewinder|T-
>
Rex|Architext|Backrub|Gulliver|Slurp|ZyBorg|Yahoo|Scooter|msnbot|mozDex|
ps
> bot|ia_archiver|Wotbox|Teoma|Gigablast|Testingbot).*
> RewriteRule ^(/ord/basket.html.*)$ - [F,L]
>
> Beware that this is a broad net ... anything with bot or pider in the
> name will be banned. To just do inktomi's spider put only Slurp in
> the second line like
> RewriteCond %{HTTP_USER_AGENT} ^.*(Slurp).*
>
> **Disclaimer** Using rewrites is great, but may break all sorts of
> things so test and re-test and do research before you implement.
>
> I saw your note on the interchange-users digest.
Thank you Bryan.
As my problem is only with Inktomisearch at the moment I will keep it
basic and just redo my robots.txt but I will look more into what you are
suggesting.
Andrew Rich
More information about the interchange-users
mailing list