[ic] HELP - Inktomisearch stuck on ord/basket.html page

Andrew Rich andrew at mapsdownunder.com.au
Wed Oct 27 22:04:11 EDT 2004

> > Banning spiders from the basket is your best bet.
> While robots.txt is great not all robots follow it and as far as I'm
> concerned I'd rather have more control by banning them directly rather
> than hoping the bots will do the work for me after they read
> robots.txt
> To do it with rewrites you can use something like (in httpd.conf
> <VirtualHost ..> directive or in .htaccess):
> RewriteEngine  On
> RewriteCond %{HTTP_USER_AGENT}
> ^.*(bot|pider|Googlebot|FAST|Sidewinder|T-
> bot|ia_archiver|Wotbox|Teoma|Gigablast|Testingbot).*
> RewriteRule ^(/ord/basket.html.*)$ -  [F,L]
> Beware that this is a broad net ... anything with bot or pider in the
> name will be banned.  To just do inktomi's spider put only Slurp in
> the second line like
> RewriteCond %{HTTP_USER_AGENT}      ^.*(Slurp).*
> **Disclaimer**  Using rewrites is great, but may break all sorts of
> things so test and re-test and do research before you implement.
> I saw your note on the interchange-users digest.

Thank you Bryan.

As my problem is only with Inktomisearch at the moment I will keep it
basic and just redo my robots.txt but I will look more into what you are

Andrew Rich

More information about the interchange-users mailing list