[ic] HELP - Inktomisearch stuck on ord/basket.html page

Jamie Neil jamie at versado.net
Mon Oct 25 19:27:00 EDT 2004

Andrew Rich wrote:
> I initially stopped it by adding an entry to robots.txt for
> /cgi-bin/cartname/ord/basket.html and this pushed it off the page.
> I then thought orders were reduced and removed the entry from
> robots.txt.  Inktomisearch came back and became stuck on the page again.

I don't see any reason why adding pages (or directories) to your 
robots.txt file would affect orders - this is the preferred way of 
limiting spider activity on your site.

Use something like:

   User-agent: *
   Disallow: /cgi-bin/cartname/ord/
   Disallow: /cgi-bin/cartname/scan/
   Disallow: /images/
   Disallow: /cgi-bin/cartname/login.html

As well as "ord" we also ban robots from our images directory (to stop 
our graphics turning up in image searches), and "scan" to stop them 
indexing "more" links.

Jamie Neil | <jamie at versado.net> | 0870 7777 454
Versado I.T. Services Ltd. | http://versado.net/ | 0845 450 1254

More information about the interchange-users mailing list