[ic] HELP - Inktomisearch stuck on ord/basket.html page

Jamie Neil jamie at versado.net
Mon Oct 25 19:27:00 EDT 2004


Andrew Rich wrote:
> I initially stopped it by adding an entry to robots.txt for
> /cgi-bin/cartname/ord/basket.html and this pushed it off the page.
> 
> I then thought orders were reduced and removed the entry from
> robots.txt.  Inktomisearch came back and became stuck on the page again.

I don't see any reason why adding pages (or directories) to your 
robots.txt file would affect orders - this is the preferred way of 
limiting spider activity on your site.

Use something like:

   User-agent: *
   Disallow: /cgi-bin/cartname/ord/
   Disallow: /cgi-bin/cartname/scan/
   Disallow: /images/
   Disallow: /cgi-bin/cartname/login.html

As well as "ord" we also ban robots from our images directory (to stop 
our graphics turning up in image searches), and "scan" to stop them 
indexing "more" links.

-- 
Jamie Neil | <jamie at versado.net> | 0870 7777 454
Versado I.T. Services Ltd. | http://versado.net/ | 0845 450 1254


More information about the interchange-users mailing list