[ic] HELP - Inktomisearch stuck on ord/basket.html page

Andrew Rich andrew at mapsdownunder.com.au
Mon Oct 25 20:28:16 EDT 2004


> Jamie Neil wrote:
> > Andrew Rich wrote:
> >
> >> I initially stopped it by adding an entry to robots.txt for
> >> /cgi-bin/cartname/ord/basket.html and this pushed it off the page.
> >>
> >> I then thought orders were reduced and removed the entry from
> >> robots.txt.  Inktomisearch came back and became stuck on the page
> again.
> >
> >
> > I don't see any reason why adding pages (or directories) to your
> > robots.txt file would affect orders - this is the preferred way of
> > limiting spider activity on your site.
> >
> > Use something like:
> >
> >   User-agent: *
> >   Disallow: /cgi-bin/cartname/ord/
> >   Disallow: /cgi-bin/cartname/scan/
> >   Disallow: /images/
> >   Disallow: /cgi-bin/cartname/login.html
> >
> > As well as "ord" we also ban robots from our images directory (to
stop
> > our graphics turning up in image searches), and "scan" to stop them
> > indexing "more" links.
> 
> I should add that banning robots from "scan" will prevent them
indexing
> _any_ category/search links on a stock foundation system, which is
> obviously undesirable. We use action maps to do most of our links,
e.g.:
> 
>    /cgi-bin/cartname/category/spanners.html
> 
> and so this is not a problem for us.
> 
> We also feed spiders slightly different search results to normal
users:
> normal users get 10 results per page while spiders get one page with
all
> the results on it. This allows the search engines to index all the
> products in a category without having to follow any "more" links.
> 

Thank you Jamie for your feedback.

As we receive about 75% of orders via search engines we do not want to
do anything to stop them indexing our site.

Thia would obviously be a good start.

   User-agent: *
   Disallow: /cgi-bin/cartname/ord/
   Disallow: /images/
   Disallow: /cgi-bin/cartname/login.html

I am not familiar with action maps so I will have to do some reading
there.

I have tried to move from the Foundation site to the new Standard site
structure (perhaps only new name) so not sure if this makes any
difference!

Regards,
Andrew Rich



More information about the interchange-users mailing list