[ic] Re: Interchange Freezing Up

[Bryan]

Hi,

Over concerns that Google got 'angry' at my robots.txt (i.e. seemed to
go away when I put too much stuff in it) i just made it very simple:

User-agent: *
Disallow:

(i.e. wide open)
This may just be paranoia but in my experience when it comes to search
engines the question isn't are you paranoid, it's are you paranoid
_enough_
But whether your robots.txt is simple or complex I suggest you search
on google for 'robots.txt validator' and check yours there!

Cheers,
Bryan

On Mon, 1 Nov 2004 09:00:22 -0600, Steve <icdev at mrlock.com> wrote:
> 
> 
> > So I tried simply switching to low
> > traffic mode and both orders and googlebot seem to be doing OK
> > (Googlebot grabbed over 25,000 pages today).  So for now I'll stay
> > with this I guess!  Thanks for everyone's help.
> >
> > Bryan
> >
> > p.s. I posted about 'what is rpc mode'  ... does anyone have detailed
> > references about what low, high and rpc mode are and what all of their
> > configs mean?
> 
> Hello Bryan,
> 
> What does your robots.txt file look like?
> 
> I have been modifying mine slightly to keep inktomi and googlebot from
> access certain pages like:
> basket.html, process.html, order.html
> 
> It really helped with inktomi getting stuck on certain pages....
> 
> 
> -Steve
> 
> 
> 
> _______________________________________________
> interchange-users mailing list
> interchange-users at icdevgroup.org
> http://www.icdevgroup.org/mailman/listinfo/interchange-users
>