[ic] Re: Interchange Freezing Up
tquark at gmail.com
Mon Nov 1 16:09:06 EST 2004
Over concerns that Google got 'angry' at my robots.txt (i.e. seemed to
go away when I put too much stuff in it) i just made it very simple:
(i.e. wide open)
This may just be paranoia but in my experience when it comes to search
engines the question isn't are you paranoid, it's are you paranoid
But whether your robots.txt is simple or complex I suggest you search
on google for 'robots.txt validator' and check yours there!
On Mon, 1 Nov 2004 09:00:22 -0600, Steve <icdev at mrlock.com> wrote:
> > So I tried simply switching to low
> > traffic mode and both orders and googlebot seem to be doing OK
> > (Googlebot grabbed over 25,000 pages today). So for now I'll stay
> > with this I guess! Thanks for everyone's help.
> > Bryan
> > p.s. I posted about 'what is rpc mode' ... does anyone have detailed
> > references about what low, high and rpc mode are and what all of their
> > configs mean?
> Hello Bryan,
> What does your robots.txt file look like?
> I have been modifying mine slightly to keep inktomi and googlebot from
> access certain pages like:
> basket.html, process.html, order.html
> It really helped with inktomi getting stuck on certain pages....
> interchange-users mailing list
> interchange-users at icdevgroup.org
More information about the interchange-users