[ic] Maximum size for a session?
DB at M-and-D.com
Sun Nov 15 15:36:04 UTC 2009
Recently I saw Yahoo's shopping bot hitting my site pretty hard. Apache
access log has lines like this every 20 seconds or so:
...HTTP/1.0" 200 19391 "-" "YahooSeeker/1.2 (compatible; Mozilla 4.0;
MSIE 5.5; yahooseeker at yahoo-inc dot com ;
I saw no entry for this in my system's robots.cfg and I suspect (can't
prove) that this robot was obtaining a session which grew *very* large.
So I have two questions:
What exactly should I add to my robots.cfg
Is there a way to set a maximum size for sessions so that the next time
a robot that's not in my robots.cfg file comes along this problem won't
More information about the interchange-users