[ic] Maximum size for a session?

DB DB at M-and-D.com
Sun Nov 15 15:36:04 UTC 2009


Recently I saw Yahoo's shopping bot hitting my site pretty hard. Apache
access log has lines like this every 20 seconds or so:

...HTTP/1.0" 200 19391 "-" "YahooSeeker/1.2 (compatible; Mozilla 4.0;
MSIE 5.5; yahooseeker at yahoo-inc dot com ;
http://help.yahoo.com/help/us/shop/merchant/)"

I saw no entry for this in my system's robots.cfg and I suspect (can't
prove) that this robot was obtaining a session which grew *very* large.
So I have two questions:

What exactly should I add to my robots.cfg

and

Is there a way to set a maximum size for sessions so that the next time
a robot that's not in my robots.cfg file comes along this problem won't
repeat?

DB



More information about the interchange-users mailing list