[ic] Googlebot Getting 500 Errors ... but he's the only one

Bryan Gmyrek bryangmyrek at yahoo.com
Tue May 17 16:23:13 EDT 2005


Hi.  First I'll quote an email I sent around on Oct 15, 2004:
(note that logs and interchange info follow .. I am using 5.2)

"
I'm seeing a strange error.  When the actual Googlebot visits my
site right now he is getting '500 Internal Server' errors on
almost every page access.  However, when I use wget or switch my 
useragent in
Firefox to be Googlebot I don't get any errors at all.  I have
RobotLimt 0 and there are no entries in the catalog or
interchange error.log corresponding to all of these errors (over
4,000 so far today :( ).  In the apache access_log there are of
course the logs showing the 500 errors and in the apache
error_log are entries like:
[Fri Oct 15 10:24:54 2004] [error] [client 66.249.64.33]
Malformed header return by Interchange:

Thinking that maybe it had something to do with RobotIP I removed
that from interchange.cfg and restarted.  No luck.  Then thinking
it might have to do with the mode of interchange I tried rpc
mode, no luck (I had it in high mode with MaxServers 20) then
tried low mode.  In low mode Googlebot fortunately gets 200 codes
meaning the pages are actually served to him.  This is very good
news to me but causes a problem.  I had to have it in RPC mode to
get AuthorizeNet to always work properly.  So what to do now?
Can anyone suggest what might be causing this?  Or maybe where I
could uncomment/put a logDebug to track down what is going on?

By the way, the strange thing is <edited>...
msnbot has visited over 1000 times today and gotten no 500
errors.  Seems to me that if Googlebot were getting errors then
msnbot would too...
"

All of this is true exept that now this is happening with the catalog 
on low mode.
I tried switching to high mode but googlebot still gets the 500 errors.
Also, last week this started happening so I blew away sesssions and tmp 
dirs and made fresh ones.
This seemed to work for a while at least.  I set up a job to do that 
every night in
hopes Googlebot would be able to crawl the whole site (tons of pages) 
if I got rid of the 500 errors.
So now I'm stuck with Googlebot getting 120,000 pages of 500 errors ... 
doh.

Update:  For now it seems that for whatever reason switching to high 
traffic mode and restarting interchange has fixed it (googlebot getting 
200 codes now).
But I'm sure it'll be back.
Any ideas as to what's really causing this or how I could track that 
down would help a lot.

Thanks,
Bryan

LOGS::

Interchange details:
...
Vend::Payment::AuthorizeNet payment module initialized, using 
Net::SSLeay
High traffic settings.
Calling UI......UI is loaded...
Interchange V5.2.0

Apache details:
[Mon May 16 23:51:14 2005] [warn] Loaded DSO libexec/mod_rewrite.so 
uses plain Apache 1.3 API, this module might crash under EAPI! (please 
recompile it with -DEAPI)
[Mon May 16 23:51:15 2005] [notice] Apache/1.3.33 (Unix) 
mod_interchange/1.31 PHP/4.3.9 mod_ssl/2.8.22 OpenSSL/0.9.7d configured 
-- resuming normal operations

Apache error log for googlebot:
[Tue May 17 11:30:14 2005] [error] [client 66.249.71.39] Malformed 
header return by Interchange:
[Tue May 17 11:30:14 2005] [error] [client 66.249.64.18] Malformed 
header return by Interchange:
[Tue May 17 11:30:16 2005] [error] [client 66.249.71.72] Malformed 
header return by Interchange:
[Tue May 17 11:30:18 2005] [error] [client 66.249.64.33] Malformed 
header return by Interchange:
[Tue May 17 11:30:21 2005] [error] [client 66.249.71.69] Malformed 
header return by Interchange:
[Tue May 17 11:30:23 2005] [error] [client 66.249.71.28] Malformed 
header return by Interchange:
...

Apache access log:
66.249.71.69*[17/May/2005:11:27:35 -0700]*GET /liTEL2725.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.64.30*[17/May/2005:11:28:23 -0700]*GET /liDARFAS-01.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.71.40*[17/May/2005:11:28:25 -0700]*GET /liIMC70091.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.64.55*[17/May/2005:11:28:37 -0700]*GET /liACCREF242.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.71.39*[17/May/2005:11:28:45 -0700]*GET /liTELMT3003.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.71.40*[17/May/2005:11:28:52 -0700]*GET /liTELK4497.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.64.66*[17/May/2005:11:29:11 -0700]*GET /liOWPB1120.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
66.249.71.18*[17/May/2005:11:29:20 -0700]*GET /liDARWAR-04.html 
HTTP/1.0*-*Googlebot/2.1 (+http://www.google.com/bot.html)*500*532*-
...

Interchange error log:
NO RELATED ENTRIES



More information about the interchange-users mailing list