[ic] Googlebot Getting 500 Errors ... but he's the only one

Bryan Gmyrek bryangmyrek at yahoo.com
Wed Jun 1 16:06:58 EDT 2005


--- Bryan Gmyrek <bryangmyrek at yahoo.com> wrote:

> Hi,
> 
> By the way the site I'm having the trouble with is http://www.neartexpress.com/ 
> I haven't been able to check email much lately so sorry for the late reply.
> Thanks Jonathan for providing that script!  I ran it and got the same results as Mike
> (everything
> OK).

Considering my access logs (i.e. googlebot gets some success and then failure) I altered
Jonathan's code to loop until there was a 500 error and then die (see end of this email for the
code).  Here are the results:

$ perl test_google_bug.pl http://www.neartexpress.com/liROSSPL3153.html | tee out.log
<many lines edited out ... all were ok status>
ok 613 - check http status is 200 without if-modified-since
#     header
ok 614 - check http status is not 500 with if-modified-since
#     header (status is 200)
Tue May 31 13:31:10 MST 2005
ok 615 - check http status is 200 without if-modified-since
#     header
ok 616 - check http status is not 500 with if-modified-since
#     header (status is 200)
Tue May 31 13:31:14 MST 2005
ok 617 - check http status is 200 without if-modified-since
#     header
ok 618 - check http status is not 500 with if-modified-since
#     header (status is 200)
Tue May 31 13:31:18 MST 2005
not ok 619 - check http status is 200 without if-modified-since
#     header
not ok 620 - check http status is not 500 with if-modified-since
#     header (status is 500)


So it eventually does get an error...  I'll have to look into it more though since Googlebot
definitely gets the error more often than that.

Best,
Bryan


Code::
#!/usr/bin/perl -w

use WWW::Mechanize;
use Test::More tests => 2;

my $url = $ARGV[0];
my $ua = WWW::Mechanize->new;
my $date = `date`;

my $status = 200;
while($status != 500){
print `date`;

$ua->get($url);
ok($ua->status == 200, 'check http status is 200 without if-modified-since
    header');

#$ua->add_header('IF_MODIFIED_SINCE' => 'Wed, 08 Sep 2004 11:09:13 GMT');
$ua->add_header('IF_MODIFIED_SINCE' => $date);
$ua->get($url);
ok($ua->status != 500, "check http status is not 500 with if-modified-since
    header (status is ".$ua->status.")");

$status = $ua->status;
}



More information about the interchange-users mailing list