[ic] Googlebot Getting 500 Errors ... but he's the only one

Peter peter at pajamian.dhs.org
Wed Jun 1 17:39:13 EDT 2005


On 06/01/2005 01:06 PM, Bryan Gmyrek wrote:
> --- Bryan Gmyrek <bryangmyrek at yahoo.com> wrote:
> 
> 
>>Hi,
>>
>>By the way the site I'm having the trouble with is http://www.neartexpress.com/ 
>>I haven't been able to check email much lately so sorry for the late reply.
>>Thanks Jonathan for providing that script!  I ran it and got the same results as Mike
>>(everything
>>OK).
> 
> 
> Considering my access logs (i.e. googlebot gets some success and then failure) I altered
> Jonathan's code to loop until there was a 500 error and then die (see end of this email for the
> code).  Here are the results:
> 
> $ perl test_google_bug.pl http://www.neartexpress.com/liROSSPL3153.html | tee out.log
> <many lines edited out ... all were ok status>
> ok 613 - check http status is 200 without if-modified-since
> #     header
> ok 614 - check http status is not 500 with if-modified-since
> #     header (status is 200)
> Tue May 31 13:31:10 MST 2005
> ok 615 - check http status is 200 without if-modified-since
> #     header
> ok 616 - check http status is not 500 with if-modified-since
> #     header (status is 200)
> Tue May 31 13:31:14 MST 2005
> ok 617 - check http status is 200 without if-modified-since
> #     header
> ok 618 - check http status is not 500 with if-modified-since
> #     header (status is 200)
> Tue May 31 13:31:18 MST 2005
> not ok 619 - check http status is 200 without if-modified-since
> #     header
> not ok 620 - check http status is not 500 with if-modified-since
> #     header (status is 500)
> 
> 
> So it eventually does get an error...  I'll have to look into it more though since Googlebot
> definitely gets the error more often than that.
> 
> Best,
> Bryan
> 
> 
> Code::
> #!/usr/bin/perl -w
> 
> use WWW::Mechanize;
> use Test::More tests => 2;
> 
> my $url = $ARGV[0];
> my $ua = WWW::Mechanize->new;
> my $date = `date`;
> 
> my $status = 200;
> while($status != 500){
> print `date`;
> 
> $ua->get($url);
> ok($ua->status == 200, 'check http status is 200 without if-modified-since
>     header');
> 
> #$ua->add_header('IF_MODIFIED_SINCE' => 'Wed, 08 Sep 2004 11:09:13 GMT');
> $ua->add_header('IF_MODIFIED_SINCE' => $date);
> $ua->get($url);
> ok($ua->status != 500, "check http status is not 500 with if-modified-since
>     header (status is ".$ua->status.")");
> 
> $status = $ua->status;
> }

Just on a hunch, try setting the if-modified-since header to a date in 
the future for this test and see what happens...

Peter


More information about the interchange-users mailing list