[ic] one store gone nuts - 500 errors - HELP!

Peter peter at pajamian.dhs.org
Tue Sep 4 21:01:39 EDT 2007


On 09/04/2007 05:39 PM, Glenn McCalley wrote:
> 
> ----- Original Message ----- From: "Peter" <peter at pajamian.dhs.org>
> To: <interchange-users at icdevgroup.org>
> Sent: Tuesday, September 04, 2007 7:41 PM
> Subject: Re: [ic] one store gone nuts - 500 errors - HELP!
> 
> 
>> On 09/04/2007 04:27 PM, Glenn McCalley wrote:
>>> ----- Original Message ----- From: "Peter" <peter at pajamian.dhs.org>
>>> To: <interchange-users at icdevgroup.org>
>>> Sent: Tuesday, September 04, 2007 7:05 PM
>>> Subject: Re: [ic] one store gone nuts - 500 errors - HELP!
>>>
>>>
>>>> On 09/04/2007 02:22 PM, Glenn McCalley wrote:
>>>>> Hi all.
>>>>>
>>>>> Store works fine until you press the final "Place Order" button on the
>>>>> checkout page.
>>>>> 500 - Internal Server Error.
>>>>>
>>>>
>>>> The only time I have seen where the global error.log file doesn't show
>>>> something is if Perl itself crashes, check for a core dump.  Was
>>>> Perl or
>>>> one of its modules recently upgraded?  Do you use the stock Perl or one
>>>> you compiled yourself?  What about the modules?  What platform are you
>>>> running on?
>>>>
>>>> Peter
>>>>
>>> Hi Peter, yes I do get perl.core dumps.  Sorry should have said that
>>> before.
>>>
>>> - FreeBSD 5.3
>>> - Perl 5.8.5 compiled myself, not the port.
>>> - Interchange 5.2.0
>>> - No Perl updates additions in some time.
>>>
>>> BUT -- the other stores are fine.  Hald a dozen of them.  Same IC.
>>> Selling things, issuing receipts, making money.
>>> That's what's got me swinging - this box has been rock solid for over a
>>> year and it's only -this-one-store-.
>>> I'd think if it was Perl by itself they'd all be smoking ruins.
>>
>> If you're getting core dumps from Perl then that's definitely your
>> problem.  They can be caused by the oddest combination of things and can
>> easily happen on just one shop.
>>
>> Upgrade to the latest perl (5.8.8) and build a new
>> Bundle::InterchangeKitchenSink, DBD::mysql (or whatever other database
>> you have) and any other perl modules you might be using.  I would
>> install in a seperate directory (I usually use something like
>> /usr/local/perl-5.8.8)
>> that way you can keep your old perl and switch back if there's any new
>> problems, though 5.8.8 has been out for quite some time now so I doubt
>> you will find any problems with it.
>>
>> Do that and your problems will almost certainly disappear.
>>
>> Peter
> 
> Agreed, except how can it easily happen on just one store?  They all use
> the same Perl,

Bugs like this are typically buffer overflow bugs that happen only in
rare circumstances, so a particular combination of things could be
triggering it (ie this internal function is called with that exact value
that triggers this other off-by-one overflow bug that would only show up
when this other function is called with some other value), changing any
one of the factors that is triggering the bug will make it go away.

> AND it worked fine until this afternoon.

It could be some very slight change, even a counter could trigger it by
getting over a certain value.  That said, I would look to any upgrade or
change in perl or any of it's modules or any system/other library that
is called or used by perl or one of the modules.

> I'm not arguing Perl should be updated but I don't think Perl just
> decided to not work - something CHANGED from one moment to the next and
> that should be able to be identified and changed back.

Agreed, and you could go down the path of trying to identify what
changed to trigger it, but at the end of the day the right way to fix
the problem is to upgrade Perl and its modules pretty much regardless of
what is triggering the bug.*

Of course the other option is to leave perl and the bug in place, find
the trigger in IC, and avoid triggering it.  If upgrading Perl doesn't
fix the problem that may be your best solution.

* Note that it might be a bug in a system library or another shared or
statically compiled library and not directly inside Perl itself or one
of the modules.  I would still recommend upgrading Perl and the modules
first and if you continue to experience problems then you should dig
into the core dumps and trace out where the problem actually is.  That
can be a rather painful process, though, so best to try upgrading Perl
first.

Peter



More information about the interchange-users mailing list