[ic] Really strange [calc] evaluationinterchange-users at icdevgroup.org

Grant emailgrant at gmail.com
Wed Jan 27 20:55:54 UTC 2010


>>>>>> The following should evaluate to zero:
>>>>>>
>>>>>> [calc]197.7 - (32.95 * 6)[/calc]
>>>>>>
>>>>>> but instead I get:
>>>>>>
>>>>>> -2.8421709430404e-14
>>>>>>
>>>>>> Everything else is functioning normally.  Does anyone know what's
>>>>>> going on here?  I'm on 5.6.1.
>>>>>
>>>>> Standard floating point arithmetic. Same thing occurs in every
>>>>> programming
>>>>> language that uses FP. Use rounding (printf) or CPAN modules to handle
>>>>> it.
>>>>
>>>> OK, I'd like to change the code so that it produces the correct
>>>> result.  I tried:
>>>>
>>>> [calc]sprintf("%.0f",(197.7 - (32.95 * 6)))[/calc]
>>>>
>>>> but I get -0.  This situation seems pretty strange to me.  How can I
>>>> get [calc] to calculate reliably?
>>>>
>>>> - Grant
>>>>
>>>
>>> Assuming 197.7 is $197.70, how about converting to integers and
>>> specifying
>>> everything as cents, then convert it back to dollars.cents at the end.
>>>
>>> [calc]sprintf("%.2f", (int (197.70 * 100) - (int(32.95*100) * 6) )/100
>>> )[/calc]
>>>
>>> Hmm. Is there something in sprintf  that you can tell it to print an
>>> integer
>>> like 3295 as 32.95? That would reduce the risk of the /100 causing
>>> floating
>>> point problems. Or is that pick or cobol or something else buried deep in
>>> the back of my brain?
>>>
>>> Angus
>>
>> Can anyone tell me what it is about this calculation that is causing
>> the floating point problem?
>
> It's all about how the decimal numbers are represented in binary.
>
> For a quick experiment that you can safely try at home:
>
> On a piece of paper add up 1/3 + 1/3 + 1/3. You should get a total of 1.
>
> On your calculator, do the same thing.
> The calculator says:
>        1/3= 0.33333333333
> so the calculator says:
>        1/3 + 1/3 + 1/3 = 0.33333333333 + 0.33333333333 + 0.33333333333
>                        = 0.99999999999
> which is not equal to 1.
>
> The same sort of thing is happening in your calculation. Using google's
> convert feature:
>        http://www.google.ca/search?q=convert+197.7+to+floating+point
> shows
>        197.7 = 0f4068B66666666666
> All those hexadecimal 6's mean that the internal representation of the
> number in binary has 01100110011001100...
> In the same way that the 0.3333 does not add up, so with the 110011001100...
>
>> I use [calc] extensively and now I'm
>> wondering if this could pop up anywhere.  Wrapping all of my
>> calculations in sprintf and int seems strange.
>
> Yes, it does seem strange. The problem is not in your use of calc, it is in
> the very very common practice of using real numbers (which are meant for
> measuring things) to perform tasks which are actually counting. We should be
> counting the number of pennies (an integer task) instead of measuring
> portions of dollars. But enough philosophy, everybody does it that way, so
> you just need to decide how often this type of thing will cause an error,
> and whether that error matters.
>
> If this value 197.7 - (32.95 * 6) was being used to test for being < 0,
> (maybe  shipping is free after that) then it does matter. You could code it
> as 'if blah < 0.00000001 then do something' or 'if 197.7 < 32.95*6 then do
> something'.
>
> If this value is being added to the discount or total or shipping or
> something then the infinitesimal non-zero part will just disappear, so it
> does not matter.
>
> Good luck.
>
> Angus

Thank you very much everyone for your help with this.  I decided to go
with this:

[tmp floating_point_error_check][calc]sprintf("%.3f",([scratch
total))[/calc][/tmp]
[if !scratch floating_point_error_check]
   [tmpn total]0[/tmpn]
[/if]

Does that seem OK?

- Grant



More information about the interchange-users mailing list