[ic] Really strange [calc] evaluationinterchange-users at icdevgroup.org

Grant emailgrant at gmail.com
Wed Jan 27 21:37:55 UTC 2010


>>>>>>> The following should evaluate to zero:
>>>>>>>
>>>>>>> [calc]197.7 - (32.95 * 6)[/calc]
>>>>>>>
>>>>>>> but instead I get:
>>>>>>>
>>>>>>> -2.8421709430404e-14
>>>>>>>
>>>>>>> Everything else is functioning normally.  Does anyone know what's
>>>>>>> going on here?  I'm on 5.6.1.
>>>>>>
>>>>>> Standard floating point arithmetic. Same thing occurs in every
>>>>>> programming
>>>>>> language that uses FP. Use rounding (printf) or CPAN modules to handle
>>>>>> it.
>>>>>
>>>>> OK, I'd like to change the code so that it produces the correct
>>>>> result.  I tried:
>>>>>
>>>>> [calc]sprintf("%.0f",(197.7 - (32.95 * 6)))[/calc]
>>>>>
>>>>> but I get -0.  This situation seems pretty strange to me.  How can I
>>>>> get [calc] to calculate reliably?
>>>>>
>>>>> - Grant
>>>>>
>>>>
>>>> Assuming 197.7 is $197.70, how about converting to integers and
>>>> specifying
>>>> everything as cents, then convert it back to dollars.cents at the end.
>>>>
>>>> [calc]sprintf("%.2f", (int (197.70 * 100) - (int(32.95*100) * 6) )/100
>>>> )[/calc]
>>>>
>>>> Hmm. Is there something in sprintf  that you can tell it to print an
>>>> integer
>>>> like 3295 as 32.95? That would reduce the risk of the /100 causing
>>>> floating
>>>> point problems. Or is that pick or cobol or something else buried deep in
>>>> the back of my brain?
>>>>
>>>> Angus
>>>
>>> Can anyone tell me what it is about this calculation that is causing
>>> the floating point problem?
>>
>> It's all about how the decimal numbers are represented in binary.
>>
>> For a quick experiment that you can safely try at home:
>>
>> On a piece of paper add up 1/3 + 1/3 + 1/3. You should get a total of 1.
>>
>> On your calculator, do the same thing.
>> The calculator says:
>>        1/3= 0.33333333333
>> so the calculator says:
>>        1/3 + 1/3 + 1/3 = 0.33333333333 + 0.33333333333 + 0.33333333333
>>                        = 0.99999999999
>> which is not equal to 1.
>>
>> The same sort of thing is happening in your calculation. Using google's
>> convert feature:
>>        http://www.google.ca/search?q=convert+197.7+to+floating+point
>> shows
>>        197.7 = 0f4068B66666666666
>> All those hexadecimal 6's mean that the internal representation of the
>> number in binary has 01100110011001100...
>> In the same way that the 0.3333 does not add up, so with the 110011001100...
>>
>>> I use [calc] extensively and now I'm
>>> wondering if this could pop up anywhere.  Wrapping all of my
>>> calculations in sprintf and int seems strange.
>>
>> Yes, it does seem strange. The problem is not in your use of calc, it is in
>> the very very common practice of using real numbers (which are meant for
>> measuring things) to perform tasks which are actually counting. We should be
>> counting the number of pennies (an integer task) instead of measuring
>> portions of dollars. But enough philosophy, everybody does it that way, so
>> you just need to decide how often this type of thing will cause an error,
>> and whether that error matters.
>>
>> If this value 197.7 - (32.95 * 6) was being used to test for being < 0,
>> (maybe  shipping is free after that) then it does matter. You could code it
>> as 'if blah < 0.00000001 then do something' or 'if 197.7 < 32.95*6 then do
>> something'.
>>
>> If this value is being added to the discount or total or shipping or
>> something then the infinitesimal non-zero part will just disappear, so it
>> does not matter.
>>
>> Good luck.
>>
>> Angus
>
> Thank you very much everyone for your help with this.  I decided to go
> with this:
>
> [tmp floating_point_error_check][calc]sprintf("%.3f",([scratch
> total))[/calc][/tmp]
> [if !scratch floating_point_error_check]
>   [tmpn total]0[/tmpn]
> [/if]
>
> Does that seem OK?
>
> - Grant

Actually, I think it's more like:

[tmp floating_point_error_check][calc] sprintf("%.3f",([scratch
total])) [/calc][/tmp]
[if scratch floating_point_error_check eq -0.000]
   [tmpn total]0[/tmpn]
[/if]

- Grant



More information about the interchange-users mailing list