[ic] Really strange [calc] evaluationinterchange-users at icdevgroup.org

Angus Rogerson arogerso at admmail.uwaterloo.ca
Wed Jan 27 19:05:14 UTC 2010

On 27-Jan-10, at 1:12 PM, Grant wrote:

>>>>> The following should evaluate to zero:
>>>>> [calc]197.7 - (32.95 * 6)[/calc]
>>>>> but instead I get:
>>>>> -2.8421709430404e-14
>>>>> Everything else is functioning normally.  Does anyone know what's
>>>>> going on here?  I'm on 5.6.1.
>>>> Standard floating point arithmetic. Same thing occurs in every
>>>> programming
>>>> language that uses FP. Use rounding (printf) or CPAN modules to 
>>>> handle
>>>> it.
>>> OK, I'd like to change the code so that it produces the correct
>>> result.  I tried:
>>> [calc]sprintf("%.0f",(197.7 - (32.95 * 6)))[/calc]
>>> but I get -0.  This situation seems pretty strange to me.  How can I
>>> get [calc] to calculate reliably?
>>> - Grant
>> Assuming 197.7 is $197.70, how about converting to integers and 
>> specifying
>> everything as cents, then convert it back to dollars.cents at the end.
>> [calc]sprintf("%.2f", (int (197.70 * 100) - (int(32.95*100) * 6) )/100
>> )[/calc]
>> Hmm. Is there something in sprintf  that you can tell it to print an 
>> integer
>> like 3295 as 32.95? That would reduce the risk of the /100 causing 
>> floating
>> point problems. Or is that pick or cobol or something else buried 
>> deep in
>> the back of my brain?
>> Angus
> Can anyone tell me what it is about this calculation that is causing
> the floating point problem?

It's all about how the decimal numbers are represented in binary.

For a quick experiment that you can safely try at home:

On a piece of paper add up 1/3 + 1/3 + 1/3. You should get a total of 1.

On your calculator, do the same thing.
The calculator says:
	1/3= 0.33333333333
so the calculator says:
	1/3 + 1/3 + 1/3 = 0.33333333333 + 0.33333333333 + 0.33333333333
			= 0.99999999999
which is not equal to 1.

The same sort of thing is happening in your calculation. Using google's 
convert feature:
	197.7 = 0f4068B66666666666
All those hexadecimal 6's mean that the internal representation of the 
number in binary has 01100110011001100...
In the same way that the 0.3333 does not add up, so with the 

> I use [calc] extensively and now I'm
> wondering if this could pop up anywhere.  Wrapping all of my
> calculations in sprintf and int seems strange.

Yes, it does seem strange. The problem is not in your use of calc, it 
is in the very very common practice of using real numbers (which are 
meant for measuring things) to perform tasks which are actually 
counting. We should be counting the number of pennies (an integer task) 
instead of measuring portions of dollars. But enough philosophy, 
everybody does it that way, so you just need to decide how often this 
type of thing will cause an error, and whether that error matters.

If this value 197.7 - (32.95 * 6) was being used to test for being < 0, 
(maybe  shipping is free after that) then it does matter. You could 
code it as 'if blah < 0.00000001 then do something' or 'if 197.7 < 
32.95*6 then do something'.

If this value is being added to the discount or total or shipping or 
something then the infinitesimal non-zero part will just disappear, so 
it does not matter.

Good luck.


Angus Rogerson
Retail Services, University of Waterloo

> - Grant

More information about the interchange-users mailing list