Weird Flash Math problem

[excogitator]

Hi,
I have faced this question since the earlier version of Flash but this one beats them all. Can anyone please let me know why this calculation gives this arbitrary result in Flash CS3 using Number data type.

If tried in Flash 8 AS 2.0 and any other calculator and even manually calculated it
================
var tx:Number;
tx=(357.15-116.65);
trace(tx);

The result 240.5 (is correct)

In AS 3.0 same code
================
var tx:Number;
tx=(357.15-116.65);
trace(tx);

The result 240.49999999999997

If you think this does not matter at all. Think of the various calculations that are processed while performing major calculations in serious projects. Let me know any inputs or if I am going wrong anywhere.

[McGuffin]

Well, this is pretty weird to me. In AS3, the Number datatype is a 64-bit double precision floating point, which can represent 357.15, 116.65 and 240.5 without rounding. You run into problems with representing numbers like 1/3, they end up just being approximations which can lead to some weird calculations. You can do what I've done on the last line here to get the exact value, but it's less than ideal:



var tx:Number = Number(357.15 - 116.65);
trace(Number(357.15), Number(116.65), Number(240.5));
trace(tx);
trace(Math.round(tx * 10)/10);

/* Outputs:
357.15 116.65 240.5
240.49999999999997
240.5
*/


So you can see the Number datatype is representing the numbers fine.

[excogitator]

You are right there. But a simple calculation that requires workarounds as such baffles me. Because considering a use case scenario if the above calculation was wrong. Every other calculation pointing to that variable would be an error. I hope some better solution exists because it is a very simple calculation. Fortunately we know here that it is giving an incorrect value which is why we are rounding it but what if unknown elsewhere.

[Jerryscript]

It's due to round-off error, and it can accumulate in larger values than might be expected.

Here's a google search that has a bunch of discussions about it, and how to deal with it:
http://www.google.com/search?sourceid=navclient&ie=UTF-8&rls=GGLG,GGLG:2007-31,GGLG:en&q=actionscript+roundoff+error

[excogitator]

One thing I noticed this problem not only occurs in Flash but in many other apps too. However they need to be addressed with a better solution. For now I think I will have to brace myself to some rounding technique such as mentioned by McGuffin as and when I find the errors.

[Jerryscript]

One thing you can do is try to use integers as much as possible. If you can perform your calculations with integer math using bitwise shifts etc, you can virtually eliminate roundoff error.

[excogitator]

Bitwise shifts has always been fast processing for solving calculations. I think that would be a better operation to explore. Thanks Jerry. I will check up on that and yup again as mentioned use also integers.