6 thoughts on “what happens when you calculate “0.4 times 48” ?

  1. Quite simple: the problem is that most floating point numbers can not be represented exactly in binary. Therefore the 0.4 is not 0.4 but 0.400000$something. If $something times 40 is big enough to be visible, then output like the above will occour. Simply a rounding error. For the very same reason one should never ever compare two floating point numbers with the equality operator in a program. Always use something like number1-number2<epsilon and set epsilon to a very small number!=0.

  2. hmm.. and why does the same tool on the same series of CPU (G4) produce a different output on any other machine different from ahzf’s notebook?

  3. Maybe this is due to his scientific layout and your standard layout. I’ve read that the calculator included in Windows obeys to the order of of operation application (1+1*2=3) only in scientific mode. Just a guess. (Hopefully Apple’s tool will be not that confusing – besides the rounding problem.)

  4. Ahzf has requested a high-precision (16-digits) result. I suppose you’re using the default 11-digit precision, that’s why your result gets corrected by rounding.
    You can change the precision in the “view – precision” menu.

Comments are closed.