what happens when you calculate “0.4 times 48” ?

Well – the answer is: 19.2 – But not on the iBook of Ahzf. He is getting something like that:

Hmm… strange, eh? Can anyone explain that?

Oh… on my machine it’s this:

  1. #1 by NAMENLOS on November 12, 2005 - 16:07

    Quite simple: the problem is that most floating point numbers can not be represented exactly in binary. Therefore the 0.4 is not 0.4 but 0.400000$something. If $something times 40 is big enough to be visible, then output like the above will occour. Simply a rounding error. For the very same reason one should never ever compare two floating point numbers with the equality operator in a program. Always use something like number1-number2<epsilon and set epsilon to a very small number!=0.

  2. #2 by Bietiekay on November 12, 2005 - 16:22

    hmm.. and why does the same tool on the same series of CPU (G4) produce a different output on any other machine different from ahzf’s notebook?

  3. #3 by Alexander Gross on November 13, 2005 - 01:10

    Maybe this is due to his scientific layout and your standard layout. I’ve read that the calculator included in Windows obeys to the order of of operation application (1+1*2=3) only in scientific mode. Just a guess. (Hopefully Apple’s tool will be not that confusing – besides the rounding problem.)

  4. #4 by bietiekay on November 13, 2005 - 13:03

    nope, in scientific mode the results are the same.

  5. #5 by pck on November 19, 2005 - 18:25

    Ahzf has requested a high-precision (16-digits) result. I suppose you’re using the default 11-digit precision, that’s why your result gets corrected by rounding.
    You can change the precision in the “view – precision” menu.

  6. #6 by nobody on November 27, 2005 - 02:06

    Try another one 0.6 x 48, thats even more strange

Comments are closed.