2

I have tried this on my Windows 7 calculator with $\sqrt9 -3 $ it too gives some weird answer- ie$1.1546388020691628168216106791278e-37$. And so for any $n$(positive) $\sqrt n^2-n= wierd_ .answer$
Why does this happen?

Shivam Patel
  • 4,012

2 Answers2

3

Presumable, the Windows 7 calculator uses floating point arithmetic to do these computations.

The IEEE 754 floating point standard requires that basic arithmetic operations (which include sqrt) are correctly rounded: the result should be as if it was computed with infinite precision and then correctly rounded to fit in a double.

Apparently, the Windows 7 calculator has a bug: instead of the correct answer 2 (which is of course exactly representable in a double) it seems to come up with the largest double smaller than 2. If you then subtract 2, you get something very close to but not exactly equal to 0.

Magdiragdag
  • 15,049
  • 1
    I don't think it is a bug, roots are calculated afaik via iterations so the errors may grow extremely. If you have double floating point precision, shouldn't the largest machine number smaller than 2 be something like $2-10^{-16}$ and not something like $2-10^{-39}$ ? – Dominic Michaelis Oct 06 '13 at 16:02
  • 1
    I understand why it happens, but it's still wrong - it does not meet the specification. The makers probably didn't care and don't consider it a bug, though, but there I'm guessing. As for the the $2 - 10^{-16}$ vs. $2 - 10^{-39}$, it's probably extended double instead of double (128 bits instead of 64), but I didn't check. – Magdiragdag Oct 06 '13 at 16:05
1

The weird answer $-8.16\ldots e-39$ stands for $-8.16\ldots \cdot 10^{-39}$. It happens because a calculator just calculates numerical, so the maximal precision is the so called machine precision, which is usally something about $10^{-16}$

Dominic Michaelis
  • 19,935