3

I have experimental data that I used to plot a calibration curve.

If I have 2 significant figures in my data and a R-squared of 0.9959, is it correct to round the R-Squared to 1.0?

How I think about it is that, since Excel must have used my data to find that R-squared value, significant figures should apply to it too. However I was wondering if determination coefficient was some special case where everybody leaves all the digits no matter what.

Nij
  • 2,991

1 Answers1

0

You're not correctly using significant figures. $\def\rr{\mathbb{R}}$

If you have a function $f : \rr \to \rr$ such that $f(x) = 10-x$ for any $x \in \rr$, consider that:

  • $f(0.0100) = 9.99$
  • $f(0.0101) = 9.9899$

Would you say it is meaningful to round off the output of $f$ to $3$ significant figures given that its input is accurate to $3$ significant figures? No, because you can see that both will result in $10.0$ even though the two inputs differ in the third significant figure. It's not exactly invalid but you've lost some information, and actually need to give more significant figures of the output to capture the information that was in the input.

Likewise if you have a function $g : \rr \to \rr$ such that $g(x) = 1-x$ for any $x \in \rr$, consider that:

  • $g(0.999) = 0.001$
  • $g(0.9985) = 0.0015$
  • $g(0.998) = 0.002$

Again, it would be invalid to give the output of $g$ to $3$ significant figures even if the input was accurate to $3$ significant figures, since as you can see changing the third significant figure of the input can cause even the first significant digit to change!

Similarly, you cannot just round the correlation coefficient to the precision of the input since it is not meaningful, in the same sense as for the above examples.

user21820
  • 57,693
  • 9
  • 98
  • 256