1

I've been trying to work out the difference between scale and precision in relation to decimal numbers.

Google

Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

Most of the articles seem to be related to SQL data types, as per the following image:

enter image description here

Questions

  • Does the same hold true in a purely mathematical context?
  • If the above question is true, given the value 0.001, is its scale 3 and precision 4?

Edit (for Bounty)

Based on feedback in the comments I've received so far, it seems that "no, scale and precision aren't the terms used to define these numbers", so perhaps a more appropriate question is:

What are the equivalent terms in mathematics?

Matthew Layton
  • 129
  • 3
    I have never heard the words "scale" used to describe the numbers after the decimal point before. As far as I know, the words "significant figures" are used, rather than "precision", in science and computing. In floating point numbers in computing the terminology seen at this web site https://float.exposed/ are the usual ones. – Suzu Hirose Aug 17 '22 at 21:31
  • 1
    "scale" is for changes of size when drawing bluprints, given as a ratio, a better suit for the concept you are presenting is the "order of magnitude". – Joako Aug 22 '22 at 11:33

1 Answers1

1

Google

Precision is the number of digits in a number.

enter image description here

given the value 0.001, is its scale 3 and precision 4?

I've never heard of the word ‘scale’ used in this context. According to the definition in the above image, 0.001 and 7000 have ‘precision’ $1,$ 0.0100 has ‘precision’ $3,$ and 700.0400 has ‘precision’ $7.$

  • In statistics, business and science, typically, precision = consistency = low variability = closeness of the data points = reproducibility & repeatability.

    In this case, precision and accuracy are orthogonal concepts, and imprecision is not thought to lead to inaccuracy.

    Here, we are discussing ‘precision’ with reference to a set of repeated measurements.

  • In physics and numerical analysis, however, I think of precision not as a measure of data dispersion, but as a measure of the resolution of the instrument or the recordings or the calculations.

    In this sense, imprecision is about uncertainty more than about inconsistency, and can lead to inaccuracy (loss of correctness), due to the approximation (rounding and truncation) errors being cumulatively propagated by an algorithm, its numerical instability, or a problem's ill condition.

    Here, we are discussing ‘precision’ with reference to a single or set of non-repeated measurement(s).

This comment nicely links these two aspects/interpretations of ‘precision’: “The closer the repeated results are to each other, the more confident you can be about expressing the result with a greater number of digits.”

ryang
  • 38,879
  • 14
  • 81
  • 179