0

In my database, data stored as a precision of 10 digits Decimal(30,10).

User can enter x or 1/x. I need to save in 1/x. If user enters 1310 it will be saved in database as 1/1310=0.0007633588. When I want to bring it back 1/0.0007633588=1309.999963 which is not 1310.

If I do the same calculation in Excel/Calculator applications, it always returns correct value (1310 in this case).

Excel example,

-------------------------------
| 3              |  3         |
| 0.333333333    | 1/3        |
| 3              | 1/0.333333 |
-------------------------------

Is there any algorithm to follow?

Imran Qadir Baksh - Baloch
  • 491
  • There is no magic algorithm. If you store $1/x$ to the $10^{th}$ decimal then you won't be able to distinguish between the user entering $1310$ vs $1310.0001$, for example. Excel doesn't do any magic, either, it just works at a higher precision by default, and in floating-point vs. fixed-point. – dxiv Mar 22 '23 at 06:09
  • Don't you think excel is doing rounding after 15th precision? See my example – Imran Qadir Baksh - Baloch Mar 22 '23 at 06:13
  • 1
    Excel is working at the machine floating-point double-precision by default, which means 15-17 significant digits for the 64-bit double-precision format on popular architectures. This doesn't change the fact that you'll always run into loss of precision at some point, in your case sooner because of the $10$-digit fixed-point choice. – dxiv Mar 22 '23 at 06:17
  • As dxiv pointed out, you need higher precision for your inverse column such that, if you want to convert it back from 1/x to x you can round it to 10 significant digits. – thinkingeye Mar 22 '23 at 06:22
  • I can make it 15 precision same as Excel but even then do I need rounding after 15 digits of precision? – Imran Qadir Baksh - Baloch Mar 22 '23 at 06:30
  • @user960567 Making it a fixed-point precision with $15$ decimal digits is not the same as a $15$-digit floating-point precision. And no matter how you do it, you can never achieve infinite precision with a fixed-width format. All that said, it sounds more like a programming question than a math one, and you can find many more floating-point precision related questions next door on SO. – dxiv Mar 22 '23 at 06:44
  • Thanks dxiv.... – Imran Qadir Baksh - Baloch Mar 22 '23 at 06:47

0 Answers0