Questions tagged [rounding-error]
178 questions
2
votes
0 answers
How does rounding affect subsequent calculations?
When we are doing calculations in mathematics, we often express exact values, like $\sqrt 2$ or $\arctan (1)$, as decimals and round these to a finite number of decimal places/significant figures before using these approximations in subsequent…
1
vote
1 answer
Insignificant value
Let us say that a line is defined as
$$ y=\frac{\sqrt{c^2-a^2}}{a}x $$
and $a
Semar
- 97
1
vote
0 answers
How to determine rounding precision of intermediate value to avoid loss of precision?
I have a formula that can be represented as A * B where A = 1 / x and B is a fixed value with a a certain number of significant figures (say n decimal places). The result is also rounded, say to m decimal places.
To how many significant figures…
Niel de Wet
- 111
0
votes
0 answers
A rounding and distribution problem
This is a real-life problem I'm trying to solve, I have an invoice with three items A, B, and C on it, and the total cost of this invoice is $4.98, below is how much each item costs
A $1.99
B $2.17
C $0.82
Now I need to refund a partial amount of…
Arch1tect
- 139
0
votes
2 answers
Google rounds wrongly?
I know, this is not the Google bug tracker, but maybe the problem is my poor understanding of math.
Ask Google to convert pica to mm. (Pica is a measurement unit used in typography.)
https://www.google.com/search?q=pica+to+mm
In the Pica field,…
john c. j.
- 133
0
votes
1 answer
How to *correctly* round to the nearest multiple
I am struggling to find the correct implementation for rounding to the nearest multiple. I thought the simple arithmetic of [number/multiple]*multiple would give me my solution but I am running into instances where this is false. I am trying to…
S1r-Lanzelot
- 133
0
votes
1 answer
What is the round to two decimals of the number 33408.55446000001?
Is it 33408.55 or 33408.56 ?
I would have gone towards the .56 but every online round calculator give me .55 ...
0
votes
1 answer
Rounding logic in sampling
A population consist of elements provided from 4 different sources.
I have decided to take a sample of 30 elements from the population.
The size of the sample taken from each source was relative to the
rate of the contribution of this source to…
0
votes
1 answer
Geometric Mean Application: Minimising the Relative Error in Rounding a Number
I have it written down that if you have $a \lt c \lt b$ and $a\leq x \leq b$ and are going to round $x$ to the nearest of $a$, $b$ or $c$, if you want to minimise the maximum relative error of the rounding, you should set $c = \sqrt{ab}$. And…
Kristof Arnaldo
- 113
0
votes
1 answer
Rounding to nearest integer
If you round to the nearest integer, why do we look at the number it was before instead of the number which it is at that moment?
For example: 17.495
5 or higher goes up which makes it 17.50
which becomes 17.5
Why do we look at the four instead of…
0
votes
1 answer
Multiplication after rounding, missing information of order of 1000, how to find if multiplication had multiple values for each data couple?
Imagine a financial report in a certain currency, indicating the following numbers (and below them, indicating the amount after conversion to USD, from the same report).
1,394,278 (USD46,743)
225,283 (USD7,553)
111,518 (USD3,739)
62,938 …
O0123
- 143
0
votes
2 answers
rounding up to two decimal points
I just have a question regarding my answers in my exam paper, my teacher said to give the answer to two decimal points, I got the correct answer (2.89) but I rounded it up to 2.90. Now my teacher did not give me the mark for this and I want to know…
italy
- 1,001
0
votes
1 answer
Precision & significant figure equality regarding decimal place?
Considering precision to the tenths place, where x is a non-zero integer, and the value has a tenths value of zero, Is there a precision difference between x and x.0 ?
Is one more accurate or mathematically correct?
Is this the same case for 0 vs…
Don
- 103
- 5
0
votes
1 answer
Why is e.g. $e^{0,1*10^{-15}}$ equal to 1?
I mean it is logical that $e^0$ would be 1. Does the result for the calculation in the title just get rounded to 1, because the exponent is nearly 0? Every calculator gets me 1 as result for this calculation
A. Aoe
- 3