Consider that a set of numbers that can be written in the form $b^{x}$ (for some base):
$$A = \{ a_1, ..., a_n\}$$
for example maybe $\{ 2^{-2000}, 2^{-2002}, 2^{-2500}\}$. Also assume we can compute the numbers indirectly because we know its exponents (i.e. we are in log-space). I want to compute:
$$ \alpha_i = \frac{ a_i}{\sum^n_{i=1} a_i}$$
without running into numerical issues (the numerical issue arise because the original $a_i$'s are so small that the numerator and denominator are so close to zero that any reasonable computing object [with bounded precision $\epsilon$] has an error when trying to compute the ratio above. i.e. it approximately tries to compute $\frac{0}{0}$ and hence, an error).
One way that I was told to do this is by "re-scaling" the numbers. In the sense, we can rescale A such that the largest has magnitude 1 and then perform the computation for $\alpha_i$. The mathematical correctness of the computation of alpha is obvious (with rescaling), as the max term divides top and bottom (cancels out and the ratio is computed correctly).
Let the one with the largest magnitude be denoted by $a^* = argmax_{a \in A}\{a\} $. Mathematically this would involve computing a new scaled set $A'$ as follow:
$$ A' = \{ \frac{a_1}{ a^*} , ..., \frac{a_n}{ a^*} \} = \{ a'_1 , ..., a'_n \}$$
my question is, why are we choosing to divide by the largest $a^*$ and not the smallest? Another way to view this could be, if they can all be written as an exponent, we are choosing the one with the most positive exponent. What is the issue with choosing the most negative exponent. (obviously, we wouldn't compute the division directly and instead compute the difference of the exponents, as to avoid the potential overflow issue, i.e. working in logspace and hence the exponents directly, so that we don't have the overflow issue when doing $\frac{a_i}{ a^*}$).
My intuition tells me that we actually should be dividing by the smallest, since that is the one that might be too small for our computations to handle (and hence the one we want to normalize). Consider it like this, say we write all the numbers in $A$ is exponential form, i.e. $a_i = 2^{e_i}$. Assume $e_i \leq 0$. Then to resolve the issue we can do:
$$ 2^{e_i - y}$$
where we want to make $e_i - y$ less negative (so that the exponentiation does not drive us to close to zero). To do that we want $-y$ to be very positive. So in this goal we choose $y$ to be the most negative exponent we can have. Hence, we choose the smallest exponent that corresponds to the smallest $a_i$. Thats what makes sense to me. But does choosing the max also work? If it does, why does it?
In the $A = \{ 2^{-2000}, 2^{-2002}, 2^{-2500}\}$ and $A' = \{ 1, 2^{-2}, 2^{-500}\}$ and hence, if the precision of our computer is $2^{-1023}$ to $2^{1023}$, this trick would seem to work. However, I wasn't sure if this trick would always work or what conditions it would work or why choosing the largest was a good idea and not the smallest.
