First I'll count the right selections with ordering:
- The first has 100 options. (all are OK)
- The second has 95 options (1 gone, and 4 forbidden of the colour of the first)
- The third has 90 options (1 extra gone, 4 extra forbidden ones), and this pattern continues. (so $100-5(n-1)$ options for ball $n$)
So in order we have $$100 \times 95 \times 90 \times \ldots \times 55 = 5^{10}(20 \times 19 \times 18 \times \ldots \times 11)=\frac{5^{10} 20!}{10!}$$
ordered options and we divide that number by $10!$ because we have that many ways to get the same end result given 10 different balls.
To get a probability, divide by $\binom{100}{10}$.
So $$p=\frac{5^{10}\binom{20}{10}}{\binom{100}{10}}$$
which is approximately $0.10423$.
BTW, your own first formula was correct it seems, except that the numerator is $10!$ too large: upper side is ordered options and below is unordered.