I'm answering my own question with some algebra techniques (software) to find a solution to the equation with four radicals. I got the idea from the MathPages blog. This is a more general version of the problem, where the solution circle has radius $r$ and the other three circles have radii $a$, $b$, and $c$. The distances between the outer circle centers are $A$, $B$, and $C$:

The equation can be written as:
$$\sqrt{K}=\sqrt{X}+\sqrt{Y}+\sqrt{Z}$$
where:
$$K=(A + B + C) (-A + B + C) (A - B + C) (A + B - C)$$
$$X=(A + 2 r + b + c) (-A + 2 r + b + c) (A - b + c) (A + b - c)$$
$$Y=(B + 2 r + a + c) (-B + 2 r + a + c) (B - a + c) (B + a - c)$$
$$Z=(C + 2 r + a + b) (-C + 2 r + a + b) (C - a + b) (C + a - b)$$
Clearing the radicals provides us with:
$$[2(X^2+Y^2+Z^2+K^2)-(X+Y+Z+K)^2]^2 - 64 X Y Z K=0$$
After substituting and expanding to get an enormous 8th degree polynomial (too large to write down here) of $r$, we can use algebra software to show that the following quadratic polynomial can be factored out:
$$r^2 \cdot (A^4 - 2 A^2 B^2 - 2 A^2 C^2 + 4 A^2 a^2 - 4 A^2 a b - 4 A^2 a c + 4 A^2 b c + B^4 - 2 B^2 C^2 - 4 B^2 a b + 4 B^2 a c + 4 B^2 b^2 - 4 B^2 b c + C^4 + 4 C^2 a b - 4 C^2 a c - 4 C^2 b c + 4 C^2 c^2)$$
$$+$$
$$r \cdot (2 A^4 a - 2 A^2 B^2 a - 2 A^2 B^2 b - 2 A^2 C^2 a - 2 A^2 C^2 c + 4 A^2 a^3 - 2 A^2 a^2 b - 2 A^2 a^2 c - 2 A^2 a b^2 - 2 A^2 a c^2 + 2 A^2 b^2 c + 2 A^2 b c^2 + 2 B^4 b - 2 B^2 C^2 b - 2 B^2 C^2 c - 2 B^2 a^2 b + 2 B^2 a^2 c - 2 B^2 a b^2 + 2 B^2 a c^2 + 4 B^2 b^3 - 2 B^2 b^2 c - 2 B^2 b c^2 + 2 C^4 c + 2 C^2 a^2 b - 2 C^2 a^2 c + 2 C^2 a b^2 - 2 C^2 a c^2 - 2 C^2 b^2 c - 2 C^2 b c^2 + 4 C^2 c^3)$$
$$+$$
$$A^4 a^2 + A^2 B^2 C^2 - A^2 B^2 a^2 - A^2 B^2 b^2 - A^2 C^2 a^2 - A^2 C^2 c^2 + A^2 a^4 - A^2 a^2 b^2 - A^2 a^2 c^2 + A^2 b^2 c^2 + B^4 b^2 - B^2 C^2 b^2 - B^2 C^2 c^2 - B^2 a^2 b^2 + B^2 a^2 c^2 + B^2 b^4 - B^2 b^2 c^2 + C^4 c^2 + C^2 a^2 b^2 - C^2 a^2 c^2 - C^2 b^2 c^2 + C^2 c^4$$
The solutions of this quadratic represent the possible radii of the solution circle. Desmos demo here. Here is a python program using SymPy to do the factoring:
from sympy import *
a, b, c, A, B, C, r = symbols('a b c A B C r')
K = (A + B + C) * (-A + B + C) * (A - B + C) * (A + B - C)
X = (A + 2 * r + b + c) * (-A + 2 * r + b + c) * (A - b + c) * (A + b - c)
Y = (B + 2 * r + a + c) * (-B + 2 * r + a + c) * (B - a + c) * (B + a - c)
Z = (C + 2 * r + a + b) * (-C + 2 * r + a + b) * (C - a + b) * (C + a - b)
expr = (2 * (X ** 2 + Y ** 2 + Z ** 2 + K ** 2) - (X + Y + Z + K) ** 2) ** 2 - 64 * X * Y * Z * K
print(expr.factor())