The reason the answer is indeterminate (without some further specification of the nature of the collisions) is that you have (assuming everything is in a plane) more unknowns than equations.
For example, if any case were solvable, it would be the case where
- The two red balls start at rest.
The red balls have identical mass (and radius).
- The blue ball starts directly toward the tangent point of the red balls (so that by symmetry, the blue ball will not end up with any left/right velocity).
In that case, the there are three parameters describing the answer (the final Y velocity component of the blue ball, the (equal) final Y velocities of the red balls, and the final X velocity of one of the red ball -- the other one gets the negative of this velocity component). And there are naively three equations: conservation of energy, and conservation of X and Y momenta. But the conservation of X momentum is automatic by the symmetry imposed, so there are really only two equations for three unknowns.
Now, say you know something else (for example, that the red balls leave the collision at an opening angle of 120 degrees). Then that third constraint allows you to easily solve the equations and determine the respective velocities.
The way a "frictionless billiards" simulation would work is to model the repulsion force between each pair of balls as some sort of rapidly increasing conservative force with a (fuzzy, but not very fuzzy) force radius of $R$. Then you could take the limit as the potential looks more and more like a step function. You would find (ignoring introduction of spin because there is no friction) that for all three balls identical the opening angle would in fact be 120 degrees, and the solution is that the blue ball continues on with 1/3 of its original velocity, while the two red balls get a speed of twice that.
Physically, the rotation of the two red balls can't be ignored, so your simple billiards simulation would not be that accurate.