Say we have a solid background with color $\vec C_0 = (R_0,G_0,B_0)$, and we'd like to paint another color with transparency $(R,G,B,\alpha) = (\vec C,\alpha)$ over it. The result should be $\alpha \vec C + (1-\alpha) \vec C_0$. We can write this as a matrix equation (where the $\vec C$s are column vectors of $R,G,B$)
$$
\begin{bmatrix} \alpha \vec C + (1-\alpha) \vec C_0 \\ 1 \end{bmatrix}
=
\begin{bmatrix} (1-\alpha) I & \alpha \vec C \\ 0 & 1 \end{bmatrix}
\begin{bmatrix} \vec C_0 \\ 1 \end{bmatrix}.
$$
So if you have two colors with transparency $(C_1,\alpha_1)$ and $(C_2,\alpha_2)$, then the operation of painting the second and then the first would be represented by the matrix
$$
\begin{bmatrix} (1-\alpha_1) I & \alpha_1 \vec C_1 \\ 0 & 1 \end{bmatrix}
\begin{bmatrix} (1-\alpha_2) I & \alpha_2 \vec C_2 \\ 0 & 1 \end{bmatrix}
=
\begin{bmatrix} (1-\alpha_1)(1-\alpha_2) I & \alpha_1 \vec C_1 + (1-\alpha_1)\alpha_2 \vec C_2 \\ 0 & 1 \end{bmatrix}
$$
This shows that blending $(\vec C_1,\alpha_1)$ on top of $(\vec C_2,\alpha_2)$ gives the same result as blending just the single composite color $$ \left( \frac{\alpha_1 \vec C_1 + (1-\alpha_1) \alpha_2 \vec C_2)}{1 - (1 - \alpha_1)(1 - \alpha_2)}, 1 - (1 - \alpha_1)(1 - \alpha_2) \right). $$
One can see why it would be useful to represent transparent colors with premultiplied alpha $\vec c = \alpha \vec C$, and also to store $\beta = 1 - \alpha$ instead of $\alpha$ (so $\beta = 0$ is opaque and $\beta = 1$ is fully transparent). For then the form of the matrices is
$$
\begin{bmatrix} \beta I & \vec c \\ 0 & 1 \end{bmatrix}
$$
and composition formula becomes simply
$$ (\vec c_1,\beta_1), (\vec c_2,\beta_2) \mapsto (\vec c_1 + \beta_1 \vec c_2, \beta_1 \beta_2) $$
Addendum: If you are really mixing colors rather than overlaying, then we can simply average the color matrices:
$$
\frac{1}{2} \left(
\begin{bmatrix} (1-\alpha_1) I & \alpha_1 \vec C_1 \\ 0 & 1 \end{bmatrix} +
\begin{bmatrix} (1-\alpha_2) I & \alpha_2 \vec C_2 \\ 0 & 1 \end{bmatrix}
\right) =
\begin{bmatrix} \bigl(1 - \frac{\alpha_1+\alpha_2}{2}\bigr)I & \frac{\alpha_1 \vec C_1 + \alpha_2 \vec C_2}{2} \\ 0 & 1 \end{bmatrix}
$$
to get
$$(\vec C_1,\alpha_1), (\vec C_2,\alpha_2) \mapsto \left( \frac{\alpha_1 \vec C_1 + \alpha_2 \vec C_2}{\alpha_1 + \alpha_2}, \frac{\alpha_1 + \alpha_2}{2} \right). $$
or in terms of $\vec c$ and $\beta$:
$$(\vec c_1,\beta_1), (\vec c_2,\beta_2) \mapsto
\left(\frac{\vec c_1 + \vec c_2}{2},\frac{\beta_1 + \beta_2}{2}\right).$$
Moral: Using premultiplied alpha and $\beta = 1-\alpha$ makes things a lot easier.
