Please take this not yet as an answer, only it is too long for the comment-box
Hmm, I thought to help, but I'm not sure that I understand your question at all. If I read your equation correctly, then a coeffcient $a_j$ is explained by the higher ones $a_{j+1}, a_{j+2} ,... a_{k+1}$. So this ( for a certain $k$ ) looks like a matrix equation of the following type $ C*A=A$ where $C$ contains your binomial-coefficients:
$$ \begin{array}{lll}
\begin{array}{lll}
\end{array}&
*&\begin{bmatrix}
1\\ a_1\\a_2\\a_3\\a_4\\a_5
\end{bmatrix} \\
\begin{bmatrix}
c_{1,0}&.&c_{1,2}&c_{1,3}&c_{1,4}&c_{1,5}\\
c_{2,0}&.&. & c_{2,3}& c_{2,4}& c_{2,5}\\
c_{3,0}&.&. &. &c_{3,4}&c_{3,5}\\
.&.&. &. & 1 & .\\
.&.&. &. & .& 1\\
\end{bmatrix}&
=&\begin{bmatrix}
a_1\\a_2\\a_3\\a_4\\a_5\\
\end{bmatrix} \\ \end{array}$$
but where for instance $a_4$ and $a_5$ are given(?) and are inserted in the upper vector $A$ (and then "computed" into the lower vector $A$).
The iteration begins then that with them and your given coefficents $c_{j,m}$ from there $a_3$ is computed in the lower $A$-vector (and then also inserted in the upper $A$-vector) and then $a_2$ and finally $a_1$.
So for instance , your equation
$$a_j = \frac{k!}{j!(k-j+1)!}-\sum_{m=j+1}^{k+1}a_{m}\frac{m!}{j!(m-j+1)!}$$
for j=3 and k+1=5 becomes
$$a_3 = \frac{4!}{3!2!}-(a_4\frac{4!}{3!(2)!} + a_5\frac{5!}{3!(3)!}) $$
and this is expressed in the matrix-form by the third row where we get
$$ a_3 = c_{3,0}+ a_4 c_{3,4} + a_5 c_{3,5} $$
(only where I have made the minus-signs in the coefficients $c_{j,m}$ to make the appearance clearer)
and then $a_2$ can be computed by the second row of the matrix-schema:
$$ a_2 = c_{2,0}+ a_3 c_{2,3}+ a_4 c_{2,4} + a_5 c_{2,5} $$
Is that correct so far?
(The first idea would then be that this can be handled as an eigenvalue/eigenvector -problem of the matrix $C$ )