The coefficient $\beta_j$ of $x^j$ in $(a_0 + a_1 x + \ldots + a_nx^n)^k$ will be
$$\beta_j = \sum_{r_1 + r_2 + \ldots + r_k = j} a_{r_1} a_{r_2} \cdots a_{r_k}$$
So, for instance, if we look at the $x^4$ coefficient in $(a_0 + a_1 x + a_2 x^2)^3$, we sum over all the (ordered!) ways to write $4$ as a sum of exactly $3$ of our existing indices.
- $4 = 2 + 2 + 0$
- $4 = 2 + 0 + 2$
- $4 = 0 + 2 + 2$
- $4 = 1 + 1 + 2$
- $4 = 1 + 2 + 1$
- $4 = 2 + 1 + 1$
so we see
$$
\begin{align}
\beta_4
&= a_2 a_2 a_0 + a_2 a_0 a_2 + a_0 a_2 a_2 + a_1 a_1 a_2 + a_1 a_2 a_1 + a_2 a_1 a_1 \\
&= 3 a_0 a_2^2 + 3 a_1^2 a_2
\end{align}
$$
Of course, we can check this in a computer algebra system like sage.

To see why this is the case, you might be interested in cauchy products. Particularly since you tagged this question as "generating-functions". You can also find this information in chapter 2 of Wilf's excellent generatingfunctionology.
I hope this helps ^_^