Finding coefficients of $x^0,x^1,x^2$ in
$((\cdots (x-2)^2-2)^2-2)^2\cdots )-2)^2$
where there are $k$ parenthesis in the left side
Try:
Let $P(x)=((\cdots (x-2)^2-2)^2-2)^2\cdots )-2)^2$
Let we assume
$P(x)=a_{k}+b_{k}x+c_{k}x^2+d_{k}x^3+\cdots\cdots $
For constant term put $x=0$, we have
$$((\cdots (0-2)^2-2)^2-2)^2\cdots )-2)^2=a_{k}$$
Now i did not know how to find coeff. of $1,x,x^2$ in $P(x)$ . Thanks
Let the $x^2,x,1$ coefficients of the polynomial formed from the expression with $k$ brackets be $a_k,b_k,c_k$ respectively. We have $$a_1,b_1,c_1=1,-4,4$$ $$a_{k+1},b_{k+1},c_{k+1}=2a_k(c_k-2)+b_k^2,2b_k(c_k-2),(c_k-2)^2$$ We first notice that $c_k=4$ for all $k\ge1$, so we reduce the recurrences left to solve to $$a_{k+1},b_{k+1}=4a_k+b_k^2,4b_k$$ Solving the recurrence for $b_k$ gives $b_k=-4^k$ and $$a_{k+1}=4a_k+4^{2k}$$ the solution of which is OEIS A166984: $$a_k=\frac{16^k-4^k}{12}$$ Thus the three lowest-degree terms of the expression with $k$ are $$\frac{16^k-4^k}{12}x^2-4^kx+4$$
The $x^3$ coefficient
In the same vein, denote the $x^3$ coefficient for $k$ brackets $d_k$, so
$$d_1=0\qquad d_{k+1}=2(d_k(c_k-2)+a_kb_k)=4d_k-\frac{16^k-4^k}6\cdot4^k$$
$$=4d_k-\frac{64^k-16^k}6$$
This may be solved in a similar manner to $a_k$ to obtain
$$d_k=\frac{-64^k+5\cdot16^k-4\cdot4^k}{360}$$
Lets consider the cases $k=1,2,3,4$:$$P_1(x)=(x-2)^2=x^2-4x+4,\\ P_2(x)=(P_1(x)-2)^2=P_1(x)^2-2P_1(x)+4=x^4-8x^3+20x^2-16x+4,\\ P_3(x)=(P_2(x)-2)^2=x^8+\dots+336x^2-64x+4,\\ P_4(x)=x^{16}+\dots+5440x^2-256x+4.$$
From this it is reasonable to conjecture that the coefficient of $x$ in $P_k(x)$ is $4^k$. Moreover, a quick check using OEIS yields the conjecture that the coefficient of $x^2$ in $P_k(X)$ is given by the sequence $$a_1=1,\\ a_2=20,\\ a_n=20a_{n-1}+64a_{n-2}.$$ This sequence can be written in a compact formula $$a_n=\frac{4\cdot 16^{n+1}-4^{n+1}}{3}.$$
You can prove both conjectures by induction on $k$.
I will show you how to do for the coefficient of $x$. The (harder) coefficient of $x^2$ I leave to you.
We proved the conjecture for $k=1,2,3,4$. This sets up induction. Now assume we know it up to $P_k(x)$. Consider $$P_{k+1}(x)=(P_k(x)-2)^2=P_k(x)^2-4P_k(x)+4.$$ Write $P_k(x)=\sum\limits_{i=0}^{2^k}c_ix^i$. The terms with $x$ in the expression above are $$c_0c_1x+c_1c_0x-4c_1x=(2c_0c_1-4c_1)x.$$ Since $c_0=4$, this yields $8c_1-4c_1=4c_1$. Now use the induction hypothesis to deduce that the coefficient equals $4\cdot 4^k=4^{k+1}$.
\begin{align} p_0(x) &= x \\ p_1(x) &= (p_0(x) - 2)^2 = (x-2)^2 \\ & \vdots \\ p_k(x) &= (p_{k-1}(x) - 2)^2 \end{align}
For any polynomial $p(x)$, to solve for the coefficient of $x^k$, it suffices to compute $\dfrac{p^{(k)}(x)}{k!}$.
Claim 1: $\forall k \in \Bbb{N}^*, p_k(0) = 4$
Proof by induction: the case for $k = 1$ is obvious. Assume the result for $k$. $$p_{k+1}(0) = (p_k(0) - 2)^2 = (4-2)^2 = 4 \tag*{$\square$}$$
Claim 2: $\forall k \in \Bbb{N}^*, p_k'(0) = -4^k$
Proof by induction: the case for $k = 1$ follows from $p_1'(x) = 2(x-2)$ so that $p_1'(0) = -4$. Assume the result for $k$. \begin{align} p_{k+1}'(x) &= 2\,(p_k(x) - 2)\,p_k'(x) \\ p_{k+1}'(0) &= 2\,(p_k(0) - 2)\,p_k'(0) \\ &= 2\,(4 - 2)\,p_k'(0) \tag{Claim 1} \\ &= 4p_k'(0) \end{align} This gives the recurrence sequence $p_{k+1}'(x) = 4p_k'(0)$ with $p_1'(0) = -4$, so that $p_k'(0) = -4^k \quad \forall k \in \Bbb{N}^*. \tag*{$\square$}$
Claim 3: $\forall k \in \Bbb{N}^*, p_k''(0) = 2 \, \dfrac{4^{2k} - 4^k}{12}$
Proof by induction: the case for $k = 1$ follows from $p_1'(x) = 2(x-2)$ so that $p_1'(0) = -4$. Assume the result for $k$.
\begin{align} p_{k+1}'(x) &= 2\,(p_k(x) - 2)\,p_k'(x) \\ p_{k+1}''(x) &= 2\,[(p_k'(x))^2 + (p_k(x) - 2)\,p_k''(x)] \\ p_{k+1}''(0) &= 2\,[(p_k'(0))^2 + (p_k(0) - 2)\,p_k''(0)] \\ &= 2\,[(-4^k)^2 + (4 - 2)\,p_k''(0)] \tag{Claims 1 & 2} \\ &= 2\,[16^k + 2\,p_k''(0)] \\ \therefore \;p''_{k+1}(0) &= 4\,p''_k(0) + 2 \cdot 16^k \\ p''_{k+1}(0) - 4\,p''_k(0) &= 2 \cdot 16^k = 2 \cdot 4^{2k} \\ 4p''_k(0) - 4^2\,p''_{k-1}(0) &= 2 \cdot 4 \cdot 4^{2(k-1)} = 2 \cdot 4^{2k-1} \\ & \vdots \\ 4^{k-1}\,p''_2(0) - 4^k\,p_1''(0) &= 2 \cdot 4^{k-1} \cdot 4^2 = 2 \cdot 4^{k+1} \end{align}
Sum up the above $k$ eqautions to get $p''_{k+1}(0) - 4^k\,p''_1(0) = 2 \sum\limits_{i = 1}^k 16^i$. Note that $p''_1(0) = 2$, so \begin{align} p''_{k+1}(0) &= 2 \cdot 4^k + 2 \sum\limits_{i = k+1}^{2k} 4^i \\ &= 2 \,\left[ 4^k + 4^{k+1} \cdot \frac{4^k-1}{4-1} \right] \\ &= 2 \,\left[ 4^k + 4\cdot \frac{4^{2k}-4^k}{3} \right] \\ &= 2 \, \frac{4 \cdot 4^{2k} - 4^k}{3} \\ &= 2 \, \frac{4^{2(k+1)} - 4^{k+1}}{12} \tag*{$\square$} \\ \end{align}
Therefore, the coefficients of $1$, $x$ and $x^2$ in $p(x)$ are $4$, $-4^k$ and $\dfrac{4^{2k} - 4^k}{12}$ respectively.
