Prove that $(f(x)-x)^2 \not|f( f(...f(x)))) - x$

Question

Let $f(x) \in \mathbb{R}[x], \deg f \geq 2$. Then $(f(x)-x)^2 \not|f( f(...f(x)))) - x$.

I found this problem in my old notes, but there was no solution, and I could not remember one.

Answer 1

This is an incomplete answer: it remains open the case in which $f(x)-x$ is square-free.

Denote the $k$-th iterate of $f$ by $f^{[k]}$, and let $g(x)=f(x)-x$. If $a\in\mathbb C$ is a root of $g$, then we can write $g(x)=(x-a)^rh(x)$, where $r\geq1$ and $h(a)\ne0$. On the other hand, $a$ is a fixed point of $f$, so $a$ is a root of $f^{[k]}(x)-a$ for all $k\geq0$. Consequently we can write $f^{[k]}(x)-a=(x-a)p_k(x)$, where $p_k(x)$ satisfies $p_k(a)=\bigl(f^{[k]}(x)-a\bigr)'(a)=\bigl(f^{[k]}\bigr)'(a)$; it can be easily proved by induction on $k$ that $\bigl(f^{[k]}\bigr)'(a)=\bigl[f'(a)\bigr]^k$ for all $k\geq1$. Finally we have $f'(a)=g'(a)+1$. Putting these results together we get, for all $n\geq1$:

$$\begin{align*} f^{[n]}(x)-x=&\,\sum_{k=0}^{n-1}g\bigl(f^{[k]}(x)\bigr)\\[2mm] =&\,\sum_{k=0}^{n-1}\bigl(f^{[k]}(x)-a\bigr)^r\,h\bigl(f^{[k]}(x)\bigr)\\[2mm] =&\,(x-a)^r\,\,\underbrace{\sum_{k=0}^{n-1}p_k(x)^r\,h\bigl(f^{[k]}(x)\bigr)}_{p(x)}\,, \end{align*}$$

and

$$p(a)=h(a)\sum_{k=0}^{n-1}\bigl[g'(a)+1\bigr]^{kr}\,.$$

If $r\geq2$, that is, if $a$ is a multiple root of $g$, then $g'(a)=0$, which implies $p(a)=nh(a)\ne0$, so the multiplicity of $a$ as root of $f^{[n]}(x)-x$ is the same as its multiplicity as root of $g$, and so $g(x)^2$ does not divide $f^{[n]}(x)-x$ in this case.

If $r=1$ then we have $g(x)=(x-a)h(x)$ and $h(a)=g'(a)\ne0$, and $p(a)$ becomes

$$p(a)=h(a)\,\frac{\bigl[g'(a)+1\bigr]^n-1}{g'(a)+1-1}=\bigl[g'(a)+1\bigr]^n-1\,.$$

Suppose that $g$ is square-free and satisfies $g(x)^2\mid f^{[n]}(x)-x$. Then for each root $a$ of $g$ we have $p(a)=0$ (don't forget that $r,h$ and $p$ depend on $a$). Since $g$ is square-free, it follows that $g$ divides $(g'+1)^n-1$. The converse is also true: if $g$ is square-free and $g$ divides $(g'+1)^n-1$, then for each root $a$ of $g$ we have $\bigl[f^{[n]}(x)-x\bigr]'(a)=\bigl[f'(a)\bigr]^n-1=\bigr[(g'(a)+1)^n-1\bigl]=0$ and (of course) $\bigl[f^{[n]}(x)-x\bigr](a)=0$, so $g^2$ divides $f^{[n]}(x)-x$.

Consequently, a counterexample for your statement is equivalent to the existence of $d$ distinct complex numbers $a_1,\dots,a_d$ such that the polynomial $g(x)=\prod(x-a_i)$ has real coefficients (that is, the $a_i$ are real or they come in complex conjugate pairs) and for each $i$ there is an integer $k$ with $1\leq k\leq n-1$ such that $g'(a_i)=e^{2\pi ki/n}-1$.

Note that the result is true when $g$ is not square-free and the characteristic of the field of coefficients does not divide $n$. I believe that a counterexample for the square-free case can exist, but if the result is indeed true, I believe that it will be true not just for real coefficients but on characteristic $0$ in general.