Suppose that $f$ is continuous, and not the zero function, and put $M={\rm Sup}\{f(x), x\in [0,1]\}$. There exists $x_0$ such that $M=f(x_0)>0$. Taking $x=\sqrt{x_0}$, we get
$\displaystyle 4\sqrt{x_0}f(x_0)=f(x)+f(1-x)\leq 2M$, hence $\displaystyle 4\sqrt{x_0}\leq 2$, and $\displaystyle x_0\leq \frac{1}{4}$. By the functional equation with $x=x_0$, we get:
$$M+f(1-x_0)=f(x_0)+f(1-x_0)=4x_0f(x_0^2)\leq f(x_0^2)\leq M$$
Hence $f(1-x_0)\leq 0$, and $f(1-x_0)=0$. This imply $4x_0=1$ and $f(x_0^2)=M=f(1/16)=f(1/4)$. Now taking $\displaystyle x_1=1/16$ in the functional equation:
$$f(1-x_1)=4x_1f(x_1^2)-f(x_1)=4x_1f(x_1^2)-M\leq 4x_1M-M=-3M/4<0$$
a contradiction.
EDIT:
A new solution, where
I do not suppose $f$ continuous. Put $M={\rm Sup}\{f(x)\}$ ($M$ exists as $f$ is from $[0,1]$ to $[0,1]$), and suppose that $M>0$.
There exists $x_n$ such that $f(x_n)\to M$. Now let $u$ any value such that there exists a subsequence $x_{n_k}$ of $x_n$ such that $x_{n_k}\to u$ (we have used the compacity of $[0,1]$). Let $y_k=\sqrt{x_{n_k}}$. We get $\displaystyle 4\sqrt{x_{n_k}}f(x_{n_k})=f(y_k)+f(1-y_k)\leq 2M$, and hence $4\sqrt{u}M\leq 2M$. We get $u\leq 1/4$.
Now $\displaystyle f(1-x_{n_k})= 4x_{n_k}f(x_{n_k}^2)-f(x_{n_k})$. Take a subsequence of $x_{n_k}$ say $x_{n_{k_p}}$ such that $f(x_{n_{k_p}}^2)$ is convergent, say to $L\leq M$. We get that $4x_{n_{k_p}}f(x_{n_{k_p}}^2)-f(x_{n_{k_p}})$ is convergent to $4uL-M$. Now if $L<M$ or $4u<1$, $4uL-M<0$ and for $p$ large, we get $f(1-x_{n_{k_p}})<0$, a contradiction. We have thus $L=M$ and $u=1/4$. Now put $y_p=x_{n_{k_p}}^2$. We have $f(y_p)\to M$. By what we have show, there exists a subsequence of $y_p$ that converge to $1/4$. But $y_p$ is convergent to $1/16$, contradiction.
I would be very grateful if you would agree that I use this result in a paper and put your name in the acknowledgment. If this would be possible, please contact me.
– Adam Aug 26 '14 at 19:06You can use absolutely freely the material here. It was a very interesting question. Good continuation, and good luck for your paper.
– Kelenner Aug 26 '14 at 20:03