Suppose $f_0$ is a Riemann integrable function on $[0,1]$, $f_n(x)=\sqrt{\int_0^xf_{n-1}(t)dt}$, find $\lim_\limits{n \to \infty}f_n(x)$.
It is easy to see if the limit does exist and is differentiable on $[0,1]$, $\lim_\limits{n \to \infty}f_n(x)$ must be $0$ or $\frac{1}{2}x$. But I can't prove the existence of this limit.
Note that $f_1$ is always non-negative and continuous. Let
$$\alpha = \sup\{a \in [0, 1] : f_1 \equiv 0 \text{ on } [0, a] \}.$$
This $\alpha$ is also the maximum of $a$ for which $f_0 = f$ is ''essentially zero'' on $[0, a]$, although it is easier to use $f_1$ to define such threshold $\alpha$. Now we claim:
Claim. $(f_n)_{n=0}^{\infty}$ converges pointwise on $[0, 1]$ with the limit
$$\lim_{n\to\infty} f_n(x) = \tfrac{1}{2}(x-\alpha)_+, $$
where $x_+ := \max\{x, 0\}$ denotes the positive part of $x$.
We begin by an easy observation that $f_n \equiv 0$ on $[0, \alpha]$ for any $n \geq 1$. In particular, if $\alpha = 1$, then the limit is identically zero, proving the claim in this special case. This shifts focus to the case $\alpha < 1$, which we assume hereafter.
Fix an arbitrary $c \in (\alpha, 1)$ and write $m = f_2(c)$ and $M = f_2(1)$ for simplicity. Then the choice of $\alpha$ and the assumption on it guarantees that $0 < m \leq M$. Also, $f_2$ is non-decreasing. So the inequality
$$ m \mathbf{1}_{[c, 1]}(x) \leq f_2(x) \leq M \mathbf{1}_{[\alpha, 1]}(x) $$
holds true on $[0, 1]$. Integrating both sides and taking square root,
$$ m^{1/2} (x - c)_+^{1/2} \leq f_{3}(x) \leq M^{1/2} (x - \alpha)_+^{1/2}. $$
Iterating this idea, we get
\begin{gather} m^{1/4} \left(\tfrac{2}{3}\right)^{1/2} (x - c)_+^{3/4} \leq f_{4}(x) \leq M^{1/4} \left(\tfrac{2}{3}\right)^{1/2} (x - \alpha)_+^{3/4},\\ m^{1/8} \left(\tfrac{2}{3}\right)^{1/4} \left(\tfrac{4}{7}\right)^{1/2} (x - c)_+^{7/8} \leq f_{5}(x) \leq M^{1/8} \left(\tfrac{2}{3}\right)^{1/4} \left(\tfrac{4}{7}\right)^{1/2} (x - \alpha)_+^{7/8},\\ \vdots\\ m^{1/2^n} a_n (x - \alpha)_+^{1-1/2^n} \leq f_{n+3}(x) \leq M^{1/2^n} a_n (x - \alpha)_+^{1-1/2^n}, \end{gather}
where $a_n$ is defined by
$$ a_n = \prod_{k=1}^{n-1} \left(\tfrac{2^k}{2^{k+1}-1}\right)^{1/2^{n-k}} = \exp\biggl( \frac{1}{2^n} \sum_{k=1}^{n-1} 2^k \log\left( \tfrac{2^k}{2^{k+1}-1} \right) \biggr). $$
However, a simple application of Cesaro-Stolz theorem shows that $a_n \to \frac{1}{2}$, and so, as $n\to\infty$ we get
$$ \tfrac{1}{2}(x-c)_+ \leq \liminf_{n\to\infty} f_n(x) \leq \limsup_{n\to\infty} f_n(x) \leq \tfrac{1}{2}(x-\alpha)_+. $$
Since this is true for any $c \in (\alpha, 1)$, letting $c\downarrow \alpha$ proves the desired claim.
