Let's suppose for a non-zero constant $k$ and every real number $x$, we have:
$$f\big(f(x)\big)=kx\tag0\label0$$
$$f\big(x^2\big)=xf(x)\tag1\label1$$
By \eqref{0} you can find out that $f$ is injective. Now, by \eqref{0} and \eqref{1} you have:
$$f\left(f(x)^2\right)=f(x)f\big(f(x)\big)=kxf(x)=kf\left(x^2\right)=f\bigg(f\Big(f\left(x^2\right)\Big)\bigg)=f\left(kx^2\right)$$
$$\therefore\quad f(x)^2=kx^2\tag2\label2$$
Therefore by \eqref{2} we have $k=f(1)^2$ and hence $k$ is positive. Again by \eqref{2}, for every real number $x$ we get $f(x)=\pm\sqrt kx$.
Now let's define $K^\pm:=\big\{x\ne0\big|f(x)=\pm\sqrt kx\big\}$. So we get $\mathbb R=\{0\}\cup K^+\cup K^-$. By \eqref{1} we have $f(0)=0$. You can reformulate \eqref{0} and \eqref{1} this way:
$$x\in K^\pm\quad\text{iff}\quad\sqrt kx\in K^\pm$$
$$x\in K^\pm\quad\text{iff}\quad x^2\in K^\pm$$
It's easy to see that every function satisfying $f(0)=0$ and $f(x)=\pm\sqrt kx$ for $x\in K^\pm$ is a solution, where $K^\pm$ satisfy the above conditions.
The trivial cases happen when one of $K^\pm$ is empty, and give us the linear solutions $f(x)=\sqrt kx$ and $f(x)=-\sqrt kx$. For a nontrivial case, you can take $K^+=\{\pm k^q\vert q\in\mathbb Q\}$ and $K^-=\{\pm k^q\vert q\in\mathbb R\backslash\mathbb Q\}$.
EDIT:
As noted by @DanielWainfleet in the comments, the way my example above is presented, it only works for $k\ne1$. In fact, I could simply write $K^-=\mathbb R\setminus(K^+\cup\{0\})$, and then it would work for all $k>0$. But @DanielWainfleet has also provided an interesting example that works for $k=1$: $K^+=\left\{\pm 2^{2^n}\big\vert n\in\mathbb Z\right\}$.
f(x) = x*Sqrt[k]for x in set S, andf(x) = x*-Sqrt[k]for x outside of S. Just choose S carefully. – Jan 24 '16 at 22:04