Step 1: Prove : $f(y) = 0$ iff. $y=0$
It's easy to see $f(y)>0$ for $y>0$ (from porperty (b) with value $a=y,b=0$)
Following proof is given by @Kyson
To prove that $f(y)≠0$ for all$ y<0.$
Assume there is a $y<0$ such that $f(y)=0.$
Choose $x=−2y>0$ and hence $f(x)>0$ and $f(x+y)=f(−y)>0 $Then
$[f(x)+f(y)][f(x+2y)+f(y)]=0$ while $f(x+y)2+f(y)f(2y)>0.$
$ \ $
Step 2: Prove $f(y)$ is even.
Put $x+y=0$ in equation given to get:$f(y)+f(-y)=\frac{f(2y)}{2}$ for all y.
$f(y)+f(-y)=\frac{f(2y)}{2} = f(-y) +f(y) = \frac{f(-2y)}{2} $
Thus $f(x) $is a even function, so $f(y)+f(-y)=2f(y)=\frac{f(2y)}{2} \Rightarrow f(2y)=4f(y) $
(Since $f(x)$ is even ,property (b) actually shows that $f(y) > 0$ on $(-\infty,0)\cup(0,\infty)$)
$ \ $
Step 3: Prove:$ f(y)=y^2f(1)$
By induction , find $f(ny)=n^2f(y)$ where $n \in \mathbb N$
Details of induction:
Suppose $m\in \mathbb N ,m\ge 2 ,\ f(ny)=n^2f(y)$ holds for all $n\le m$
Put $x=(m-1)y$ in property (a).$\\ \Rightarrow ((m-1)^2+1)f(y)f((m+1)y)=(m^4+4-(m-1)^2-1)f^2(y) \\ \Rightarrow f((m+1)y)=(m+1)^2f(y) \ for\ y\not = 0 \ and \ it\ still\ holds\ for\ y=0$
Hint: $(m+1)^2 ((m-1)^2 +1) = (m^2-1)^2+(m+1)^2=m^4 -m^2+2 =m^4+4-(m-1)^2-1$
Remark: Result induces $f(zy)=z^2f(y)$ where $z\in \mathbb Z$ , since $f$ is even.
$ \ $
$\forall r\in \mathbb Q , \ r = \frac ab \ , \ a,b\in \mathbb Z$
. Then $f(ry)=f(\frac ab y)=a^2f(\frac 1b y)$ and $f(y)=b^2f(\frac 1b y)$
Hence $f(ry)=f(\frac ab y)=(\frac ab)^2f(y)=r^2f(y)\Rightarrow f(r)=r^2f(1)$
As @MarkBennet points out:
the function is strictly increasing for $r≥0$ and the rationals are
dense in the reals.
And $f(r)$ is even, so it's strictly decreasing for $r\le 0$.
Conclude $f(x)=x^2f(1)$ where $x\in \mathbb R $
Thus $f(x)$ can only be the form such as $ax^2$ where $a\in \mathbb R^+$