Let's see whether the function can really be decreasing.
From $\frac{f(x)}{f(ax)} = 1-(1-a)^2x^2$ we immediately see that for large enough $x$ there would be a change of sign of $f$ between $x$ and $ax$, and also between $ax$ and $a^2x$, which is incompatible with monotonic decrease (or increase, for that matter). Thus $f$ can't be monotonic on $\mathbb R_+$. Let's try and make it monotonic at least in some neighborhood of 0, maybe that would suffice for your purposes, whatever those might be.
Now, functional equations linking $f(x)$ to $f(ax)$ for some constant $a$ are "bad", in that they leave too much freedom. See, we may define $f(x)$ to be a totally arbitrary function (even nowhere continuous, if we'd like) for $x$ between $a$ and $1$, and then continue both left and right, repeatedly using the expression for $f(x)$ via $f(ax)$ or vice versa. That would be a solution defined on entire $\mathbb R_+$ and consistent with the equation.
If we want the function to be reasonably well-behaved (say, continuous at 0), that's another story. In that case we may rely on certain limit:
$$f(x) = \Big(1-(1-a)^2x^2\Big)f(ax) = \Big(1-(1-a)^2x^2\Big)\Big(1-(1-a)^2x^2a^2\Big)f(a^2x)=\Big(1-(1-a)^2x^2\Big)\Big(1-(1-a)^2x^2a^2\Big)\Big(1-(1-a)^2x^2a^4\Big)\cdot\ \dots\ \cdot f(0)$$
Wait, isn't this the q-Pochhammer symbol? Although not an elementary function, it is pretty well-known and indeed decreasing in the neighborhood of 0. So this is the thing you were after.
You could try something along the same lines but more sophisticated. Perhaps a series about $a$ or $1-a$, or a Laurent series.
– Paul Castle Jun 04 '15 at 03:38