Is it possible to write functions in the form $$f(a)=b$$ where $a$ and $b$ are algebraic expressions of $x$ (e.g $a=3x^2$ , $b=4x^5$)?
The example function would be: $$f(3x^2)=4x^5$$
If this is possible, how can I rewrite them in a form $f(a)=b=f(x)=...$?
If $a=a(x)$ and $b=b(x)$, you ask when we can find a function $f$ such that $b=f(a)$.
This is possible if and only if $a(x)=a(y)$ implies $b(x)=b(y)$. In particular, if $a$ is a one to one function, then this is possible.
For your example, plug in $x = 1$ to get $f(3) = 4$, and plug in $x = -1$ to get $f(3) = -4$. This is a contradiction, since $f(3)$ can only have one value. Hence, no such function $f$ exists.
Now, if $a$ is one-to-one, then $a^{-1}(y)$ is uniquely defined for all $y$ in the range of $a$. Then, $f(a(x)) = b(x) \leadsto f(a(a^{-1}(y))) = b(a^{-1}(y)) \leadsto f(y) = b(a^{-1}(y))$ for all $y$ in the range of $a$.
So, if $a$ is one-to-one, we can write $f(x) = b(a^{-1}(x))$ on the range of $a$, but we have no information about $f(x)$ for $x$ outside the range of $a$.
For instance, suppose we know that $f(\cos x) = \cos 2x$ for all $x \in \mathbb{R}$. Since $\cos 2x = 2\cos^2 x - 1$ and $\cos x$ has a range of $[-1,1]$, we know that $f(y) = 2y^2-1$ for all $y \in [-1,1]$, but we don't know anything about $f$ outside $[-1,1]$. In fact, we can define $f(y)$ arbitrarily for $|y| > 1$, and $f(\cos x) = \cos 2x$ will still be true for all $x \in \mathbb{R}$.
Yes. For example, we could define a function on the rational numbers
$$ f(x/y) = \frac{x^2 - xy + 3y^2}{2x^2 + y^2}$$
whenever $x$ is an integer and $y$ is a nonzero integer. This is a perfectly good definition of a function.
However, there are two potential problems to definitions made in this way that do not show up in the usual way to define a function pointwise:
If we define $f(x^2) = x$ on the reals, we run into a problem: what is $f(-1)$? If we try to define $f$ this way, it is not well-defined, because we have not assigned output values to inputs that cannot be expressed as $x^2$
Consider now the definition $f(x/y) = x + y$ on the rationals. This $f$ is not well-defined, because
and thus we've assigned two different values to the value of $f(2)$, which isn't allowed for a function.