Algebra helps. But rather than the geometric ideas of translations, dilations, inversions, one should simply look at where certain carefully chosen points go.
I'll use the notation $A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$, $a,b,c,d \in \mathbb R$, $ad-bc=1$ as suggested in your question.
The idea is to assume that $\phi_A$ takes the imaginary axis to the imaginary axis and to see what additional algebraic constraints this places on the numbers $a,b,c,d$.
First, I'm sure you know that $\phi_A$ takes the real axis to the real axis. Since the real and imaginary axes intersect in a single point, namely the origin $0$, it follows that $\phi(0)=0$. From this it follows that $\frac{a \cdot 0 + b}{c \cdot 0 + d} = 0$ and so $b=0$. So now we have $A = \begin{pmatrix} a & 0 \\ c & d \end{pmatrix}$ with $ad=1$.
Next, I'll follow your suggestion with $\phi_A(xi)=yi$, but more specifically I'll consider what happens when $x=1$, so $\phi_A(i) = yi$:
$$\frac{ai}{ci+d} = yi, \quad y>0
$$
$$ai=dyi-cy
$$
so $cy=0$ which implies $c=0$.
So now we have $A = \begin{pmatrix} a & 0 \\ 0 & d \end{pmatrix}$, $ad=1$. We can therefore let $a$ be a parameter and solve for $d$ to get
$$A = \begin{pmatrix} a & 0 \\ 0 & a^{-1} \end{pmatrix}
$$
And that's all there is to be said: that's the general form of an element of $SL(2,\mathbb R)$ that takes the imaginary axis to itself.