Normally, when showing (or even saying) that $f\colon G\to X$ is a group homomorphism, you need to know that $G$ and $X$ are groups in the first place. Without that, i.e., without group operations $\circ_G\colon G\times G\to G$ and $\circ_X\colon X\times X\to X$, you cannot show that $f(g_1\circ_G g_2)=f(g_1)\circ_X f(g_2)$, simply because it doesn't even make sense.
However, under suitable conditions on $f$, we may be able to use the very function $f$ to define a group structure on $X$ from a group structure on $G$ in a way that automatically makes $f$ a homomorphism. That is, we may (attempt to) define
$$\tag1 x_1\circ_X x_2:=f(g_1\circ_G g_2)\quad \text{for }g_1,g_2\in G\text{ with }f(g_1)=x_1, f(g_2)=x_2.$$
(And this attmept is not a wild guess, but rather the only way that could possibly work).
In order for this to work, this $\circ_X$ must be well-defined, that is: the result in $(1)$ must not depend on the choices made for $g_1,g_2$. This boils down to the following requirement for $f$:
$$\tag2\forall g_1,g_2,g_1',g_2'\in G\colon f(g_1')=f(g_1)\land f(g_2')=f(g_2)\to f(g_1'\circ_G g_2')=f(g_1\circ_G g_2).$$
After these remarks, we can apply the above to the situation at hand: Here $G=\Bbb R$ with $\circ_G=+$, and $f$ is your $\phi$.
Note that $\phi(x)=\phi(x')$ is equivalent to $\sin x=\sin x'\land \cos x=\cos x'$. Again $\sin x=\sin x'$ is equivalent to $x'-x\in2\pi\Bbb Z\lor x'+x-\pi\in 2\pi \Bbb Z$, and $\cos x=\cos x'$ is equivalent to $x'-x\in2\pi\Bbb Z\lor x'+x\in 2\pi \Bbb Z$. Hence jointly, they are equivalent to $x-x'\in2\pi \Bbb Z$. But of course if $x-x'\in2\pi \Bbb Z$ and $y-y'\in 2\pi\Bbb Z$, then also $(x'+y')-(x+y)\in2\pi \Bbb Z$. Thus the condition $(2)$ holds for our example. Thus one can use $(1)$ to define a group structure on $S$ that turns $S$ into a group such that $\phi$ is a homomorphism.
The implied group operation on $S$ is therefore such that
$$\tag3(\cos x,\sin x)^T\circ_S(\cos y,\sin y)^T=(\cos(x+y),\sin(x+y))^T. $$
Thanks to the addition theorems, we can rewrite the result as $(\cos x\cos y-\sin x\sin y,\sin x\cos y+\cos x\sin y)^T$. In other words, we can turn $(3)$ into
$$\tag 4 (a,b)^T\circ_S(c,d)^T=(ac-bd,ad+bc)^T.$$
Remark:
Incidentally, we can identify $\Bbb R^2$ with $\Bbb C$ (by identifying $(1,0)^T$ with $1$ and $(0,1)^T$ with $i$, or more generally $(u,v)^T$ with $u+iv$).
Under this identification, note that $(4)$ simply becomes multiplication:
$$(a+bi)\cdot(c+di)=ac-bd+(ad+bc)i.$$
Thus the group operation on $S$ that we found in order to enforce tat $\phi$ is a homomorphism is not so forced after all and in fact rather natural.
