I'm not sure what $Q_{4n}$ is, but it must be generated by the elements $a$ and $b$. If we already know that $f$ is a homomorphism then as soon as we know $f(a)$ and $f(b)$ we know all of $f$. For example, to compute $f(aba^{-3})$ we would do:
$$f(aba^{-3}) = f(a)f(b)f(a)^{-3}$$
Given that we might say: To write down a homomorphism we just need to pick two elements to be $f(a)$ and $f(b)$!
This is not quite right though. The problem is that there may be relations between $a$ and $b$. For example if $a^4 = e$ then
$$e = f(e) = f(a^4) = f(a)^4$$
but it's not always true that $x^4 = e$ so we can't choose any arbitrary $x$ to be $f(a)$.
Now comes the answer to your question. Given a relation, like $a^4 = e$, there are other obvious relations that follow from it, for instance $a^4b = b$ or $a^{-4} = e$. For some groups it's possible to come up with a finite set of relations such that every relation in the group is generated by this finite set. If we know all of the relations between $a$ and $b$ then we just need to make sure that the $x$ and $y$ we choose for $f(a)$ and $f(b)$ satisfy all of these relations. Then the problem of $a^4 = e$ but $f(a)^4 \neq e$ can't happen and the $f$ we get is a well defined homomorphism.
Now, what I've written above is just the idea, not a rigorous proof. For a proof of why you only have to check conditions (1) - (4) you have to first prove that $Q_{4n}$ has a finite presentation as the quotient of a free group on $2$ generators by the relations (1) - (4). The universal property of a free group says that you can specify a homomorphism by just specifying the image of the generators and then the universal property of quotients says that as long as those generators respect the relations the map passes to the quotient and so you get a homomorphism out of $Q_{4n}$.