Yes, it is the correct way. The only mistake that you made is that you focused on the tangent, rather than focusing explicitly on both the cosine and sine.
$\pi/2 < \theta \leq 3\pi/4 \implies $
- $\cos(\theta) < 0.$
- $\sin(\theta) > 0.$
- $\sin(\theta) \geq | ~\cos(\theta) ~|.$
The above constraints can be somewhat streamlined by
- $~\dfrac{1}{\sqrt{2}} \leq \sin(\theta) < 1.$
- $~\dfrac{-1}{\sqrt{2}} \leq \cos(\theta) < 0.$
However (strangely enough), the first set of constraints above will be easier to deal with, for this problem, because it will allow you to ignore the denominator (explained below).
Starting from scratch:
$$\frac{x + i(y-1)}{x + i(y+1)} \times
\frac{x - i(y+1)}{x - i(y+1)} = $$
$$\frac{[ ~x^2 + (y^2 - 1) ~] + i[-2x]}{\text{some positive real number - who cares}}. \tag1 $$
Note:
If $\displaystyle ~z = \frac{x + iy}{s} ~: ~x,y \in \Bbb{R}, ~s \in \Bbb{R^+},~$ then the standard approach is to re-express $~z~$ as
$$z = \frac{\sqrt{x^2 + y^2}}{s} \times \frac{x + iy}{\sqrt{x^2 + y^2}},$$
so that the argument to $~z~$ is the angle $~\theta~$ that has $~\displaystyle \cos(\theta) = \frac{x}{\sqrt{x^2 + y^2}}, ~\sin(\theta) = \frac{y}{\sqrt{x^2 + y^2}}.$
In this problem, since it is sufficient to interrogate the signs of $~\cos(\theta),~$ and $~\sin(\theta),~$ and the relative sizes of $~\cos(\theta)~$ and $~\sin(\theta),~$ the denominator in (1) above can be ignored.
To satisfy the original constraints, you need:
$[-2x] > 0 \implies x < 0.$
$[x^2 + y^2 - 1] < 0 \implies $
$y^2 < 1 - x^2.$
$~(-2x) \geq 1 - (x^2 + y^2) \implies $
$y^2 \geq 1 - x^2 + 2x.$
Since $~x^2 + y^2 < 1,~$ you must have that
$~-1 < x < 0.$
Then, $~y~$ can be positive or negative, but you must have: