$\color{brown}{\textbf{The task statement.}}$
Let
$$\vec x=\binom{r\cos t}{r\sin t},\quad \vec y_1=\binom {p\cos\alpha}{p\sin\alpha},\quad \vec y_2=\binom {q\cos\beta}{q\sin\beta},\quad (p,q,r)\in [0,\infty),\quad (\alpha, \beta, t)\in (-\pi,\pi].\tag1$$
If given $\vec y_1=\dbinom ab,$ then
$$p=\sqrt{a^2+b^2},$$
$$\alpha = -\pi\quad\text{if}\quad a=-p,\quad\text{and}\quad
\alpha = 2 \arctan\frac b{p+a} \quad\text{otherwize}.$$
Similarly for $y_2$.
Then the task is to optimize the function
$$f\left(\vec x\right)=\dfrac{1+p^2r^2\cos^2(t-\alpha)}{1+q^2r^2\cos^2(t-\beta)}\tag2$$
in the area
$$r\in[0,\gamma].\tag3$$
$\color{brown}{\textbf{The special cases.}}$
Firstly, let us consider special cases, when $p=0$ or $q=0.$
If $\underline{(p=0)\ \wedge\ (q=0)}$ then the equation $(2)$ does not depend of $x,$ wherein $f\left(\vec x\right)=1.$
If $\underline{(p=0)}$ then the greatest value $f\left(\vec x\right)=1$ achieves if $r=0$ or $t\in\{\alpha\pm\dfrac12\pi,\alpha\pm\frac32\pi\}.$
If $\underline{(q=0)}$ then the greatest value $f\left(\vec x\right)=1+p^2\gamma^2$ achieves if $r=\gamma$ and $t\in\{\alpha, \alpha\pm\pi, \alpha\pm 2\pi\}.$
At the common case, the greatest value of $f\left(\vec x\right)$
achieves at the innner stationary points or in the bounds.
$\color{brown}{\textbf{The inner stationay points.}}$
The inner stationary points can be defined via the system $f'_r = f'_t=0,$ or
\begin{cases}
p^2r\cos^2(t-\alpha)(1+q^2r^2\cos^2(t-\beta))=(1+p^2r^2\cos^2(t-\alpha))q^2r\cos^2(t-\beta)\\[4pt]
p^2r^2\sin2(t-\alpha)(1+q^2r^2\cos^2(t-\beta))
=(1+p^2r^2\cos^2(t-\alpha))q^2r^2\sin2(t-\beta)
\tag4\end{cases}
Solution $r=0$ gives the stationary point
$$f\left(\binom00\right)=1,\tag5$$
else there are
\begin{cases}
p^2\cos^2(t-\alpha)=q^2\cos^2(t-\beta)\\[4pt]
p^2\sin2(t-\alpha)=q^2\sin2(t-\beta),
\end{cases}
\begin{cases}
p\cos(t-\alpha)=\pm q\cos(t-\beta)\\[4pt]
p\sin(t-\alpha)=\pm q\sin(t-\beta),
\end{cases}
\begin{cases}
p=q\\[4pt]
\sin(t-\alpha)\cos(t-\beta)-\cos(t-\alpha)\sin(t-\beta)=0,
\end{cases}
$$p=q,\quad \cos(\alpha-\beta)=0,$$
therefore the case
$$\binom{p}{\alpha}\in\binom{\{q\}}{\{\beta,\beta\pm\pi\}} $$
also gives the result
$$f\left(\vec x\right)=1 \quad\forall \vec x.\tag6.$$
$\color{brown}{\textbf{The bounds.}}$
The bounds are defined by the conditons $(3),$ wherein zero case solution is given by $(5).$
If $\underline{r=\gamma}$ equation $(2)$ takes form of
$$f\left(\binom{\gamma\cos t}{\gamma\sin t}\right)=\dfrac{1+p^2\gamma^2\cos^2(t-\alpha)}{1+q^2\gamma^2\cos^2(t-\beta)},\tag7$$
so the stationary points on the bound defines by the equation
$$f\left(\binom{\gamma\cos t}{\gamma\sin t}\right)'_t=0,$$
$$p^2\sin2(t-\alpha)(1+q^2\gamma^2\cos^2(t-\beta))
=(1+p^2\gamma^2\cos^2(t-\alpha))q^2\sin2(t-\beta),$$
or
$$p^2\sin(2t-2\alpha)(2+q^2\gamma^2(1+\cos(2t-2\beta))
=q^2\sin(2t-2\beta)(2+p^2\gamma^2(1+\cos(2t-2\alpha)),$$
$$p^2(2+q^2\gamma^2)\sin(2t-2\alpha)+p^2q^2\gamma^2\sin(2t-2\alpha)\cos(2t-2\beta)$$
$$=q^2(2+p^2\gamma^2)\sin(2t-2\beta)+p^2q^2\gamma^2\sin(2t-2\beta)\cos(2t-2\alpha)),$$
$$2p^2\sin(2t-2\alpha)-2q^2\sin(2t-2\beta)
+p^2q^2\gamma^2(2\cos(2t-\alpha-\beta)\sin(\beta-\alpha)+\sin(\beta-\alpha))=0,$$
$$2p^2(\sin2t\cos2\alpha-\cos2t\sin2\alpha)
-2q^2(\sin2t\cos2\beta-\cos2t\sin2\beta)$$
$$+p^2q^2\gamma^2\sin(\beta-\alpha)(2\cos2t\cos(\alpha+\beta)+2\sin2t\sin(\alpha+\beta)+1)=0,$$
$$2p^2(2\tau\cos2\alpha -(1-\tau^2)\sin2\alpha)
-2q^2(2\tau\cos2\beta -(1-\tau^2)\sin2\beta)$$
$$+p^2q^2\gamma^2\sin(\beta-\alpha)(2(1-\tau^2)\cos(\alpha+\beta)+4\tau \sin(\alpha+\beta)+\tau^2+1)=0,$$
and unknown value
$$\tau = \tan t\tag8$$
can be found from the square equation
$$\begin{align}
&(2p^2\sin2\alpha-2q^2\sin2\beta+p^2q^2\gamma^2\sin(\beta-\alpha)(1-2\cos(\alpha+\beta)))\tau^2\\
&+4(p^2\cos2\alpha-q^2\cos2\beta+p^2q^2\gamma^2\sin(\beta-\alpha)\sin(\alpha+\beta))\tau\\
&2q^2\cos2\beta-2p^2\cos2\alpha+p^2q^2\gamma^2\sin(\beta-\alpha)(2\cos(\alpha+\beta)+1)=0.
\end{align}\tag9.$$
Looks more suitable to solve equations $(9),(8)$ and check the solutions on the maximum for given values of the five parameters.