As you stated, you are looking for cases where
$$\psi_{(\lambda,\xi)}^{'} = \frac{\lambda^2+\xi^2}{\lambda^2-\xi^2-2\lambda\xi} \tag{1}\label{eq1A}$$
is an integer. You also have the constraint of
$$\lambda^2-\xi^2-2\lambda\xi > 0 \tag{2}\label{eq2A}$$
First, note that since the sum of the powers of $\lambda$ and $\xi$ is $2$ in each term in the numerator and denominator in \eqref{eq1A}, this means that $\psi_{(\lambda,\xi)}^{'}$ is a Homogenous function. This helps to simplify the handling of the results. In particular, all solutions are based on a common multiple of those where $\gcd(\lambda,\xi) = 1$. To see this, let
$$\gcd(\lambda, \xi) = d, \; \lambda = da, \; \xi = db, \; \gcd(a,b) = 1 \tag{3}\label{eq3A}$$
Now, if $(\lambda,\xi)$ is a solution to \eqref{eq1A}, you get
$$\begin{equation}\begin{aligned}
\frac{\lambda^2+\xi^2}{\lambda^2-\xi^2-2\lambda\xi} & = \frac{(da)^2+(db)^2}{(da)^2-(db)^2-2(da)(db)} \\
& = \frac{d^2(a^2+ b^2)}{d^2(a^2- b^2 - 2ab)} \\
& = \frac{a^2+ b^2}{a^2- b^2 - 2ab}
\end{aligned}\end{equation}\tag{4}\label{eq4A}$$
Thus, $(a,b)$ would also be a solution. Conversely, for any $(\lambda,\xi)$ being a solution, then for any positive integer $m$, then $(m\lambda,m\xi)$ is also a solution (e.g., your results show that $(3,1)$, $(6,2)$ and $(9,3)$ are solutions with a result of $25$). As such, we only need to study cases of solutions $(\lambda,\xi)$ where $\gcd(\lambda,\xi) = 1$ to then determine all other solutions as being multiples of these solutions, so consider this to be the case for the rest of this answer.
In \eqref{eq1A}, let
$$\lambda^2-\xi^2-2\lambda\xi = c \tag{5}\label{eq5A}$$
Thus, you have that $c \mid \lambda^2+\xi^2$, so both the numerator and denominator must be congruent to $0$ modulo $c$. In particular, for the denominator, you have
$$\begin{equation}\begin{aligned}
& \lambda^2 - \xi^2 - 2\lambda\xi \equiv 0 \pmod c \\
& \lambda^2 \equiv \xi^2 + 2\lambda\xi \pmod c
\end{aligned}\end{equation}\tag{6}\label{eq6A}$$
Using this, for the numerator you get
$$\begin{equation}\begin{aligned}
& \lambda^2 + \xi^2 \equiv 0 \pmod c \\
& (\xi^2 + 2\lambda\xi) + \xi^2 \equiv 0 \pmod c \\
& 2\xi^2 + 2\lambda\xi \equiv 0 \pmod c \\
& 2\xi(\xi + \lambda) \equiv 0 \pmod c
\end{aligned}\end{equation}\tag{7}\label{eq7A}$$
Consider any prime factor $p$ of $c$. One possibility, as shown by \eqref{eq7A}, is that $p = 2$. Next, consider $p$ to be an odd prime factor instead. Thus, $p \mid \xi$ and/or $p \mid \xi + \lambda$. In the first case, since $p \mid \lambda^2 + \xi^2$, then $p \mid \xi$, but that's not possible since $\lambda$ and $\xi$ are relatively prime. For the second case, you have $\lambda \equiv -\xi \pmod p$, so $\lambda^2 + \xi^2 \equiv (-\xi)^2 + \xi^2 \equiv 2\xi^2 \equiv 0 \pmod p$. Since $p$ is odd, this means $p \mid \xi$. However this is the first case, which was just shown to not be possible.
This means that $p$ can only possibly be $2$, i.e., $c$ is a non-negative power of $2$. Consider that $c$ is a positive power of $2$. Thus, $2 \mid \lambda^2 + \xi^2$ means that both $\lambda$ and $\xi$ must be odd, which also gives that $\lambda^2 + \xi^2 \equiv 1 + 1 \equiv 2 \pmod 4$, so the numerator has only one factor of $2$. Similarly, $4 \mid \lambda^2 - \xi^2$, but $2\lambda\xi$ has only one factor of $2$, so the denominator has only one factor of $2$.
This shows the denominator can only be $1$ or $2$. Note also that
$$\begin{equation}\begin{aligned}
\lambda^2 - \xi^2 - 2\lambda\xi & = \lambda^2 - 2\lambda\xi + \xi^2 - 2\xi^2 \\
& = (\lambda - \xi)^2 - 2\xi^2 \\
& = 1 \text{ or } 2
\end{aligned}\end{equation}\tag{8}\label{eq8A}$$
It being $1$ is a case of Pell's equation. The Wikipedia page explains how to solve these equations. Also, the Transformations section shows how to convert the case where \eqref{eq8A} is $2$ to the case where it's $1$.
There's also a lot of other information about Pell's equations available, including on this site, with Are Pell solutions "unique"??
being a fairly good one which has some basic explanations about how to deal with & solve these equations.
