We have a $2 \times 2$ matrix. We are given the trace of the matrix as $N$. Also, all elements of the matrix are greater than or equal to $1$. And, the determinant of matrix is $\geq 1$.
QUESTIONS:
If given $N$, what are number of $2 \times 2$ matrices that can be formed in consideration to the given constraints?
for example we have $N=3$ then there are $2$ matrices which satisfy the given constraints i.e
1 1
1 2
and
2 1
1 1
$$A = \left[\begin{array}{cc}x & \frac{x(N-x)-D}{y} \\ y & N - x\end{array}\right]$$ This is the prototype of your matrix where I'm assuming that $N, x, y$ and $D$ are all natural number. It's easy to show that $\det(A) = D$.
A first constraint is: $x \in \{1, \ldots , N-1\}$.
The second constraint is: $D < x(N-x)$. Having fixed $x$, then $D \in \{1, \ldots, x(N-x) - 1\}$
The third constraint is: $y ~\text{ divides } ~x(N-x) - D $
Let's indicate with $\alpha(n)$ the number of divisors of $n$.
Then, the number of such matrices having fixed $N, x$ and $D$ is:
$$\alpha(x(N-x)-D)$$
The number of such matrices having fixed $N$ and $x$ is:
$$\sum_{D=1}^{[x(N-x)-1]}\alpha(x(N-x)-D)$$
Finally, the number of such matrices having fixed only $N$ is: $$\sum_{x=1}^{N-1}\sum_{D=1}^{[x(N-x)-1]}\alpha(x(N-x)-D)$$
The function $\alpha$ is well explained here (denoted with $\sigma_0$)
Using $N=3$, you get:
$$\sum_{x=1}^{2}\sum_{D=1}^{[x(3-x)-1]}\alpha(x(3-x)-D) = $$ $$=\sum_{D=1}^{[1(3-1)-1]}\alpha(1(3-1)-D) + \sum_{D=1}^{[2(3-2)-1]}\alpha(2(3-2)-D)=$$ $$=\sum_{D=1}^{1}\alpha(2-D) + \sum_{D=1}^{1}\alpha(2-D)=$$ $$=\alpha(1) + \alpha(1) = 1 + 1= 2$$
Case $N=4$
$$\sum_{x=1}^{3}\sum_{D=1}^{[x(4-x)-1]}\alpha(x(4-x)-D) = $$ $$=\sum_{D=1}^{2}\alpha(3-D) + \sum_{D=1}^{3}\alpha(4-D) + \sum_{D=1}^{2}\alpha(3-D) = $$ $$=2(\alpha(3-1) + \alpha(3-2)) + \alpha(4-1) + \alpha(4-2) + \alpha(4-3) = $$ $$=2(\alpha(2) + \alpha(1)) + \alpha(3) + \alpha(2) + \alpha(1) = $$ $$=2(2 + 1) + 2 + 2 + 1 = 11$$
$$\left[\begin{array}{cc}1 & 1\\ 1 & 3\end{array}\right], \left[\begin{array}{cc}3 & 1\\ 1 & 1\end{array}\right], \left[\begin{array}{cc}1 & 2\\ 1 & 3\end{array}\right], \left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right], \left[\begin{array}{cc}1 & 1\\ 2 & 3\end{array}\right], \left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right], \left[\begin{array}{cc}2 & 1\\ 1 & 2\end{array}\right], \left[\begin{array}{cc}2 & 2\\ 1 & 2\end{array}\right], \left[\begin{array}{cc}2 & 1\\ 2 & 2\end{array}\right], \left[\begin{array}{cc}2 & 3\\ 1 & 2\end{array}\right], \left[\begin{array}{cc}2 & 1\\ 3 & 2\end{array}\right].$$