Consider the two equations
$$\begin{align}
1+s+c+sc&=G\\
1-s-c+sc&=U
\end{align}$$
where $G$ represents a Given number, $U$ represents an Unknown number, and we are using abbreviations $s=\sin\theta$ and $c=\cos\theta$. Summing the two equations gives $2(1+sc)=G+U$, from which we extract
$$2sc=G+U-2$$
Subtracting the two equations and then squaring gives
$$4(s^2+c^2+2sc)=G^2-2GU+U^2$$
If we now make the substitutions $s^2+c^2=1$ and $2sc=G+U-2$, we get
$$4(G+U-1)=G^2-2GU+U^2$$
or
$$U^2-(2G+4)U+G^2-4G+4=0$$
which solves to
$$U=G+2\pm\sqrt{8G}$$
For the problem at hand, $G=5/4$, which gives $U={13\over4}\pm\sqrt{10}$. However, $(1-\sin\theta)(1-\cos\theta)$ is clearly less than $4$, so the only meaningful value for $U$ is the one with the minus sign,
$$(1-\sin\theta)(1-\cos\theta)={13\over4}-\sqrt{10}$$