As $x,y$ goes to infinity, profit goes to negative infinity. So, the maximum exists. We need to check the critical values. As economics only cares about positive numbers, critical values are $x=0,y=0$ and the values that make the partial derivatives $0$.
For $x=0$, the profit is
$$p(y)=1200-10y^2+96y$$
$p'(y)=96-20y$, thus, $p$ will be maximum at $y_0=24/5$. Plugging this into $p$, profit will be $\frac{7152}5=1430.4$
For $y=0$, the profit is
$$q(y)=1200-6x^2-8x$$ which is definitely decreasing for $x\ge0$. So, its maximum will be at $x_0=0$. Plugging this into $q$, the profit will be $1200$.
Let
$$f(x,y)=1200-6x^2-8x+4xy-10y^2+96y$$
Then, $$\frac{\partial f}{\partial x}=-12x-8+4y\qquad\frac{\partial f}{\partial y}=-20y+96+4x$$
In order to optimize $f$ we need to make the partial derivatives zero. Let $(x_0,y_0)$ be the tuple that makes $f$ maximal. We need to solve
$$4y-12x=8\implies y-3x=2\qquad (1)$$
$$20y-4x=96\implies 5y-x=24\qquad (2)$$
So,
$$(2)-5(1)\implies14=24-5\cdot2=(5y_0-2x_0)-5(y_0-3x_0)=5y_0-x_0-5y_0+15x_0=14x_0$$
Thus, $x_0=1$. Plugging this into (1), we have
$$y_0=3x_0+2=5$$
For the second order condition,
$$\frac{\partial^2 f}{\partial x^2}=-12\qquad\frac{\partial^2 f}{\partial y\partial x}=4\qquad \frac{\partial^2 f}{\partial y^2}=-20$$
So, the Hessian matrix is
$$\begin{pmatrix}
-12&4\\
4&-20
\end{pmatrix}$$
, which is clearly negative-definite; because, top-left entry is negative and the determinant $(-12)\cdot(-20)-4\cdot4=240-16=224>0$. So, the critical value, indeed gives a local maximum.
So, $(1,5)$ maximizes the profit. Plugging these two values in gives
$$f(x,y)=1436$$
Among the three possible profits, the maximum is obtained for $(x_0,y_0)=(1,5)$, which is $1436$.