Let $A=\begin{bmatrix}a&b\\b&c\end{bmatrix}$, and $z=\begin{bmatrix}x\\y\end{bmatrix}$, then you have:
$$F=ax^2+2bxy+cy^2=z^TAz$$
$$x^2+y^2=1=z^Tz$$
So you can solve this problem instead:
$$\max~~~z^TAz ~~\text{subject to}~z^Tz=1 \tag{1}$$
and
$$\min~~~z^TAz ~~\text{subject to}~z^Tz=1\tag{2}$$
For (1), the value of $\max$ is equal to the largest eigenvalue of $A$, and $z=\begin{bmatrix}x\\y\end{bmatrix}$ is the corresponding eigenvector, and for (2), similarly, the value of $\min$ is equal to the smallest (second) eigenvalue of $A$, and the corresponding eigenvector is the solution for $z=\begin{bmatrix}x\\y\end{bmatrix}$.
Therefore, the only thing that you need to do, is forming the matrix $A=\begin{bmatrix}a&b\\b&c\end{bmatrix}$, and calculating its eigenvalues and eigenvectors.