Let $AP=:p\geq0$, $AQ=:q\geq0$. Then we have to mnimize
$$f(p,q):=p^2+q^2-2pq\cos\alpha$$
under a condition of the form $g(p,q):=pq-C=0$ for some $C>0$. As $p\to0+$ enforces $q\to \infty$ and therefore $f(p,q)\to \infty$, and vice versa, it follows that the minimum of $f$ is taken for certain $p>0$, $q>0$ found by using Lagrange's method.
We therefore have to set up the auxiliary function
$$\Phi(p,q,\lambda):=f(p,q)-2\lambda g(p,q)$$ and solve the system
$$\eqalign{\Phi_p&=2p-2q\cos\alpha-2\lambda q=0 \cr
\Phi_q&=2q-2p\cos\alpha-2\lambda p=0 \cr}\tag{1}$$
together with $g(p,q)=0$ for $p$ and $q$. Now $(1)$ implies
$$p=(\lambda+\cos\alpha)q\quad\wedge\quad q=(\lambda+\cos\alpha)p\ ,$$
hence $p=q$.
It follows that you have to choose $p=q$ such that the cut off triangle just has half the area of the original triangle, i.e.,
$$p=q=\sqrt{{|AB|\>|AC|\over2}}\ .$$