As @PaulDuBois commented, the above answer (@hjhjhj57) is incorrect for obtuse angles.
In fact, we can classify the outside of the triangle in three regions:
1) $\alpha\geq 0$ and $\beta<0$, in which case $P$ is closest to either $AB$ or $AC$,
2) $\beta\geq 0$ and $\gamma<0$, in which case $P$ is closest to either $AB$ or $BC$,
3) $\gamma\geq 0$ and $\alpha<0$, in which case $P$ is closest to either $AC$ or $BC$.
The three regions are equivalent under substitution of variables and I will work out case 1:
If $\gamma<0$, and $(P-A)(B-A)>0$, $i.e.$ the projection of $P$ onto $AB$ where vector multiplication is the dot product, then $P$ is closest to $AB$ (verify that this can only be the case if $A$ is obtuse). Then
\begin{eqnarray}
\beta' &=& \text{min}(1, \frac{(P-A)(B-A)}{(B-A)(B-A)} ) \\
\gamma' &=& 0
\end{eqnarray}
Otherwise $P$ is closest to $AC$ with
\begin{eqnarray}
\beta' &=& 0 \\
\gamma' &=& \text{min}(1, \text{max}(0, \frac{(P-A)(C-A)}{(C-A)(C-A)} ) )
\end{eqnarray}
And lastly
\begin{eqnarray}
\alpha' &=& 1 - \beta' - \gamma' \\
P' &=& \alpha'A + \beta'B + \gamma'C
\end{eqnarray}