Let the two endpoints of the segment be $A$ and $B$, and let $P$ be your point.
Write
$$
f(t) = (1-t)A + t B - P
$$
That's the vector between $P$ and a point that's $t$ of the way from $A$ to $B$. (When $t = 0$, you're at $A$; when $t = 1$, you're at $B$.)
Now let
$g(t) = \| f(t) \|^2$, which is the squared length of that vector. You're looking for the value of $t$ that minimizes this. I use the squared length because it makes the algebra nicer.
Do algebra:
\begin{align}
g(t) & = \|f(t)\|^2 \\
&= ((1-t)A + t B - P)^T \cdot (1-t)A + t B - P \\
&= (t (B-A) + A-P)^T \cdot (t (B-A) + (A-P))
\end{align}
Let $v = B-A$ and $u = A - P$ to get
\begin{align}
g(t) & = \|f(t)\|^2 \\
&= (t (B-A) + A-P)^T \cdot (t (B-A) + (A-P))\\
&= (t v + u)^T \cdot (t v + u)\\
&= t^2 \|v\|^2 + 2t (v \cdot u) + \|u\|^2
\end{align}
so
\begin{align}
g'(t) & = 2t \|v\|^2 + 2 (v \cdot u) = 2t (v \cdot v) + 2 (v \cdot u).
\end{align}
Set to zero and solve to get
$$
t = -\frac{v \cdot u}{v \cdot v}.
$$
You compute this value $t$, and if it's between $0$ and $1$, then the closest point is $(1-t)A + t B$. If it's not, you compute $g(0)$ and $g(1)$, and if $g(0)$ is smaller, the answer is $A$; otherwise it's $B$.
The key idea here is that "squared distance to $P$" is a nice function of position along the segment $AB$ (i.e., of the value $t$). You now have a smooth real-valued function $g$ of a real argument $t$ in the unit interval. The min of such a function either occurs at a value $t$ with $g'(t) = 0$ or at one of the two endpoints, $t = 0$ or $t = 1$.
Because this particular function has only one min, you know that if you find a critical point, it's a min, and if it's in the unit interval, you're done -- the values at the endpoints WILL be larger, so you don't need to check them. On the other hand, if the min lies outside the interval, then you have to check whether your function's smaller at one end than at the other.