What I think you are missing is that for any point $p$, there is a unique point, $q(p)$, on the line segment $\overline{AB}$ that is nearest to $p$. In general, there is a well-defined notion of distance between two arbitrary non-empty subsets of the plane $X$ and $Y$, say, defined by:
$$
d(X, Y) = \inf \{d(x, y) \mid x \in X, y \in Y\}
$$
(where $\inf$ gives the greatest lower bound of a bounded-below non-empty set of real numbers).
In your case, take $X = \overline{AB}$ and $Y=P$.
Because $P$ is finite there will be at least one $p \in P$ such that $ d(q(p), p) = d(\overline{AB}, \{p\}) = d(\overline{AB}, P)$ and it is the point or points with that property that you are trying to find.
To find $q(p)$ for any given $p$, you first of all find the
orthogonal projection, $p_o$ say, of $p$ onto the infinite extension $\overline{AB}$ (which you can
do with a bit of vector arithmetic involving the dot product operation). $p_o$ is the closest point to $p$ on the infinitely extended line. If $p_o$ lies on $\overline{AB}$, then $q(p) = p_o$; otherwise $q(p)$ is whichever of $A$ and $B$ is nearer to $p$. (Note that $p$ cannot be equidistant from $A$ and $B$ if $p_o$ does not lie on $\overline{AB}$.)
To find the $p \in P$ and the corresponding point $q(p)$ that you are looking for, you just repeat the above construction for every $p \in P$ and then choose a $p$ that minimise $d(q(p), p)$ (there may be more than one such $p$ in general).