1

enter image description here

I am studying ML through the Bishop book myself, and I could not understand why the distance of a given point $\mathbb{x}$ is given by $y(\mathbb{x})/||w||$. Could you help me on deriving this formula from scratch? Thank you.

John
  • 103
  • 3
    Maybe try deriving it yourself first, and then ask for help? You just need calculus, and the definition of "distance" between a hyperplane and point. – V.S.e.H. Dec 17 '21 at 22:51
  • @V.S.e.H. I agree with you I should do it first. However, what I mean here is I want to know how we can derive that formula by looking at the graph, given I know vectors $w, w_0, x$. I think I'm stuck in the formulating problem, and I'm not familiar with the projection concept – John Dec 18 '21 at 11:32

2 Answers2

1

Suppose the hypeplane is defined by the equation $y(\mathbf{x}) = \mathbf{w}^T \mathbf{x} - w_0=0$.

It holds \begin{eqnarray*} \mathbf{x}-\mathbf{x}_\perp &=& \alpha \mathbf{w} \\ \mathbf{w}^T\mathbf{x} - \mathbf{w}^T\mathbf{x}_\perp &=& \alpha \| \mathbf{w} \|^2 \\ \mathbf{w}^T\mathbf{x} - w_0 &=& \alpha \| \mathbf{w} \|^2 \end{eqnarray*}

The (signed) orthogonal distance of a general point $\mathbf{x}$ from the decision surface is given by definition $$ d(\mathbf{x}) = \alpha \| \mathbf{w} \| = \frac{\mathbf{w}^T\mathbf{x} - w_0}{\| \mathbf{w} \|^2} \| \mathbf{w} \| = \frac{\mathbf{w}^T\mathbf{x} - w_0}{\| \mathbf{w} \|} = \frac{y(\mathbf{x})}{\| \mathbf{w} \|} $$

For the origin, the distance is thus $- w_0/\| \mathbf{w} \|$.

Steph
  • 3,665
1

Let $H = \{ \mathbf{x} \in\mathbb{R}^n ~|~ \mathbf{w}^\top \mathbf{x} = w_0\}$ be a hyperplane (or linear decision surface). Moreover, the Eucledian distance between a point $\mathbf{q}$ and a point $\mathbf{x}$ on the plane is $$\lVert \mathbf{x} - \mathbf{q}\rVert = \sqrt{\sum_{i=1}^n (x_i - q_i)^2 }, \mathbf{x} \in H$$

What we want to find is the shortest such distance, which we can do by solving the equivalent minimization problem \begin{eqnarray} \min_\mathbf{x} f(\mathbf{x})= \frac{1}{2} \lVert \mathbf{x} - \mathbf{q}\rVert^2,\\\text{s. t.} \\ \mathbf{w}^\top\mathbf{x} = w_0. \end{eqnarray}

Using Lagrange multipliers, we can solve this by solving the system: \begin{eqnarray} \nabla f(\mathbf{x}) = \lambda \mathbf{w},\\ \mathbf{w}^\top\mathbf{x} = w_0. \end{eqnarray}

Can you take it from here?

V.S.e.H.
  • 2,724