1

I started to study Machine Learning, but in the book I am reading there is something I don't understand.

I am a total beginner in terms of Machine Learning, and I am just trying to read as much content I can.

I simply don't understand the purpose of bias and threshold. In the book I am reading, they take the easy example of a bank that could accept or reject credits.

To do so, inputs are given to the perceptron through a training set (vector $x$), weights are generated (vector $w$), and the output is always -1 or +1. I perfectly understand this, and how to update the weights to gain accuracy.

But right after, they introduce this notion of threshold, saying "if the applicant passes the threshold, credit is approved; if not, credit is denied"

As I saw multiples times on internet, they put a bias at $x_0$ with a weight $w_0 = 1$. This bias seems to be -threshold

Can someone explain to me this notion of Threshold / Bias ? I thought the perceptron was all about outputting -1 or 1 for a given input vector (if linearly separable)

mephistolotl
  • 663
Scaraux
  • 111

1 Answers1

1

In terms of linear separability: using a bias allows the hyperplane that separates the feature space into two regions to not have to go through the origin. Without a bias, any such hyperplane would have to go through the origin, and that may prevent the separability we want.

Simple example: suppose we have two inputs $x$ and $y$ that can take on the values $0$ or $1$, and we want the output to be a $1$ when both inputs are $1$ (an $\land$ logic circuit, basically). This means that the separating hyperplane cannot go through the origin of the feature space (where $x$ and $y$ are $0$). But without using a bias (or, equivalently, by having a threshold of $0$), you can't move the hyperplane (the set of points $(x,y)$ for which $x\cdot w_1 + y \cdot w_2=0$ given some weights $w_1$ and $w_2$) away from the origin, because for any weights, the origin is a solution to this equation.

Bram28
  • 100,612
  • 6
  • 70
  • 118
  • This actually makes sense, you are right. But what value should we give to the bias at the beginning ? – Scaraux Oct 12 '17 at 04:06
  • @MalteseFalcon You can just set the bias to $1$ ... since it gets multiplied by its attached weight, that weight will effectively determine where the hyperplane crosses the axes, and the weight can be changed by the learning algorithm. – Bram28 Oct 12 '17 at 04:33
  • I now understand the purpose of the bias. But can you tell me why it is closely related to the threshold ? – Scaraux Oct 12 '17 at 04:34
  • 1
    @MalteseFalcon Without a threshold (and without a bias) we're back to the $x\cdot w_1 + y \cdot w_2=0$ equation, which forces the hyperplane to go through the origin. But with a threshold, the hyperplane is the solution to $x \cdot w_1 + y \cdot w_2 = threshold$ – Bram28 Oct 12 '17 at 04:38
  • So we swap threshold to the left, and add it in our weights vector, right ? It is now called the bias ? – Scaraux Oct 12 '17 at 04:40
  • @MalteseFalcon Yes, that's how you can think about it! – Bram28 Oct 12 '17 at 04:43