0

Using the Naive Bayes formula to classify text I have something like...

$$ P(Cat|Word1) = \frac{P(Word1|Cat) * P(Cat)}{P(Word1)} $$

Using a small example ...

Cat1 = 4 documents
     =  1x word 'Hello'
     =  3x word 'World'

Cat2 = 10 documents
     = 10x word 'Hello'
     =  1x word 'World'

Total of 14 docs. with 2 'categories'

I can then calculate the probability of Cat1 and Cat2

$$ P(Cat1|Hello,World) = \frac{P(Hello|Cat1) * P(World|Cat1) * P(Cat1)}{P(Hello) * P(World)} $$

For category 1

$$ P(Cat1|Hello,World) = \frac{\frac{1}{4} * \frac{4}{4} * \frac{4}{14}}{\frac{11}{14} * \frac{4}{14}} \approx 0.31818 $$

And category 2

$$ P(Cat2|Hello,World) = \frac{\frac{10}{10} * \frac{1}{10} * \frac{11}{14}}{\frac{11}{14} * \frac{4}{14}} = 0.35 $$

But I am struggling to interpret the values been returned,

  • Does it mean that there is a 31% chance of category 1 and 35% chance of category 2?
  • Does it mean that there is a slightly better chance of category 1 vs category 2
  • How much more likely is one category over the other?

How can I interpret the actual values been returned?

Simon Goodman
  • 173

1 Answers1

1

So Bayes Theorem says: $P(A|B) = \frac{P(B|A)P(A)}{P(B)}$ And the Naive Bayes assumes the class conditional $P(B|A)$ is independent so you can have $P(B|A) = \prod P(b_i|A)$

However, the example you give is not Naive Bayes, because you give the exact data and that data does not seem to satisfy the independent class conditional assumption. $P(Hello,World|Cat1) = 0$ but $P(Hello|Cat1)*P(World|Cat1) = 1/4 * 3/4 = 3/16$, clear the 2 are not the same.

Nevertheless if you just use the Bayes Theorem, you should be able to get to the right probability.

Assume your data is: Cat1 = [(World) $\times$ 3, (Hello)]; Cat2 = [(Hello) $\times$ 9, (Hello, World)]

Then use Bayes Theorem (not naive bayes):

$P(Cat1|Hello,World) = \frac{P(Hello,World|Cat1)*P(Cat1)}{P(Hello,World)} = \frac{0*\frac{4}{14}}{\frac{1}{14}} = 0$

(Since we know the exact distribution, we can get $P(Hello,World|Cat1)=0$ and $P(Hello,World)=\frac{1}{14}$)

$P(Cat2|Hello, World) = \frac{P(Hello,World|Cat2)*P(Cat2)}{P(Hello,World)} = \frac{\frac{1}{10}*\frac{10}{14}}{\frac{1}{14}}=1$

So in this case, yes

  • there is 0% chance of Cat1 and 100% chance of Cat2

  • it means it got to be in Cat2

  • 100% more likely

You can interpret the same way in Naive Bayes, but when you use real data to describe the distribution, please check the assumptions.

wenxi
  • 11
  • Sorry I am not sure I get it, P(Cat1|Hello,World) = 0.31 and P(Cat2|Hello,World) = 0.35, (using the Bayes Theorem), I am not sure how you got 0 and 1 using Naive Bayes. Can you explain how you calculate P(Cat1|Hello,World) and P(Cat2|Hello,World) and where I might have gone wrong. – Simon Goodman Nov 05 '16 at 05:44
  • @SimonGoodman I just edited the answer a little bit so the probability calculation is more clear. So first of all the 0.31 and 0.35 you calculated is not using Bayes Theorem. If you use Bayes Theorem you should get 0 and 1, which is the exactly calculation I did in the answer. There are two parts you overlooked. 1) $P(Hello,World|Cat1) \ne P(Hello|Cat1)*P(World|Cat1)$2) – wenxi Nov 05 '16 at 15:39
  • @SimonGoodman 2) $P(Hello,World) \ne P(Hello)*P(World)$. So then, you can't calculate the way you did. The Naive Bayes assumes 1) is equal, but in your distribution, it is not, so you cannot use naive bayes in this problem. – wenxi Nov 05 '16 at 15:43
  • I think the problem is a mix-up on my side between Naive Bayes and Bayes theorem. My formula was for Naive Bayes classifier. In any case, the question about interpreting the numbers still stands. if the result was 0.31 and 0.35, how would those be converted to percentages, (or are they percentages). – Simon Goodman Nov 07 '16 at 07:01
  • Yes, they are percentages (if you mean probabilities). Assume your calculation of 0.31 and 0.35 is right, they can be interpret as: given the document (Hello, World), there is 31% chance it belongs to Cat1, and 35% chance it belongs to Cat2. Sometime we also interpret the 31% as the likelihood of (Hello, World) given Cat1, denote as $L(Hello,World|Cat1)$ – wenxi Nov 07 '16 at 17:12
  • Again, this interpretation is from Bayes Theorem. You need to distinguish Naive Bayes from Bayes Theorem. Basically Naive Bayes is one of the Generative Models. And all (if not, most) generative models use bayes theorem to derive the conditional probability. In real a problem, we most likely won't have the real distribution. So in a generative model, we estimate the posterior probability of the given example which is $P(Hello,World|Cat1)$ and $P(Hello,World|Cat2)$ in your case. Then we classify the example to the biggest posterior probability (in your case Cat2). – wenxi Nov 07 '16 at 17:22
  • In fact, interpreting the numbers has little to do with Naive Bayes. Naive Bayes just specifies a way to get the posterior probability (e.g., $P(Cat1|Hello,World)$). There are many other ways to get the posterior probability, thus many classification models (not necessary generative models). Hopefully, this clear some confusion for your, though there's many things I talked about here. – wenxi Nov 07 '16 at 17:45