Part 4: Naive Bayes
▶ Part 1: Conditional Probability
▶ Part 3: Bayes' Theorem in AI
We have seen how Bayes' theorem lets us update a probability when we get new information.
Now let's turn that idea into a simple machine-learning algorithm called Naive Bays.
Suppose an email arrives:
Congratulations! You have won a FREE prize. Click here.
Is it Spam or Not Spam?
A human might look at the words and immediately think:
Probably Spam.
But how could a computer make that decision?
This is where Naive Bayes comes in.
1. Start With Some Data
Imagine we have 1,000 old emails.
Suppose:
- 600 were Spam
- 400 were Not Spam
So:
and:
Now we look at two words in our new email:
free and prize
Let's see how often these words appeared in our old emails.
Among the 600 Spam emails:
- 240 contained free
- 180 contained prize
Therefore:
and:
Among the 400 Not Spam emails:
- 20 contained free
- 8 contained prize
Therefore:
and:
So:
| Spam | Not Spam | |
|---|---|---|
| "free" | 0.40 | 0.05 |
| "prize" | 0.30 | 0.02 |
So far, everything is straightforward.
But now we hit an important problem.
2. How Many Spam Emails Contain BOTH?
We know:
and:
But what we actually need for our new email is:
And we cannot calculate that from 0.4 and 0.3 alone.
Think about our 600 Spam emails.
We know:
- 240 contain free
- 180 contain prize
But how many contain both?
We don't know.
It could be 180.
It could be 100.
It could be 50.
It could even be 0.
The two numbers:
and:
don't tell us how much the two groups overlap.
This is important:
Knowing how common two things are does not tell us how often they occur together.
To know that, we need to know something about their relationship.
3. This Is Where "Naive" Comes In
So what does Naive Bayes do?
It makes a simplifying assumption:
Once we know the class, pretend the features are independent.
In our example:
If we already know an email is Spam, pretend that knowing whether it contains "free" tells us nothing about whether it contains "prize".
This may not be true.
But let's see what happens if we make that assumption.
If the two features are independent:
So:
Therefore Naive Bayes estimates:
Notice what happened.
We did not discover that 12% of Spam emails contain both words.
We assumed independence, and under that assumption we estimated 12%.
That's the "naive" part.
4. Now Do the Same for Not Spam
Among the Not Spam emails:
and:
Again, assuming independence:
So we have:
and:
The evidence is therefore much more compatible with Spam.
5. Now Bring Bayes Back
So far, we have calculated how well the evidence fits each class.
For Spam:
For Not Spam:
But these are not yet answering our real question.
Our real question is:
In plain English:
Now that we have seen "free" and "prize", how likely is it that this email is Spam?
Bayes' theorem tells us:
We could calculate the denominator.
But notice something important.
If we calculate the corresponding probability for Not Spam, we get:
The denominator is the same in both cases:
So, if our only goal is to decide which class is more likely, we don't need to calculate that common denominator.
We can simply compare the numerators.
For Spam:
For Not Spam:
So we have two competing scores:
Now comes the important question:
Why do we choose the larger number?
Because both scores have the same denominator.
The actual posterior probabilities are:
and:
Dividing both numbers by the same positive number cannot change which one is larger.
So:
means:
Therefore:
So the evidence supports Spam more strongly than Not Spam.
We don't need the exact posterior probabilities just to make the classification.
We only need to know which one is larger.
The key idea is:
Naive Bayes calculates a score for each possible explanation. Since the same denominator applies to every class, we can compare the scores directly and choose the largest one.
6. So What Is Naive Bayes Actually Doing?
It is doing something quite simple.
For each possible class:
- Start with how common the class is.
- Look at the evidence.
- Ask how likely each piece of evidence is for that class.
- Pretend the pieces are independent.
- Multiply the probabilities.
- Compare the resulting scores.
In our example:
while:
So:
and we choose Spam.
That's Naive Bayes.
7. Why Make a "Wrong" Assumption?
This is perhaps the most interesting part.
The independence assumption may be wrong.
In reality, free and prize might be strongly related.
If we had enough data, we could simply count how often they occur together.
But imagine we have thousands of features.
For a text classifier, those features might be thousands of different words.
Now we would need to model relationships between those words:
- free with prize
- free with winner
- free with money
- prize with winner
- prize with money
- winner with money
- and so on...
The number of possible relationships becomes enormous.
Naive Bayes takes a shortcut:
Let's ignore those relationships and pretend the features are independent.
That makes the calculation manageable.
The assumption is simplistic.
But the resulting model can still be surprisingly useful.
8. A Second Worked Example
Let's see what happens when another email arrives.
Suppose the new email contains:
free money winner
We don't need to have seen this exact combination before.
During training, we looked at our old labelled emails and calculated how frequently each word appeared in Spam and Not Spam emails.
Let's make that concrete.
Suppose our training data contains:
- 600 Spam emails
- 400 Not Spam emails
Among the 600 Spam emails:
- 240 contain free
- 120 contain money
- 150 contain winner
Among the 400 Not Spam emails:
- 20 contain free
- 12 contain money
- 4 contain winner
We can now calculate the probabilities.
Where did the Spam probabilities come from?
For free:
For money:
For winner:
So the Spam probabilities we learned during training are:
| Feature | Calculation | Probability |
|---|---|---|
| "free" | \(240/600\) | 0.40 |
| "money" | \(120/600\) | 0.20 |
| "winner" | \(150/600\) | 0.25 |
Where did the Not Spam probabilities come from?
We do exactly the same thing using the 400 Not Spam emails.
For free:
For money:
For winner:
So the Not Spam probabilities we learned during training are:
| Feature | Calculation | Probability |
|---|---|---|
| "free" | \(20/400\) | 0.05 |
| "money" | \(12/400\) | 0.03 |
| "winner" | \(4/400\) | 0.01 |
So after training, our model has learned:
| Feature | Spam | Not Spam |
|---|---|---|
| "free" | 0.40 | 0.05 |
| "money" | 0.20 | 0.03 |
| "winner" | 0.25 | 0.01 |
It also knows the class frequencies:
and:
These are the numbers the model has learned from the training data.
Now the new email arrives:
free money winner
We don't need to have stored this exact combination.
We simply look up the probabilities we already learned.
Calculate the Spam Score
We have:
Using the naive independence assumption:
Substitute the numbers:
So the likelihood of seeing these three features under the Spam class is approximately:
But we must also account for how common Spam was in our original data.
We already calculated:
Therefore:
So:
Calculate the Not Spam Score
Now do exactly the same thing for Not Spam.
We have:
Using the same naive independence assumption:
So the likelihood of seeing these three features under the Not Spam class is:
Now include the prior probability of Not Spam.
We already calculated:
Therefore:
So:
Compare the Two Scores
We now have:
These are scores for two competing explanations of the same email.
Since:
the Spam explanation is much better supported by the evidence.
Therefore:
Notice what happened.
We never had to calculate or store a special probability for:
free + money + winner
That exact combination may never have appeared in our training data.
Instead, during training we learned the probabilities of the individual features:
and the corresponding probabilities for Not Spam.
When a new email arrives, Naive Bayes combines those learned pieces to produce a score.
So the process is:
That is the practical intuition behind Naive Bayes.
9. The Big Idea
Naive Bayes is not saying:
"The features really are independent."
It is saying:
"Let's assume they are independent because that gives us a simple way to calculate."
That's why the word Naive matters.
And this is an important lesson in machine learning:
A model does not have to describe reality perfectly to be useful.
Sometimes a deliberately simplified model is good enough to make a useful prediction.
The progression is:
Bayes gives us the probability framework.
Naive Bayes adds a simplifying assumption.
Together, they give us a simple way to classify new observations based on evidence.