Part 1: Conditional Probability
▶ Part 1: Conditional Probability
▶ Part 3: Bayes' Theorem in AI
Probability and Bayes are at the heart of how AI deals with uncertainty, evidence, and prediction.
This series builds the intuition behind conditional probability and Bayes' theorem, without getting lost in unnecessary mathematical machinery.
The reader is expected to be familiar with basic probability concepts, such as events, outcomes, sample spaces, and probability distributions.
1. Conditional Probability: Probability AFTER we learn something
Probabilitye BEFORE we know anything
Suppose we roll a six-sided die.
Before we know anything about the result, the possibilities are:
What is the probability of rolling a 2?
Probabilitye AFTER we know anything
Now suppose someone tells us:
"The number is even."
That information changes the situation.
We can immediately eliminate:
We are now left with:
Among these three possibilities, only one is 2.
So the probability of rolling a 2, given that we know the number is even, is:
Notice what happened.
BEFORE receiving the information:
AFTER receiving the information:
Nothing about the die changed.
Our information changed. And because our information changed, the probability changed. This is idea behind conditional probaility.
The basic idea of conditional probaility
Conditional probability is simply:
What is the probability of A, now that I know B is true?
We write this as:
Read it as:
"The probability of A given B."
The vertical bar | means given that.
So:
means:
"What is the probability of getting 2, given that I already know the number is even?"
The important shift
The easiest way to think about conditional probability is not as a new kind of probability.
It is ordinary probability calculated over a restricted set of possibilities.
Initially, we had:
After learning that the number is even:
The information has effectively changed the universe we are considering.
That simple idea is the foundation for everything that follows.
And it leads directly to the question that causes most of the confusion:
How exactly do we calculate the probability once the possibilities have been restricted?
Forget the possibilities that are no longer possible. Work only with the possibilities consistent with what we now know.
2. Worked example of Conditional Probability
The formula is:
At first glance, this looks like one of those formulas that was designed to make a simple idea look complicated.
It is actually saying something very simple:
Once I know B has happened, I only care about the world where B is true.
The denominator tells us the size of that new world.
Back to the die example
We already saw this example:
1 2 3 4 5 6
Suppose we learn that the number is even.
Our new universe is:
2 4 6
We want the probability that the number is 2.
There is:
1 possibility we want (i.e. 2)
3 possibilities we are considering (i.e. 2,4,6)
So:
Using the conditional probability formula
Now let's put the actual values into every part of the formula defined above, and we should get the same result as above (i.e 0.33) .
The conditional probability formula is:
=
Lets find values of each part:
The probability of rolling an even number is:
Now consider the intersection:
This means:
The set of numbers in 2 AND even (2 AND {2,4,6}) is only one number i.e. 2
And the probability of rolling a 2 is:
Now substitute the actual values into the conditional probability formula:
Therefore:
That is really all the formula is saying.
3. Independent vs. dependent events
Now we have an important question.
Does knowing one event happened always change the probability of another event?
No.
Sometimes the information changes the probability.
Sometimes it does absolutely nothing.
That gives us the idea of dependence and independence.
Dependent events
Suppose we have a bag containing:
We pick one ball and do not put it back.
The probability of picking a red ball on the first draw is:
Suppose we pick a red ball.
Now there are only:
left.
The probability of getting a red ball on the second draw is now:
The first draw changed the second probability.
The events are therefore dependent.
Independent events
Now imagine tossing a coin twice.
The first toss is:
Does that change the probability of Heads on the second toss?
No.
The second toss still has:
The first toss gives us no useful information about the second toss.
The events are independent.
The key idea
Two events are independent when knowing that one happened does not change the probability of the other.
In other words:
If knowing \(B\) happened changes the probability of \(A\), then the events are dependent.
So the simplest way to think about independence is:
Does knowing one thing tell me anything about the other?
If yes, they are dependent.
If no, they are independent.
One important trap
Independence does not mean that two events cannot happen together.
For example, getting Heads on the first coin toss and Heads on the second toss can obviously happen together.
Independence simply means:
The outcome of one does not affect the probability of the other.
This distinction becomes very important later when we start looking at probabilities of multiple events happening together.
Independent vs. Dependent: The formulas
| Independent events | Dependent events |
|---|---|
| ### Independent vs. Dependent: The formulas |
| Independent | Dependent |
|---|---|
| \(P(A \mid B) = P(A)\) | \(P(A \mid B) = \frac{P(A \cap B)}{P(B)}\) |
| B does not change the probability of A. | B changes the probability of A. |
4. The difference between \(P(A \mid B)\) and \(P(B \mid A)\)
This is one of the easiest things to confuse in probability.
Consider this statement:
"If someone is a doctor, they are likely to have studied medicine."
That is:
Now reverse it:
"If someone studied medicine, they are likely to be a doctor."
That is:
These are not the same probability.
The information after the | tells us what we already know.
So:
means:
"What is the probability of A, given that I know B?"
while:
means:
"What is the probability of B, given that I know A?"
The direction matters.
A simple example
Suppose a company has 100 employees.
60 employees are developers
40 employees are not developers
20 developers are managers
5 non-developers are managers
So there are 25 managers in total.
Now ask:
What is the probability that an employee is a manager, given that they are a developer?
We are looking only at the 60 developers.
Among them, 20 are managers.
Therefore:
Now reverse the question:
What is the probability that an employee is a developer, given that they are a manager?
We are now looking only at the 25 managers.
Among them, 20 are developers.
Therefore:
So:
but
Same people. Same data.
Different question, different probability.
The easiest way to remember it
Look at what comes after the bar.
means:
Start with B. Among B, how many are A?
And:
means:
Start with A. Among A, how many are B?
This distinction is extremely important because Bayes' theorem is largely about moving from one direction to the other.
That is where we go next. ▶ Part 2: Bays' Theorem