Part 1: Conditional Probability

This part builds the intuition behind Bayes' theorem, starting from the conditional probability concepts developed in Part 1.

The reader is expected to be familiar with basic probability concepts, conditional probability, independent and dependent events, and the difference between \(P(A|B)\) and \(P(B|A)\).

1. Bayes' Theorem: Turning evidence into belief

We already know how conditional probability works.

\[ P(A \mid B) \]

means:

Given that B has happened, what is the probability of A?

But sometimes we know:

\[ P(A \mid B) \]

and want to find:

\[ P(B \mid A) \]

In other words, we want to reverse the direction of the conditional probability.

That is the basic idea behind Bayes' theorem:

Bayes gives us a way to move from \(P(A \mid B)\) to \(P(B \mid A)\), and vice versa.

The experiment

Let's make this concrete.

We have two boxes:

Box A: 9 red balls, 1 blue ball
Box B: 2 red balls, 8 blue balls

The experiment happens in two steps:

Step 1: We randomly choose one of the two boxes.

Step 2: From the chosen box, we randomly pick one ball.

So the process is:

Choose a box
     ↓
Choose a ball from that box

Now consider Box B.

Direction 1: Box → Ball

Suppose I tell you:

"Box B was chosen."

Now we pick a ball from Box B.

What is the probability that the ball is red?

Box B contains:

2 red
8 blue

So:

\[ P(\text{Red} \mid B) = \frac{2}{10} = 0.2 \]

We know the box, and we are predicting something about the ball.

This is:

\[ P(\text{Ball} \mid \text{Box}) \]

Direction 2: Ball → Box

Now run the same experiment again.

A box is chosen.

A ball is picked.

But this time I tell you:

"The ball picked was red."

Now I ask:

What is the probability that Box B was chosen?

This is:

\[ P(B \mid \text{Red}) \]

We know something about the ball, and we are trying to infer something about the box.

This is:

\[ P(\text{Box} \mid \text{Ball}) \]

The reversal

The experiment itself has not changed.

Choose a box
     ↓
Choose a ball

What changes is the direction of the question.

P(Ball | Box)
       ↓
   Bayes
       ↓
P(Box | Ball)

The first question is:

If I know the box, what can I say about the ball?

The second question is:

If I know the ball, what can I say about the box?

The second question is often the interesting one.

We observe an outcome and want to work backwards to the possible explanation.

That is the basic problem Bayes' theorem solves.

A Worked Example

Box	Red Balls	Blue Balls	Total Balls
Box A	9	1	10
Box B	2	8	10

Each box is equally likely to be selected:

\[ P(A)=P(B)=\frac{1}{2} \]

We will use these numbers in both examples.

Example 1: Find Probability of Red Ball When Box B Is Selected

Suppose Box B is selected.

Box B contains:

2 red balls
8 blue balls
10 balls total

We want to find:

\[ P(\text{Red}\mid B) \]

Using the definition of conditional probability:

\[ P(\text{Red}\mid B) = \frac{\text{Number of red balls in Box B}} {\text{Total number of balls in Box B}} \]

Substitute the values:

\[ P(\text{Red}\mid B) = \frac{2}{10} \]

\[ =\frac{1}{5} \]

\[ =0.2 \]

\[ =20\% \]

Therefore:

\[ \boxed{P(\text{Red}\mid B)=20\%} \]

Example 2: Find Probability of Box B When the Red Ball Is Picked

Suppose the ball picked was red.

We want to find:

\[ P(B\mid\text{Red}) \]

From the definition of conditional probability:

\[ P(B\mid\text{Red}) = \frac{P(B\text{ and Red})}{P(\text{Red})} \]

We can write the joint probability as:

\[ P(B\text{ and Red}) = P(\text{Red}\mid B)P(B) \]

Therefore:

\[ P(B\mid\text{Red}) = \frac{P(\text{Red}\mid B)P(B)} {P(\text{Red})} \]

This is Bayes' theorem:

\[ \boxed{ P(B\mid\text{Red}) = \frac{P(\text{Red}\mid B)P(B)} {P(\text{Red})} } \]

Now we calculate each part.

From the table:

\[ P(B)=\frac12=0.5 \]

and:

\[ P(\text{Red}\mid B) = \frac{2}{10} = 0.2 \]

We also need \(P(\text{Red})\).

A red ball can come from either Box A or Box B:

\[ P(\text{Red}) = P(\text{Red}\mid A)P(A) + P(\text{Red}\mid B)P(B) \]

Substitute the values:

\[ = \left(\frac{9}{10}\right)\left(\frac12\right) + \left(\frac{2}{10}\right)\left(\frac12\right) \]

\[ = 0.9\times0.5+0.2\times0.5 \]

\[ = 0.45+0.10 \]

\[ =0.55 \]

Now apply Bayes' theorem:

\[ P(B\mid\text{Red}) = \frac{(0.2)(0.5)} {0.55} \]

\[ = \frac{0.10}{0.55} \]

\[ = \frac{10}{55} \]

\[ = \frac{2}{11} \]

\[ \approx0.1818 \]

Therefore:

\[ \boxed{P(B\mid\text{Red})\approx18.18\%} \]

So we have converted:

\[ \boxed{P(\text{Red}\mid B)} \]

into:

\[ \boxed{P(B\mid\text{Red})} \]

using Bayes' theorem.

2. Another Example: An Imperfect Test

Let's continue with the mammogram example.

Suppose:

	Cancer	No Cancer
Probability	1%	99%
Positive test	80%	9.6%
Negative test	20%	90.4%

Now suppose the test is positive.

There are two ways this can happen:

Cancer       → Positive
No Cancer    → Positive

True positive

For a true positive, two things must happen:

The person has cancer.
The test is positive given that the person has cancer.

Therefore:

\[ P(\text{Cancer and Positive}) = P(\text{Cancer})P(\text{Positive}\mid\text{Cancer}) \]

\[ = 0.01 \times 0.80 = 0.008 \]

False positive

For a false positive, two things must happen:

The person does not have cancer.
The test is positive given that the person does not have cancer.

Therefore:

\[ P(\text{No Cancer and Positive}) = P(\text{No Cancer})P(\text{Positive}\mid\text{No Cancer}) \]

\[ = 0.99 \times 0.096 = 0.09504 \]

All positive results

A positive result can come from either group.

So:

\[ P(\text{Positive}) = P(\text{Cancer and Positive}) + P(\text{No Cancer and Positive}) \]

\[ = 0.008 + 0.09504 = 0.10304 \]

Now we can finally ask the question we actually care about:

Among all the positive results, what proportion are from people who have cancer?

\[ P(\text{Cancer}\mid\text{Positive}) = \frac{0.008}{0.10304} \approx 7.8\% \]

So although the test is positive 80% of the time when cancer is present, a positive result corresponds to only about a 7.8% probability of cancer in this example.

The difference comes from the fact that positive results can come from both groups, and cancer is relatively uncommon.

3. From the Calculation to Bayes' Theorem

From the previous section:

\[ P(\text{Cancer}\mid\text{Positive}) = \frac{ P(\text{Cancer and Positive}) }{ P(\text{Positive}) } \]

We also know:

\[ P(\text{Cancer and Positive}) = P(\text{Cancer}) P(\text{Positive}\mid\text{Cancer}) \]

Therefore:

\[ P(\text{Cancer}\mid\text{Positive}) = \frac{ P(\text{Cancer}) P(\text{Positive}\mid\text{Cancer}) }{ P(\text{Positive}) } \]

This is Bayes' theorem in the specific case of the mammogram.

Now replace the specific terms with \(A\) and \(B\):

\[ P(A\mid B) = \frac{ P(A)P(B\mid A) }{ P(B) } \]

Or:

\[ \boxed{ P(A\mid B) = \frac{ P(B\mid A)P(A) }{ P(B) } } \]

Notice what Bayes is doing.

We start with:

\[ P(B\mid A) \]

and use it to calculate:

\[ P(A\mid B) \]

But we also need:

\[ P(A) \]

and:

\[ P(B) \]

The important point is that Bayes is not a new probability concept.

It is the conditional probability relationship rearranged so that we can calculate the probability in the direction we want.

4. Prior, Likelihood and Posterior

Now we can give names to the quantities in Bayes' theorem:

\[ P(A\mid B) = \frac{P(B\mid A)P(A)}{P(B)} \]

Term	Name	What does it represent?
\(P(A)\)	Prior	Probability of A before seeing B
\(P(B\mid A)\)	Likelihood	Probability of seeing B if A is true
\(P(B)\)	Evidence	Probability of seeing B overall
\(P(A\mid B)\)	Posterior	Probability of A after seeing B

For the mammogram example:

Term	Name	What does it represent?
\(P(\text{Cancer})\)	Prior	Probability of cancer before the test result
\(P(\text{Positive}\mid\text{Cancer})\)	Likelihood	Probability of a positive test if cancer is present
\(P(\text{Positive})\)	Evidence	Probability of a positive test overall
\(P(\text{Cancer}\mid\text{Positive})\)	Posterior	Probability of cancer after a positive test

The key picture is:

\[ \text{Prior + Evidence} \rightarrow \text{Posterior} \]

with the Likelihood telling us how strongly the evidence fits the situation we are considering.

5. Putting Bayes to Work

A patient gets a positive mammogram.

The question is:

Given this positive result, what is the probability that the patient has cancer?

\[ P(\text{Cancer}\mid\text{Positive}) \]

We know:

Term	Name	Probability
\(P(\text{Cancer})\)	Prior	1%
\(P(\text{Positive}\mid\text{Cancer})\)	Likelihood	80%
\(P(\text{Positive}\mid\text{No Cancer})\)	False positive rate	9.6%

Step 1: Start with the prior

Before seeing the test result:

\[ P(\text{Cancer}) = 0.01 \]

So the starting probability is 1%.

Step 2: Look at the positive result

The test is positive.

But a positive result can happen in both situations:

Cancer       → Positive
No Cancer    → Positive

Step 3: Calculate the true positive probability

The probability of:

Cancer and positive test

is:

\[ P(\text{Cancer})P(\text{Positive}\mid\text{Cancer}) \]

\[ =0.01\times0.80 =0.008 \]

Step 4: Calculate the false positive probability

The probability of:

No cancer and positive test

is:

\[ P(\text{No Cancer})P(\text{Positive}\mid\text{No Cancer}) \]

\[ =0.99\times0.096 =0.09504 \]

Step 5: Find all positive results

A positive result can come from either group:

\[ P(\text{Positive}) = 0.008+0.09504 = 0.10304 \]

Step 6: Among positive results, find the proportion with cancer

\[ P(\text{Cancer}\mid\text{Positive}) = \frac{0.008}{0.10304} \]

\[ \approx 0.0776 \]

Therefore:

\[ \boxed{P(\text{Cancer}\mid\text{Positive})\approx7.8\%} \]

What happened?

We started with:

\[ 1\% \]

Then we received new evidence:

Positive mammogram

After accounting for how the test behaves, our probability became:

\[ 7.8\% \]

So Bayes is a way of updating a probability when new evidence arrives.

The important part is not just the final 7.8%.

It is the process:

Start with what we know
        ↓
Observe new evidence
        ↓
Consider how that evidence can occur
        ↓
Update the probability

6. Base Rate: Why the Starting Point Matters

The mammogram example reveals something important.

The test result was the same, but the final probability depends heavily on where we started.

The prior was:

\[ P(\text{Cancer}) = 1\% \]

That 1% is the base rate: how common the disease is before we see the test result.

Change the base rate

Suppose the same test is used in a population where cancer is much more common:

\[ P(\text{Cancer}) = 20\% \]

Keep the same test:

\[ P(\text{Positive}\mid\text{Cancer})=80\% \]

\[ P(\text{Positive}\mid\text{No Cancer})=9.6\% \]

The test has not changed.

But the answer will change because the starting probability changed.

Using Bayes:

\[ P(\text{Cancer}\mid\text{Positive}) = \frac{(0.80)(0.20)} {(0.80)(0.20)+(0.096)(0.80)} \]

\[ = \frac{0.16}{0.2368} \approx67.6\% \]

So:

Scenario	Prior: \(P(\text{Cancer})\)	Posterior: \(P(\text{Cancer}\mid\text{Positive})\)
Current example: Original mammogram example	1%	7.8%
Another example: Another population, with a higher cancer rate	20%	67.6%

Same test.

Same positive result.

Very different answer.

The reason is simple:

Evidence does not act on a blank slate. It updates what we already believed.

That starting belief is the base rate, and Bayes forces us to take it into account.

7. Why Base Rates Matter

The previous example gives us an important lesson.

A positive test result does not have a fixed meaning.

Its meaning depends partly on how common the condition was before the test.

In the first example:

\[ P(\text{Cancer})=1\% \]

After a positive test:

\[ P(\text{Cancer}\mid\text{Positive})=7.8\% \]

In the second scenario, the starting probability was higher:

\[ P(\text{Cancer})=20\% \]

With the same test and the same positive result:

\[ P(\text{Cancer}\mid\text{Positive})=67.6\% \]

So:

Same test
Same positive result
Different starting probability
        ↓
Different final probability

This starting probability is called the base rate.

The common mistake is to look only at the test result and ignore the base rate.

Bayes does not make that mistake.

It combines:

\[ \text{What we knew before} + \text{What the new evidence tells us} \]

to produce:

\[ \text{What we should believe now} \]

Relevant Link(s)

AI in Context Main Page

An Intuitive Example