Bayes' Rule at Work: Odds and Evidence

An email containing “prize” is evidence for spam, but the strength of that evidence depends on how often the word occurs in both spam and legitimate mail. Bayes’ rule combines that comparison with the base rate.

Partition the possibilities

For disjoint hypotheses $H_i$ covering all possibilities, the law of total probability is

$P(E)=\sum_iP(E\mid H_i)P(H_i).$

Bayes’ rule reverses a conditional: $P(H_i\mid E)=P(E\mid H_i)P(H_i)/P(E)$ when $P(E)>0$. The denominator makes the posterior probabilities sum to one.

Suppose 20% of mail is spam. A word occurs in 40% of spam and 5% of legitimate mail. Its overall frequency is $0.4(0.2)+0.05(0.8)=0.12$. Among messages containing it, the spam probability is $0.08/0.12=2/3$.

Evidence multiplies odds

For a binary hypothesis H, odds are $O(H)=P(H)/(1-P(H))$. Divide Bayes’ rule for H by the rule for its complement:

$O(H\mid E)=O(H)\frac{P(E\mid H)}{P(E\mid H^c)}.$

The last factor is the likelihood ratio. A ratio above one favours H; below one favours its complement. Odds return to probability by $P=O/(1+O)$. Log-odds turn products into sums, reducing numerical underflow from many tiny probabilities.

Prior spam odds of $1/4$, followed by likelihood ratios 8 and 3, become odds 6 and probability $6/7$. Forgetting the prior gives a different answer.

Conditional independence is an assumption

Naive Bayes assumes that features are independent given the class. It does not assume they are independent without conditioning. Words such as “prize” and “winner” may remain correlated even within spam, so multiplying their evidence can be overconfident.

For binary presence features, a full Bernoulli model includes both present and absent words: $P(x_j\mid C)=p_{jC}^{x_j}(1-p_{jC})^{1-x_j}$. A multinomial word-count model instead uses token frequencies. Our lab uses a deliberately small presence-only evidence model, stated explicitly so its probabilities have a clear interpretation.

Zero estimated probabilities would erase a whole class likelihood. Additive smoothing uses positive pseudocounts; it changes the estimate rather than proving an unseen event impossible. Binary naive Bayes yields a log-odds expression linear in its features, resembling logistic regression. Naive Bayes fits class-conditional distributions; logistic regression fits the conditional class probability directly.

The observation protocol changes the likelihood

In standard Monty Hall, you select one of three doors. A host who knows the prize location always opens another door showing a goat and offers a switch. Switching wins exactly when your initial choice was wrong, with probability $2/3$.

A host who opens another door uniformly at random without knowing the prize creates a different experiment. Conditional on revealing a goat, your original door and the other closed door each have probability $1/2$. The observation “a goat appeared” has different likelihoods under the two protocols. Write those likelihoods before importing a familiar answer.

Read beyond

Read the equation in context

The paper treats junk-mail filtering as a probabilistic decision problem with different costs for different mistakes. The displayed factorisation is the naive Bayes modelling assumption. A posterior probability and the action chosen from it are separate: a costly false positive can justify a higher spam threshold.

Your turn

Build two three-word messages from the vocabulary. Use likelihood ratios to make one strongly favour spam and the other strongly favour legitimate mail.

Coding problems