Intervals, Tests and Honest Evaluation

Two classifiers disagree on a shared test set. Their accuracy difference is paired evidence: the same examples challenge both models. A valid comparison must preserve that pairing and account for how the models and evaluation were selected.

An interval is a procedure

For iid normal data with known standard deviation σ, $\bar X\pm1.96\sigma/\sqrt n$ has about 95% coverage. If σ is estimated from a normal sample, use the sample standard deviation and a Student t critical value with n−1 degrees of freedom. A normal approximation with an estimated standard error is an asymptotic method, not an exact small-sample result.

The frequentist statement concerns repeated random intervals covering a fixed parameter. After seeing one interval, the parameter is still fixed; assigning it a 95% posterior probability requires a Bayesian model. Reporting the interval width also shows the scale of uncertainty, which a binary significance label hides.

A p-value is conditional on a null model

Choose a null hypothesis, a test statistic and which outcomes count as at least as extreme. The p-value is the probability under the null of such an outcome. It is not the probability the null is true, nor the probability that chance caused the result.

A significance level α controls the test’s Type I error under its assumptions. Power is the probability of rejection under a specified alternative. A nonsignificant result may reflect little evidence or low power; it does not establish equivalence. A tiny p-value with a huge sample can accompany an unimportant effect.

Preserve the pair

For two fixed classifiers on the same independently sampled test examples, count b examples only A gets right and c only B gets right. Under equal discordant win probabilities and conditional on n=b+c, the number of A-only wins is Binomial(n,1/2).

A common two-sided exact McNemar p-value is $\min(1,2P(B\le\min(b,c)))$. If there are no disagreements, there is no evidence of a difference and we use p=1. The large-count approximation in the opening equation can be useful, but the exact calculation avoids its approximation error.

For paired numeric measurements, a sign-flip permutation test needs exchangeability under the null. Shuffling model outcomes independently across subjects breaks the pairing and changes the question.

Bootstrap the unit you actually sampled

A nonparametric bootstrap draws n observations with replacement from the observed n and recomputes the statistic. Repeating this produces an empirical distribution for the statistic; percentile endpoints are one possible interval method. They are approximate and can perform poorly for small samples, severe bias or nonregular statistics.

For paired model differences, resample whole paired examples. For clustered data, resample clusters when that matches the sampling design. The bootstrap cannot recover a missing population or undo a biased dataset.

Multiple comparisons and selection

If m valid null tests each use level α/m, the union bound limits the chance of any false rejection to at most α. This Bonferroni correction does not require independence. With independent tests each at α, the chance of at least one false rejection is $1-(1-\alpha)^m$.

Repeatedly checking results and stopping at the first $p<0.05$ also changes the procedure’s false-positive rate. Predefine a stopping rule or use a method designed for sequential testing. Tune models on validation data, then evaluate the chosen procedure on held-out data. Report effect sizes, uncertainty, seeds and the comparisons considered.

Read beyond

Read the equation in context

Dietterich compares tests for classifier evaluation and discusses McNemar’s test for paired predictions. The approximate statistic uses discordant counts; for a small count, an exact binomial calculation is preferable. Comparing two already-trained models on fixed test data also addresses a narrower question than comparing entire training algorithms.

Your turn

Repeat an interval-building procedure on simulated data with a known mean. Find an interval that is too narrow, repair its standard error, and compare the long-run coverage.

Coding problems