Experiments & Scientific Rigor

From hunch to hypothesis

Research starts with a hunch ("warmup seems to help") and becomes science when the hunch is turned into a hypothesis: a specific, falsifiable prediction, stated before you look at the results.

Decide your metric, your comparison and your number of runs in advance. If you choose them after peeking at the results, it becomes frighteningly easy to fool yourself, and the first principle of science, in Richard Feynman's words, is that "you must not fool yourself, and you are the easiest person to fool."

Baselines: compared to what?

A number means nothing on its own. "92% accuracy" is impressive if the previous best was 80% and embarrassing if always predicting the majority class gets 95%. Every claim needs a baseline: a reference point.

Good baselines are:

• Strong and recent: the best existing methods, not a strawman.
• Fairly tuned: given the same hyperparameter budget as your method.
• Matched: the same data, the same splits, similar compute.
• Simple, too: a trivial baseline (majority class, linear model) tells you how hard the task really is.

Ablations: what actually matters?

A method with several new components raises an obvious question: which parts are doing the work? An ablation study answers it by removing one component at a time:

(Illustrative numbers, mean ± standard deviation over 5 seeds.) Reading it like a reviewer: the gating module is doing almost all of the work. The augmentation's effect (0.3 points) is smaller than the run-to-run noise, so we can't claim it helps at all.

Randomness and error bars

Train the same network twice with different random seeds and you'll get different numbers. Randomness enters through initialisation, data order, dropout, augmentation and even non-deterministic GPU kernels. So a single run is a single draw from a distribution, and you need several draws to see its shape.

For $n$ runs with scores $x_1, \dots, x_n$:

The sample standard deviation $s$ describes how much individual runs vary. The standard error $\mathrm{SE}$ describes how precisely you know the mean, and it shrinks as $1/\sqrt{n}$. A 95% confidence interval for the mean is $\bar{x} \pm t^\star \cdot \mathrm{SE}$, where $t^\star \approx 2.78$ for 5 runs, $2.09$ for 20, and tends to $1.96$ as $n$ grows.

Now feel it for yourself. Two methods, one of which is genuinely a little better, but you don't know which or by how much. Gather evidence:

Data leakage: the silent killer

Leakage is any path by which information from your evaluation data sneaks into training or model selection. It makes results look better than they are, often dramatically, and it's one of the most common reasons published results fail to reproduce. Classic forms:

• Duplicates: the same (or nearly the same) example appears in both training and test sets.
• Preprocessing on everything: normalisation statistics, vocabularies or feature selection computed on the full dataset before splitting.
• Label proxies: a feature that secretly encodes the answer, such as a hospital billing code when predicting a diagnosis.
• Time travel: training on the future to predict the past in time-series data.
• Test-set model selection: choosing checkpoints or hyperparameters by test score.
• Contamination: benchmark questions that appeared in a large model's pretraining data.

Put on Reviewer #2's hat

In academic folklore, "Reviewer #2" is the reviewer who finds every flaw. Today, that's you.

Reproducibility

A result you can't reproduce is a result you can't trust, and one nobody can build on. Before you report anything, make sure you could rerun it exactly. Record:

• the code version (for example, the git commit hash);
• the full configuration: every hyperparameter, not just the interesting ones;
• the random seeds;
• the data version and the exact splits;
• the environment: library versions and hardware.

A minimal, reproducible experiment script looks like this:

import json, random, subprocess

import numpy as np
import torch

def set_seed(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # For bit-exact GPU runs you may also need:
    # torch.use_deterministic_algorithms(True)

config = {"lr": 3e-4, "batch_size": 128, "epochs": 20, "seed": 0}
set_seed(config["seed"])
config["git_commit"] = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()

with open("run_config.json", "w") as f:
    json.dump(config, f, indent=2)

# ...train, then save metrics next to run_config.json