Maximum Likelihood: Fitting Models to Data

A language model assigns conditional probabilities to the next word. Training can choose parameters that make the observed sequence more probable. The same principle yields least squares for Gaussian observations and cross-entropy for classification.

Reverse what is fixed

A probability model $p_\theta(x)$ varies x while θ is fixed. A likelihood $L(\theta;x)$ uses the same expression with the observed x fixed and θ varying. It need not integrate to one as a function of θ. A continuous-data likelihood is a density value, not the probability of one exact observation.

For iid observations, $L(\theta)=\prod_i p_\theta(x_i)$. Maximising it is equivalent to maximising $\ell(\theta)=\sum_i\log p_\theta(x_i)$ because log is strictly increasing. Working with logs also avoids multiplying many tiny values.

Derive three estimators

For s successes in n Bernoulli trials,

$\ell(p)=s\log p+(n-s)\log(1-p),\qquad \hat p=s/n.$

Differentiate to get $s/p-(n-s)/(1-p)=0$. If s=0 or s=n, the maximum is at the boundary and must be checked separately; an interior derivative equation is insufficient.

For exponential waiting times with rate λ, $\ell(\lambda)=n\log\lambda-\lambda\sum_i x_i$, giving $\hat\lambda=n/\sum_i x_i$ when the total waiting time is positive.

For normal data with unknown mean and variance, differentiating gives

$\hat\mu=\bar x,\qquad \hat\sigma^2_{MLE}=\frac1n\sum_i(x_i-\bar x)^2.$

A zero empirical variance creates a degenerate boundary case if the model requires strictly positive variance.

Likelihood and bias are different questions

An estimator is unbiased if its expectation over repeated datasets equals the true parameter. An MLE need not have this property.

The identity

$\sum_i(X_i-\bar X)^2=\sum_i(X_i-\mu)^2-n(\bar X-\mu)^2$

shows why the normal MLE variance is biased. For iid finite-variance data, taking expectations gives $n\sigma^2-n(\sigma^2/n)=(n-1)\sigma^2$. Thus division by n has expectation $(n-1)\sigma^2/n$, and division by n−1 gives an unbiased variance estimator when n>1. Unbiasedness alone does not guarantee lower mean squared estimation error.

Familiar losses from observation models

Suppose $y_i=f_\theta(x_i)+\epsilon_i$ with independent Gaussian errors of fixed variance σ². The negative log-likelihood is a constant plus $\sum_i(y_i-f_\theta(x_i))^2/(2\sigma^2)$. With σ fixed, maximising likelihood is equivalent to minimising squared error.

For a categorical label y, the negative log-likelihood is $-\log p_\theta(y\mid x)$, equal to one-hot cross-entropy. For soft target distributions, cross-entropy is an expected categorical negative log-likelihood.

These equivalences depend on the noise model. Heavy-tailed errors may motivate a different likelihood. A likelihood can be maximised accurately even when the model is misspecified; the fit does not certify that its assumptions match the data.

Read beyond

Read the equation in context

The paper learns distributed word representations through a probabilistic language model. The displayed objective keeps its log-likelihood term and explicitly omits regularisation. Sequence factorisation uses conditional probabilities; it does not require adjacent words to be independent.

Your turn

Move the model parameters while the data stays fixed. Compare each observation’s log-likelihood contribution, then locate the maximum for three datasets.

Coding problems