Error distribution

Most parameter estimates are unbiased (or at least asymptotically unbiased) and a formula for their standard deviation (or an approximation) can be found. In particular, from the asymptotic properties of maximum likelihood estimators,

\[ \begin{align} E[\hat{\theta}] \;\; &\approx \; \; \theta \\ \se(\hat {\theta}) \;\;&\approx\;\; \sqrt {- \frac 1 {\ell''(\hat {\theta})}} \end{align} \]

Since MLEs are also asymptotically normally distributed, we can find an approximate distribution for the estimation error,

\[ error \;\;=\;\; \hat {\theta} - \theta \;\; \sim \;\; \NormalDistn\left(\mu=0, \;\;\sigma=\se(\hat{\theta})\right) \]

Binomial example

In a series of independent Bernoulli trials with probability \(\pi\) of success, the maximum likelihood estimator of \(\pi\) is the proportion of successes. From the binomial distribution, we have exact formulae for its mean and standard deviation (standard error) and approximate normality in large samples,

\[ \hat {\pi} \;\; \sim \;\; \NormalDistn\left(\pi, \;\;\sigma=\sqrt{\frac {\pi(1-\pi)} n} \right) \]

In the example below, the standard error is used to sketch the approximate distribution of the sampling errors.

From this error distribution, we can conclude that:

Our estimate of the proportion of beginners getting injured during a week of skiing, 0.25, is unlikely to be more than 0.1 from the similar proportion of skiers in general, \(\pi\).