Target of small errors

Say we have two possible sample statistics that could be used to estimate a population parameter. For example, we might use either the sample mean or median to estimate the centre of a symmetric population distribution.

For each estimator, there is an error

error for mean  =  µ

error for median  =  medianµ

The better estimator will be the one whose estimation error is usually closer to zero.

This corresponds to two characteristics of the error distribution.

Centred on zero

Ideally, we want the error distribution to be centred on zero. Such an estimator is called unbiased.

The estimator whose error distribution is shown on the left above tends to have negative errors — it usually underestimates the parameter that is being estimated.

Small spread

Ideally, we also want error distribution to be tightly concentrated on zero — i.e. to have a small spread.

The estimator whose error distribution is shown on the left tends to have errors that are further from zero.

Standard error

Since most estimators that we will consider are unbiased, the spread of the error distribution is most important. We call the standard deviation of the error distribution the standard error of the estimator.

standard error   =   standard deviation of the error

Ideally, we want estimators with small standard errors.

Note that the standard error is also the standard deviation of the estimator itself,

standard error   =   standard deviation of the estimator

The standard error can be interpreted as either the standard deviation of the estimator or the standard deviation of the estimation error.