Target of small errors
Consider two possible sample statistics that could be used to estimate the centre of a symmetric population distribution — the sample mean and sample median. For each, there is an error,
error for mean = - µ
error for median = median - µ
The best estimator will be the one whose estimation erro is usually "closer to zero". This corresponds to two desirable characteristics of the error distribution.
Centred on zero
Ideally, we want the error distribution to be centred on zero. Such an estimator is called unbiased.
Sample means and proportions are unbiased estimators of the corresponding population parameters.
Small spread
Ideally, we also want error distribution to be tightly concentrated on zero — i.e. to have a small spread.
We call the standard deviation of the error distribution the standard error of the estimator. We ideally want an estimator with a small standard error.
standard error = standard deviation of the error
Note also that
standard error = standard deviation of the estimator
so a good estimator is one with a small standard deviation.