Long page
descriptions

Chapter 4   Estimating a Parameter

4.1   Estimation

4.1.1   Unknown parameters

Thinking about how data arose often suggests a possible model for the values, but this usually involves one or more unknown values called parameters.

4.1.2   Estimators of parameters

The unknown parameters in a family of distributions may be estimated from a random sample, but there may be more than one way to do this from the data.

4.1.3   Bias

An estimator is a function of a random sample and therefore has a distribution itself. Its bias is the difference between the distribution's mean and the actual parameter. An unbiased estimator is good.

4.1.4   Standard error

A good estimator's distribution should also have small spread. Its standard deviation is called the standard error of the estimator.

4.1.5   Mean squared error

The mean squared error of an estimator combines its bias and standard error into a single value that can be used to compare competing estimators.

4.1.6   Consistency

Most estimators that are based on random samples have bias and standard error that tend to zero as the sample size increases. Such estimators are called consistent.

4.1.7   Sample mean or median?

The sample mean and median are compared as estimators of parameters from different families of distributions.

4.2   Method of moments

4.2.1   The method of moments

For distributions with a single unknown parameter, a general estimation method chooses its value to make the distribution's mean equal to the mean of a random sample.

4.2.2   Examples

The method of moments is illustrated using samples from binomial and geometric distributions.

4.3   Maximum likelihood

4.3.1   Likelihood function

The likelihood function for a parameter expresses the probability of getting the observed data for different values of the parameter. It is a function of the parameter and treats the observed data as constants.

4.3.2   Maximising the likelihood

The parameter value that maximises the likelihood is called its maximum likelihood estimate. It is usually where the derivative of the likelihood function is zero. The value is usually easiest to find by solving the equation that sets the derivative of the log-likelihood to zero.

4.3.3   Examples

Maximum likelihood is illustrated using samples from binomial and geometric distributions.

4.4   Asymptotic properties of MLEs

4.4.1   Bias, variance and normality

Maximum likelihood estimators cannot be beaten in large samples. They are asymptotically unbiased and normally distributed, and there is an approximate formula for their variance.

4.4.2   Standard error

A numerical approximation to the standard error of a maximum likelihood estimator can be found from the second derivative of the log-likelihood.

4.4.3   Examples

The standard error is found for the probability of success, firstly when estimated from a single binomial proportion, and then when estimated from a random sample from a geometric distribution.

4.5   Numerical methods for MLEs

4.5.1   Newton-Raphson algorithm

Maximum likelihood requires the solution to an equation involving the derivative of the log-likelihood. The Newton-Raphson algorithm is an iterative algorithm to solve an equation such as this.

4.5.2   Log-series distribution

A new type of discrete distribution called the log-series distribution is described. The Newton-Raphson algorithm is used to find the maximum likelihood estimate of the distribution's unknown parameter.

4.5.3   Standard error

The standard error of the maximum likelihood estimator can be easily found from the iterations of the Newton-Raphson algorithm.

4.6   Confidence intervals

4.6.1   Interval estimates

A point estimate (single value) may be far from the actual parameter. An interval estimate is a range of values that gives an indication of accuracy.

4.6.2   Distribution of estimation error

Many estimators are approximately normally distributed, unbiased, and their standard error can be found. This allows us to sketch the approximate distribution of the estimation error.

4.6.3   Confidence intervals

An approximate 95% confidence interval can be found as the estimate ± 1.96 standard errors. Replacing 1.96 with other constants gives intervals with other confidence levels.

4.6.4   Other examples

Examples are given in which parameters in geometric and log-series distributions are estimated with confidence intervals.

4.6.5   Properties of confidence intervals

A 95% confidence interval has probability 0.95 of including the true parameter value. However from a single random sample, we cannot tell whether the CI will actually be one of these lucky ones.