Thinking about how data arose often suggests a possible model for the values, but this usually involves one or more unknown values called parameters.
The unknown parameters in a family of distributions may be estimated from a random sample, but there may be more than one way to do this from the data.
An estimator is a function of a random sample and therefore has a distribution itself. Its bias is the difference between the distribution's mean and the actual parameter. An unbiased estimator is good.
A good estimator's distribution should also have small spread. Its standard deviation is called the standard error of the estimator.
The mean squared error of an estimator combines its bias and standard error into a single value that can be used to compare competing estimators.
Most estimators that are based on random samples have bias and standard error that tend to zero as the sample size increases. Such estimators are called consistent.
The sample mean and median are compared as estimators of parameters from different families of distributions.
For distributions with a single unknown parameter, a general estimation method chooses its value to make the distribution's mean equal to the mean of a random sample.
The method of moments is illustrated using samples from binomial and geometric distributions.
The likelihood function for a parameter expresses the probability of getting the observed data for different values of the parameter. It is a function of the parameter and treats the observed data as constants.
The parameter value that maximises the likelihood is called its maximum likelihood estimate. It is usually where the derivative of the likelihood function is zero. The value is usually easiest to find by solving the equation that sets the derivative of the log-likelihood to zero.
Maximum likelihood is illustrated using samples from binomial and geometric distributions.
Maximum likelihood estimators cannot be beaten in large samples. They are asymptotically unbiased and normally distributed, and there is an approximate formula for their variance.
A numerical approximation to the standard error of a maximum likelihood estimator can be found from the second derivative of the log-likelihood.
The standard error is found for the probability of success, firstly when estimated from a single binomial proportion, and then when estimated from a random sample from a geometric distribution.
Maximum likelihood requires the solution to an equation involving the derivative of the log-likelihood. The Newton-Raphson algorithm is an iterative algorithm to solve an equation such as this.
A new type of discrete distribution called the log-series distribution is described. The Newton-Raphson algorithm is used to find the maximum likelihood estimate of the distribution's unknown parameter.
The standard error of the maximum likelihood estimator can be easily found from the iterations of the Newton-Raphson algorithm.
A point estimate (single value) may be far from the actual parameter. An interval estimate is a range of values that gives an indication of accuracy.
Many estimators are approximately normally distributed, unbiased, and their standard error can be found. This allows us to sketch the approximate distribution of the estimation error.
An approximate 95% confidence interval can be found as the estimate ± 1.96 standard errors. Replacing 1.96 with other constants gives intervals with other confidence levels.
Examples are given in which parameters in geometric and log-series distributions are estimated with confidence intervals.
A 95% confidence interval has probability 0.95 of including the true parameter value. However from a single random sample, we cannot tell whether the CI will actually be one of these lucky ones.