Standard errors for two distributions

We now derive formulae for the standard errors of the maximum likelihood estimators of the unknown parameters in two standard distributions.

Example: Single binomial value

In a series of \(n\) independent success/failure trials with probability \(\pi\) of success, \(x\) successes were observed. What is the maximum likelihood estimator of \(\pi\) and what are its bias and standard error?

The number of successes, \(X\), has a binomial distribution, so the likelihood function is the binomial probability,

\[ L(\pi) = {n \choose x} \pi^x(1-\pi)^{n-x} \]

The log-likelihood is

\[ \ell(\pi) \;\; = \;\; x \log(\pi) + (n-x) \log(1 - \pi) + K\]

where \(K\) is a constant that does not depend on \(\pi\). The derivative of the log-likelihood is

\[ \ell'(\pi) \;\; = \;\; \frac x {\pi} - \frac {n-x} {1 - \pi} \]

and setting this to zero gives the maximum likelihood estimate

\[ \hat{\pi} \;\; = \;\; \frac x n \]

Bias and standard error

This estimator is unbiased since \(E[X] = n\pi\) for a binomial distribution.

We will now apply the asymptotic formula to find the approximate standard error of this kind of estimator. The second derivative of the log-likelihood is

\[ \ell''(\pi) \;\; = \;\; -\frac x {\pi^2} - \frac {n-x} {(1 - \pi)^2} \]

Writing \(x = n \hat{\pi} \) and also replacing \(\pi\) by \(\hat{\pi}\) in the above equation,

\[ \begin{align} \ell''(\hat{\pi}) \;\; & = \;\; -\frac {n\hat{\pi}} {\hat{\pi}^2} - \frac {n-n\hat{\pi}} {(1 - \hat{\pi})^2} \\ & = \;\; -\frac n {\hat{\pi}} - \frac n {(1 - \hat{\pi})} \\ & = \;\; -\frac n {\hat{\pi}(1-\hat{\pi})} \\ \end{align} \]

The general asymptotic formula for the standard error is

\[ \se(\hat {\pi}) \;\;\approx\;\; \sqrt {- \frac 1 {\ell''(\hat {\pi})}} \;\;=\;\; \sqrt {\frac {\hat{\pi}(1-\hat{\pi})} n} \]

Since \(\hat{\pi}\) is the sample proportion from a binomial distribution, we can also find the variance of the estimator directly from the properties of the binomial distribution,

\[ \Var(\hat{\pi}) \;\; = \;\; \frac {\pi(1-\pi)} n \]

The asymptotic formula is therefore consistent with the usual formula for the standard error.

Our second example involves a random sample from a geometric distribution.

Example: Geometric random sample

If \(\{x_1, x_2, \dots, x_n\}\) is a random sample from a geometric distribution with parameter \(\pi\), what is the maximum likelihood estimator of \(\pi\) and what are its bias and standard error?

The geometric probability function is

\[ p(x\;|\;\pi) = \pi(1-\pi)^{x-1} \]

The log-likelihood is

\[ \ell(\pi) \;\; = \;\; \sum_{i=1}^n {p(x_i\;|\;\pi)} = n \log(\pi) + (\sum x - n) \log(1 - \pi)\]

Its derivative is

\[ \ell'(\pi) \;\; = \;\; \frac n {\pi} - \frac {\sum x - n} {1 - \pi} \]

and setting this to zero gives the maximum likelihood estimate

\[ \hat{\pi} \;\; = \;\; \frac n {\sum x} \;\; = \;\; \frac 1 {\overline x}\]

Bias and standard error

The maximum likelihood estimator \(\hat\pi\) is not unbiased since \(E[\hat{\pi}] \ne \pi\). Although we cannot easily find a formula for the bias, we know from the properties of maximum likelihood estimators that \(\hat\pi\) is asymptotically unbiased — its bias decreases as \(n\) increases.

We similarly cannot find an exact formula for \(Var[\hat{\pi}]\), but an approximate standard error can be obtained using the asymptotic formula for standard errors of maximum likelihood estimators. The second derivative of the log-likelihood is

\[ \ell''(\pi) \;\; = \;\; -\frac n {\pi^2} - \frac {\sum x - n} {(1 - \pi)^2} \]

Writing \(\sum x = \large\frac n {\hat{\pi}} \) and replacing \(\pi\) by \(\hat{\pi}\),

\[ \begin{align} \ell''(\hat{\pi}) \;\; & = \;\; -\frac n {\hat{\pi}^2} - \frac {{\large\frac n {\hat{\pi}}}-n} {(1 - \hat{\pi})^2} \\ & = \;\; -\frac n {\hat{\pi}^2(1-\hat{\pi})} \\ \end{align} \]

Applying the general asymptotic formula for the standard error

\[ \se(\hat {\pi}) \;\;\approx\;\; \sqrt {- \frac 1 {\ell''(\hat {\pi})}} \;\;=\;\; \sqrt {\frac {{\hat{\pi}}^2(1-\hat{\pi})} n} \]