We now give two applications of the delta method.
Estimator of a geometric distribution's parameter, π
If \(X \sim \GeomDistn(\pi)\) is the number of independent success-failure trials until the first success, it has probability function
\[ p(x) = \pi (1-\pi)^{x-1} \quad \quad \text{for } x = 1, 2, \dots \]We showed earlier that the method of moments estimator of \(\pi\) and its maximum likelihood estimator are both the inverse of the sample mean,
\[ \hat{\pi} \;\;=\;\; \dfrac 1{\overline{X}} \]Although we cannot easily find the distribution of \(1 /{\overline{X}}\), we do have formulae for the mean and variance of a geometric distribution's sample mean,
\[ E[\overline{X}] = E[X] = \frac 1 {\pi} \spaced{and} \Var(\overline{X}) = \frac {\Var(X)}{n} = \frac{1 - \pi}{n\pi^2} \]Defining \(g(y) = 1 / y\) so that \(\hat{\pi} = g(\overline{X}) = 1 / \overline{X}\) and \(g'(y) = -1 / {y^2}\), we can use the delta method to find the approximate variance of \(\hat{\pi}\).
\[ \Var(\hat{\pi}) \;\;\approx\;\; \left(g'(\mu_{\overline{X}})\right)^2 \Var(\overline{X}) \;\;=\;\; \left(-\pi^2\right)^2 \times \frac{1 - \pi}{n\pi^2} \;\;=\;\; \frac {\pi^2(1-\pi)} n\]The unknown parameter \(\pi\) can be replaced by its maximum likelihood estimate, \(\hat{\pi}\), to get a numerical value for the variance and its square root, the estimator's standard error.
In this example, the delta method gives the same approximate standard error as the one that was obtained earlier using the second derivative of the log-likelihood, but approximations from the two methods are not always equal.
Odds
We have described uncertainty about whether an event will happen using its probability, but an alternative way to describe it is with its odds. Odds are used in gambling but are also used as the basis of some advanced statistical models.
Definition
The odds for an event are the ratio of the probability of the event happening to the probability of it not happening,
\[ \operatorname{odds}(E) \;\;=\;\; \frac{P(E)}{1 - P(E)} \]For example, the odds for getting a head when tossing a coin are 1.0 since there are the same probabilities of getting a head and tail. The odds for getting a "6" when rolling a fair 6-sided die are \(\frac 1 5\) since a "6" has a fifth of the probability of "1-5".
Note that whereas probabilities must be between 0 and 1, the odds of an event can be greater than 1.
Example: Odds of success
In a series of \(n\) independent success/failure trials that each have odds \(\theta\) of success, \(x\) successes are observed. What is the maximum likelihood estimator of \(\theta\)? If \(n\) is large, what is its approximate standard error?
Firstly note that the parameter \(\theta\) is closely related to the usual binomial distribution parameter \(\pi\),
\[ \theta \;=\; g(\pi) \;=\; \frac{\pi}{1 - \pi} \spaced{and} \pi \;=\; \frac{\theta}{1 + \theta} \]Therefore the number of successes, \(X\) has the binomial distribution
\[ X \;\;\sim\;\; \BinomDistn\left(n,\; \frac{\theta}{1 + \theta}\right) \]The log-likelihood is
\[ \ell(\theta) \;\;=\;\; x \log\left(\frac{\theta}{1 + \theta}\right) + (n-x)\log\left(1 - \frac{\theta}{1 + \theta}\right) + K \]whose derivative could be set to zero to find the maximum likelihood estimator of \(\theta\). However it is easier to use our knowledge that the value of \(\pi\) that maximises the likelihood is
\[ \hat{\pi} = \frac x n \]so the corresponding maximum likelihood estimate of \(\theta\) must be
\[ \hat{\theta} = \frac{\hat{\pi}}{1 - \hat{\pi}} = \frac x{n-x} \]Note that this can be infinite if all observed events are successes, \(x = n\).
The variance of the sample proportion, \(\hat{\pi}\), is known to be
\[ \Var(\hat{\pi}) \;\;=\;\; \frac{\pi(1 - \pi)} n \]so we can use the delta method to get the approximate variance of
\[ \hat{\theta} \;\;=\;\; g(\hat{\pi}) \;\;=\;\; \frac{\large \hat{\pi}}{\large 1 - \hat{\pi}} \]We first find
\[ g'(\pi) \;\;=\;\; \frac 1{(1-\pi)^2} \]The delta method gives the approximation
\[ \Var(\hat{\theta}) \;\;\approx\;\; \left(g'(\pi)\right)^2 \Var(\hat{\pi}) \;\;=\;\; \frac {\pi}{n(1 - \pi)^3} \;\;=\;\; \frac{\theta(1 + \theta)^2} n \]The approximate standard error of the estimator is the square root of this,
\[ \se(\hat{\theta}) \;\;\approx\;\; (1 + \theta) \sqrt{\frac{\theta}{n}} \]A numerical value can be found if \(\theta\) in this formula is replaced by \(\hat{\theta}\).