Power of a test

A decision rule about whether to accept or reject H0 can result in one of two types of error. The probabilities of making these errors describe the risks involved in the decision.

Prob(Type I error)
This is the significance level of the test. The decision rule is usually defined to make the significance level 5% or 1%.
Prob(Type II error)
When the alternative hypothesis includes a range of possible parameter values (e.g. µ ≠ 0), this probability is not a single value but depends on the parameter.

Instead of the probability of a Type II error, it is common to use the power of the test, defined as one minus the probability of a Type II error,

The power of a test is the probability of correctly rejecting H0 when it is false.

When the alternative hypothesis includes a range of possible parameter values (e.g. µ ≠ 0), the power depends on the actual parameter value.

Decision
  accept H0     reject H0  
Truth H0 is true      Significance level =
P (Type I error)
HA (H0 is false)     P (Type II error) Power =
1 - P (Type II error)

Increasing the power of a test

It is clearly desirable to use a test whose power is as close to 1.0 as possible. There are three different ways to increase the power.

Increase the significance level
If the critical value for the test is adjusted, increasing the probability of a Type I error decreases the probability of a Type II error and therefore increases the power.
Use a different decision rule
For example, in a test about the mean of a normal population, a decision rule based on the sample median has lower power than a decision rule based on the sample mean.

In CAST, we only describe the most powerful type of decision rule to test any hypotheses, so you will not be able to increase the power by changing the decision rule.

Increase the sample size
By increasing the amount of data on which we base our decision about whether to accept or reject H0, the probabilities of making errors can be reduced.

When the significance level is fixed, increasing the sample size is therefore usually the only way to improve the power.

Illustration

The following diagram again investigates decision rules for testing the hypotheses

H0 :   μ = 10
HA :   μ > 10

based on a samples from a normal population with known standard deviation σ = 4. We will fix the significance level of the test at 5%.

The top half of the diagram shows the normal distribution of the mean for a sample of size n = 16. Use the slider to increase the sample size and observe that: