Significance level

We usually concentrate most on the probability of a Type I error. This is called the significance level of the decision rule.

The worse the consequence of incorrectly rejecting H0, the lower the significance level that should be used, but it must be remembered that reducing the significance level of the test increases the probability of a Type II error.

The choice of significance level should depend on the type of problem, but the worse the consequence of incorrectly rejecting H0, the lower the significance level that should be used. In many applications the significance level is set at 5%.

If the significance level of the test is set to 5% and we decide to reject H0 then we say that H0 is rejected at the 5% significance level.

Illustration

Decision rules for other tests

The critical value for a hypothesis test about a population mean (known standard deviation) with any significance level (e.g. 5% or 1%) can be obtained from the quantiles of normal distributions. For other hypothesis tests, it is possible to find similar critical values from quantiles of the relevant test statistic's distribution

When testing the mean of a normal population when the population standard deviation is unknown, the test statistic is a t-value and its critical values are quantiles of a t distribution.

P-values and decisions

An alternative to a decision rule that is defined in terms of critical values of a test statistic, is to base the decision rule on the p-value from a hypothesis test. A decision rule to accept the null hypothesis when the p-value is greater than 0.05 has 5% significance level. Rejecting it for p-values over 0.01 has significance level 1%.

A decision rule can therefore be found for any kind of hypothesis test provided computer software can find a p-value for the test.

Illustration