Significance level

The decision rule affects the probabilities of Type I and Type II errors and there is always a trade-off between these two probabilities. Selecting a critical value to reduce one error probability will increase the other.

In practice, we usually concentrate on the probability of a Type I error. The decision rule is chosen to make the probability of a Type I error equal to a pre-chosen value, often 5% or 1%. This probability is called the significance level of the test. In many applications the significance level is set at 5%.

P-values and decisions

Since the distribution of a test statistic is always fully known when H0 is true, the critical value for any significance level can be found as a quantile of this distribution. For example,

The critical value for a test depends on the distribution of its test statistic (e.g. a normal, t or other distribution). However the decision rule can equivalently be based in a simple way on the p-value for the test. For example, for a test with significance level 5%, the decision rule can always be expressed as:

Decision
p-value > 0.05     accept H0
p-value < 0.05     reject H0

For a test with significance level 1%, the null hypothesis, H0, should be rejected if the p-value is less than 0.01.

If computer software provides the p-value for a hypothesis test, it is easy to translate it into a decision to accept (or reject) the null hypothesis at any significance level.