Distribution of p-values

In any hypothesis test,

When the null hypothesis, H0, is true
All p-values between 0 and 1 are equally likely. In other words, the p-value has a distribution whose probability density function has constant height between 0 and 1, \(\text{p-value} \sim \RectDistn(0, 1)\).
When that alternative hypothesis, HA is true
The p-values then have a distribution for which p-values near zero are more likely than p-values near 1. The precise distribution under the alternative hypothesis depends on the specific hypotheses being tested and the true value of the parameter, but it always favours values near 0.

The diagram below shows typical distributions that might be obtained.

P-values and probability

P-values have a rectangular distribution between 0 and 1 when H0 holds. A consequence of this is that the probability of obtaining a p-value of 0.1 or lower is exactly 0.1 (when H0 holds). This is illustrated on the left of the diagram below.

Similarly, the probability of obtaining a p-value of 0.01 or lower is exactly 0.01, etc. (when H0 holds).

P-values are more likely to be near 0 than near 1 if the alternative hypothesis holds


Simulation

To illustrate the distribution of p-values, we will perform a hypothesis test about the parameter \(\mu\) of a \(\NormalDistn(\mu, \sigma^2)\) distribution based on a sample of size \(n=40\). We will test the following hypotheses:

The details of the test will be described in a later section, but we will use a simulation to illustrate the distribution of p-values that would result when the null hypothesis does and does not hold.

When the null hypothesis is true
Initially the simulation is conducted using random samples from a normal distribution in which \(\mu = 0\) and the null hypothesis therefore holds. Click the button Take sample 50 or more times (or hold the button down) to take samples from this distribution and add their p-values to the display on the right.
The diagram on the top right shows the proportion of p-values that are less than any value — an approximation to the cumulative distribution function of the p-values. Approximately 50% of p-values are less than 0.5, 20% are less than 0.2, etc. when the null hypothesis is true.

It is still possible to get small p-values, even if H0 is true. It is always possible to be misled by an 'unlucky' sample!

When the null hypothesis is false
Use the slider to change the distribution's true mean to \(\mu = 1.5\) and repeat. From the diagram on the top right, you should observe that more than 50% of p-values are less than 0.5, more than 20% are less than 0.2, etc. when the alternative hypothesis holds.
 
When \(\mu\) is increased to 3.0 (much further from the null hypothesis value) and the simulation repeated, the p-value is almost certain to be very small.
 

When HA is true, large p-values are often still possible, but are less likely than small p-values. It is again possible to be misled by an 'unlucky' sample!