Distribution of p-values

In any hypothesis test,

When the null hypothesis, H0, is true
All p-values between 0 and 1 are equally likely. In other words, the p-value has a distribution whose probability density function has constant height between 0 and 1, \(\text{p-value} \sim \RectDistn(0, 1)\).
When that alternative hypothesis, HA is true
The p-values then have a distribution for which p-values near zero are more likely than p-values near 1. The precise distribution under the alternative hypothesis depends on the specific hypotheses being tested and the true value of the parameter, but it always favours values near 0.

The diagram below shows typical distributions that might be obtained.

P-values and probability

P-values have a rectangular distribution between 0 and 1 when H0 holds. A consequence of this is that the probability of obtaining a p-value of 0.1 or lower is exactly 0.1 (when H0 holds). This is illustrated on the left of the diagram below.

Similarly, the probability of obtaining a p-value of 0.01 or lower is exactly 0.01, etc. (when H0 holds).

P-values are more likely to be near 0 than near 1 if the alternative hypothesis holds