Distribution of p-values
Use this diagram to explain that a p-value is evaluated from a sample, so it is a sample statistic that has a distribution.
With µ = 0, take several samples and explain that any value between 0 and 1 is equally likely.
Click on any cross on the right to see the sample that gave that p-value — p-values near zero correspond to samples whose means were, by chance, far from zero.
As more samples are taken, the cumulative distribution function approaches a straight line, also indicative of a rectangular distribution.
When H0 holds, it is still possible to get low p-values.
Use the slider to change µ to 2.0 and take more samples. Now that HA is true, the p-values tend to be closer to 0.
p-values near 0 are more likely than when H0 is true
Repeat with different values for µ. The further µ is from 0.0, the more tightly clustered the p-values are around 0.