Distribution of p-values

Use this diagram to explain that a p-value is evaluated from a sample, so it is a sample statistic that has a distribution.

With µ = 0, take several samples and explain that any value between 0 and 1 is equally likely.

Click on any cross on the right to see the sample that gave that p-value — p-values near zero correspond to samples whose means were, by chance, far from zero.

As more samples are taken, the cumulative distribution function approaches a straight line, also indicative of a rectangular distribution.

When H0 holds, it is still possible to get low p-values.

  • A p-value below 0.05 occurs with probability 0.05
  • A p-value below 0.01 occurs with probability 0.01

Use the slider to change µ to 2.0 and take more samples. Now that HA is true, the p-values tend to be closer to 0.

p-values near 0 are more likely than when H0 is true

Repeat with different values for µ. The further µ is from 0.0, the more tightly clustered the p-values are around 0.