Distribution of p-values
In any hypothesis test,
The diagram below shows typical distributions that might be obtained.
P-values and probability
P-values have a rectangular distribution between 0 and 1 when H0 holds. A consequence of this is that the probability of obtaining a p-value of 0.1 or lower is exactly 0.1 (when H0 holds). This is illustrated on the left of the diagram below.
Similarly, the probability of obtaining a p-value of 0.01 or lower is exactly 0.01, etc. (when H0 holds).
P-values are more likely to be near 0 than near 1 if the alternative hypothesis holds
Simulation
To illustrate the distribution of p-values, we will perform a hypothesis test about the parameter \(\mu\) of a \(\NormalDistn(\mu, \sigma^2)\) distribution based on a sample of size \(n=40\). We will test the following hypotheses:
The details of the test will be described in a later section, but we will use a simulation to illustrate the distribution of p-values that would result when the null hypothesis does and does not hold.
It is still possible to get small p-values, even if H0 is true. It is always possible to be misled by an 'unlucky' sample!
When HA is true, large p-values are often still possible, but are less likely than small p-values. It is again possible to be misled by an 'unlucky' sample!