Testing for a difference between two means

The difference between two groups that is of most practical importance is a difference between their means.

H0 :   μ2μ1  =  0
HA :   μ2μ1  ≠  0

The summary statistic that throws most light on these hypotheses is the difference between the sample means, . Testing therefore involves assessment of whether this difference is unusually far from zero.

As with all other hypothesis tests, a p-value near zero gives evidence that the null hypothesis does not hold — evidence of a difference between the group means.

Example

General properties of p-values

A statistical hypothesis test cannot provide a definitive answer about whether two groups have different means. The randomness of sample data means that p-values are also random quantities.

It is possible to get a small p-value (supporting HA) when H0 is true, and it is possible to get a large p-value (consistent with H0) when HA is true.

There is some chance of being misled by an 'unlucky sample.

If H0 is true
All p-values between 0 and 1 are equally likely. For example, there is a 5% probability of getting a p-value less than 0.05.
If HA is true
The p-value is more likely to be near zero, though there is still some chance of a larger p-value.

Effect of increasing the sample size

If H0 is true
The p-values remain equally likely between 0 and 1.
If HA is true
The distribution of p-values becomes more concentrated near zero, so you are more likely to conclude that the population means are really different.