Are the two population means equal?
Testing whether the two underlying population means are equal is done by assessing whether the difference between the sample means is more than could be expected by chance.
As with all other hypothesis tests, a p-value near zero gives evidence that the null hypothesis does not hold — evidence of a difference between the group means.
The p-value is the probability of getting sample means as different as those observed if the two groups really had the same population means.
Examples
Properties of p-values
It is important to understand that a statistical hypothesis test cannot provide a definitive answer about whether two groups have different means. The randomness of sample data means that:
We now demonstrate the properties of p-values with a simulation.
Effect of increasing the sample size
There are two types of possible error when reaching a conclusion with a hypothesis test. It is instructive to note how increasing the sample size affects the probabilities of these two types of error.