Choice between paired data or two independent samples
It is sometimes possible to answer questions about the difference between two means by collecting two alternative types of data.
Which experimental design is better?
If the individuals in the 2 groups can be paired so that the pairs are relatively similar, a paired design gives more accurate results.
Car repair costs from two garages
Consider an insurance company that is investigating whether Garage B is over-charging for car repairs. Data should be collected to compare the average estimates for repairs from Garage B and another garage, Garage A.
Simulation
We will conduct a simulation based on a pool of 20 cars. In the simulation, all repair estimates are normally distributed with standard deviation σ = $120, but with means shown in the table below
Mean repair estimate, µ ($) | |||||
---|---|---|---|---|---|
Car | Garage A | Garage B | |||
|
|
|
Note that Garage B over-charges by $200 on average for each car.
Two independent samples
We first simulate an experiment in which 10 cars are randomly selected to be sent to Garage A, and the other 10 cars are assessed by Garage B.
A 95% confidence interval for the over-charging (difference between the mean estimates from the two garages) is shown, and the p-value for a 2-tailed test for a difference is also given.
Repeat the simulation several times and observe from the p-values that:
The 2-sample test rarely gives evidence that Garage B over-charges — the p-value is usually over 0.05.
Click Show paired values to see the (unobserved) data that would have been obtained if all cars had been assessed by all garages.
Paired data
We next simulate an experiment in which 10 cars are randomly selected and are assessed by both Garage A and Garage B.
A 95% confidence interval of the over-charging is again shown, this time based on the differences between the estimates in the pairs. The p-value for a 2-tailed paired t-test for a difference is also given.
Repeat the simulation several times and observe from the p-values that:
The paired t-test usually finds strong evidence that Garage B over-charges.
Matched pairs in experiments
It is often impossible to repeat the same experiment twice with the same experimental units. In the Car Repair Costs example, if the comparison was to be made of actual repair costs rather than estimates, it would be impossible to obtain measurements for the same car from both garages.
However it is often possible to group together the experimental units into pairs that are similar in some way. These are called matched pairs. The two experimental units in each pair are randomly assigned to the two treatments.
In each example, pairing gives more accurate estimates than randomly allocating the units (cars, students or fields) to the two treatments if the units in the pairs are more similar to each other than to units in other pairs.