Bad experimental design
We noted earlier that the experimental units often have considerable variability. If the treatments are allocated to experimental units in a way that is associated with their characteristics, these varying characteristics can distort the apparent relationship between the treatments and the response.
In a badly designed experiment, the characteristics of the experimental units act in the same way as lurking variables in observational studies.
Good experimental design
Since variability in the experimental units is usually unavoidable, we cannot prevent their effect on the response. However, in an experiment it is possible to allocate treatments to the experimental units in a way that either eliminates, or at least reduces, the relationship between the treatment, X, and characteristics of the experimental units.
Good experimental design can avoid the potential effect of lurking variables.
Weight gain in calves — a badly designed experiment
Eighteen calves were used in an experiment to asses whether a feed supplement improves their weight gain over a 2-month period. The calves were driven into a barn and the first nine to enter were separated and given the supplement. We will conduct a simulation of this experiment in which the supplement increases weight gain by exactly 5.
The circles on the left of the diagram below represent the 18 calves with their initial weights represented by the colours of the circles.
Click Allocate treatments to simulate the selection of nine of the calves (the first nine to enter the barn) to be given the feed supplement. Now click Run experiment to simulate the weight gains of the calves over two months.
Repeat the experiment a few times and observe that most runs of the experiment estimate the effect of the supplement to be an increased weight gain of between 6 and 11.
Why does the experiment consistently over-estimate the effect of the supplement?
The problem lies in the method of choosing the calves to get the supplement. Larger calves tend to push ahead and enter the barn first, so the calves getting the supplement tend to be larger and we saw in the previous page that larger cows tend to gain more weight even without being given a supplement. (The circles representing the calves have a slight tendency to be bluer.)
Weight gain with ineffective supplement
The reason for the misleading results is clearer in the following simulation in which the supplement has zero effect.
Repeat the simulation a few times and observe that the supplement is usually estimated to have a positive effect even though we know that it has no effect.
Observe that the calves receiving the supplement tend to have higher initial weight — their circles are bluer. The difference between the means of the two groups of calves is caused by the difference in their average weights, not the supplement being tested.
Good experimental design means ensuring that there are no major differences between the two groups of experimental units.
Later pages in this section describe some strategies to follow when designing experiments that avoid the problem of lurking variables.