Non-random allocation of treatments

Experiments should always try to randomise allocation of treatments to the available experimental units either overall (a completely randomised design) or within blocks (a randomised block design).

If this allocation is not randomly done, a standard analysis may result in a biased estimate of the differences between the treatments — the experiment's design may be expected to estimate a factor's effect too high or low. (An example was given in the introduction to this e-book.)

Subsequent measurements about the variability of the experimental units (covariates) not only reduce unexplained variation, but also:

Use of covariates in analysing the data can reduce or eliminate bias due to non-random allocation.


Recovering from a badly designed experiment

Eighteen calves were used in an experiment to assess whether a feed supplement improves their weight gain over a 2-month period. The calves were driven into a barn and the first nine to enter were separated and given the supplement. We will conduct a simulation of this experiment in which the supplement increases weight gain by exactly 5.

The circles on the left of the diagram below represent the 18 calves with their initial weights represented by the colours of the circles.

Click Allocate treatments to simulate the selection of nine of the calves (the first nine to enter the barn) to be given the feed supplement. Larger calves tend to push ahead and enter the barn first so the calves getting the supplement tend to be larger (the circles tend to be bluer).

Now click Run experiment to simulate the weight gains of the calves over two months. Repeat the experiment a few times and observe that most runs of the experiment estimate the effect of the supplement to be an increased weight gain of between 6 and 11.

The standard analysis gives a biased estimate of the effect of the supplement — it is overestimated by about 3.

Using calf weight to analyse the data

Randomised allocation of the supplement to half of the calves would remove the bias, but it is also possible to recover from the bad design by recording the weights of the calves after the treatments have been allocated and using this covariate in the analysis.

Unselect the checkbox Animate. The plot on the right changes to show the standard estimate of the treatment effect (ignoring the covariate) on the left and the corresponding estimate that does use the covariate on the right.

Click Run experiment several times to build up the sampling distributions of the two estimates. Observe that the estimate using the covariate is not only less variable (more accurate) but the bias in the estimate is removed — the distribution is centred round the correct improvement due to the feed supplement, 5.0.