Design of an experiment

There are two main aspects to the design of an experiment.

Treatments
The treatments that will be used in the experiment must first be chosen. These are combinations of levels of the different factors that will be varied and all possible combinations may not be used the same number of times. Indeed, some such treatments may not be used at all.
Allocation of treatments to experimental units
Once the treatments have been chosen, they must be allocated to the experimental units. If the treatments are badly allocated, the experimental results can be misleading and can result in wrong conclusions being made.

Allocation of treatments to the experimental units should involve some form of randomisation. In a completely randomised experiment, randomisation involves randomly picking experimental units for each treatment.


Assembly times for components: bad design

A component must be assembled by human operators and an engineer has devised a new assembly method and want to test whether it results in faster assembly times than the original assembly method. In order to compare assembly times with the two methods (original and new), the first nine operators who arrive one morning are told to use the original assembly method and the next nine operators are told to use the new method.

We will conduct a simulation of this experiment in which the new method reduces assembly time by exactly 5 minutes.

The circles on the left of the diagram below represent the 18 operators.

Click Allocate treatments to simulate the selection of nine of the operators (the first nine to arrive in the morning) who will be told to use the new assembly method. Now click Run experiment to simulate the average time taken to assemble components for each operator during the morning shift.

Repeat the experiment a few times and observe that most runs of the experiment estimate that the new method will reduce assemble time by between 6 and 11 minutes.

Why does the experiment consistently over-estimate the improvement caused by the new method?

The problem lies in the method of choosing which operators use the new method. Faster operators tend to arrive earlier, so the operators using the new method tend to be inherently faster. (The colour of the circles representing the operators show their inherent skill level and those getting the new method have a slight tendency to be redder.)

Random allocation

Randomising the allocation of the two assembly methods to the operators reduces the chance of any association between the treatment used and the operator skill level. In the simulation below, the new assembly method again decreases assembly time by exactly 5.0 minutes.

Click Allocate treatments to randomly pick which operators will use the new assembly method then click Run experiment to find the resulting assembly times and estimate the improvement from using the new assembly method.

Repeat a few times and observe that the new method is usually estimated to reduce assembly time by between 2 and 8 minutes. The results are therefore consistent with the true effect that was built into the simulation, 5.0.

With randomisation, there is no tendency to over- or under-estimate the effect of using the new assembly method.