When fewer runs of the experiment are used than the number required for a complete factorial design, it should come as no surprise that something is lost.
Interactions
When only a subset of the runs in a complete factorial design are used, some interactions get 'mixed up' with main effects.
When main effects and interactions are mixed up in this way, they are said to be confounded or aliased with each other.
It is necessary to assume that high-order interactions between the factors are either not present or negligible in relation to the main effects of the factors. If the number of runs is much smaller than the number required for a complete experiment, it may be necessary to assume that there are no interactions between the variables at all.
Testing
We often cannot test the significance of the main effects or interactions since the experiment may not have used enough runs to leave any residual degrees of freedom. Without an estimate of the variance of the experimental error, tests cannot be conducted.
As in complete factorial experiments with no replicates, it may be possible to get some residual degrees of freedom by either assuming interactions to be zero or by adding a few runs at the centre point of the experiment.
However in a screening experiment, it is often enough to rank the factors without testing the significance of their effects.
Hardness of car paint
This experiment is an extreme example of a small subset of the 215 = 32,768 runs needed for a complete factorial design. With 16 runs, it is only possible to estimate the overall mean and 15 other quantities, so only the 15 main effects of the 15 factors can be estimated — there is no independent information to estimate interactions.
Moreover, the main effects of all factors are confounded with various interactions between other factors. For example, factor L had the largest main effect — we estimate that adding this additive will increase the hardness of the paint by 6.0.
Additive | Hardness | ||||||||
---|---|---|---|---|---|---|---|---|---|
Run | ... | D | E | ... | L | ... | DE | ... | Y |
1 | ... | +1 | +1 | ... | +1 | ... | +1 | ... | 53.3 |
2 | ... | -1 | +1 | ... | -1 | ... | -1 | ... | 46.6 |
3 | ... | +1 | +1 | ... | +1 | ... | +1 | ... | 53.8 |
4 | ... | -1 | +1 | ... | -1 | ... | -1 | ... | 44.6 |
5 | ... | +1 | -1 | ... | -1 | ... | -1 | ... | 44.8 |
6 | ... | -1 | -1 | ... | +1 | ... | +1 | ... | 58.9 |
7 | ... | +1 | -1 | ... | -1 | ... | -1 | ... | 56.5 |
8 | ... | -1 | -1 | ... | +1 | ... | +1 | ... | 60.5 |
9 | ... | +1 | -1 | ... | -1 | ... | -1 | ... | 48.2 |
10 | ... | -1 | -1 | ... | +1 | ... | +1 | ... | 47.4 |
11 | ... | +1 | -1 | ... | -1 | ... | -1 | ... | 54.8 |
12 | ... | -1 | -1 | ... | +1 | ... | +1 | ... | 45.9 |
13 | ... | +1 | +1 | ... | +1 | ... | +1 | ... | 56.3 |
14 | ... | -1 | +1 | ... | -1 | ... | -1 | ... | 50.2 |
15 | ... | +1 | +1 | ... | +1 | ... | +1 | ... | 62.3 |
16 | ... | -1 | +1 | ... | -1 | ... | -1 | ... | 44.7 |
However the term for an interaction between factors D and E, found by multiplying the +1 and -1 values for D and E, is identical to the levels of L, so the main effect of 6 percent for L could alternatively have been caused by a strong interaction between the effects of D and E.
The main effect for factor L is confounded with the interaction between D and E — it is impossible to say whether the increase of 6 percent is caused by L, a DE interaction or some combination.
Use of the estimated main effects in the previous page to rank the importance of the factors therefore requires the assumption that the interactions are much smaller than the main effects of the factors.
A second problem in this example is that all 15 degrees of freedom are used to estimate main effects of the 15 factors and there are no residual degrees of freedom and it is impossible to estimate the error variance.
It is therefore impossible to test the significance of any main effects — it is possible that the estimated main effects are all simply a result of random variation in the experiment.
We will show in the following sections that some subsets of complete factorial experiments are much less severe in their confounding than the paint hardness experiment.