Structure in the factor levels
Multiple comparison tests are appropriate to help assess differences between the effects of many factor levels if we have no outside knowledge about how the levels are related to each other. If we know something about how the levels are related, this information can often be used in one or two hypothesis tests of more meaningful differences between the levels.
In many experiments, there is a well defined order (or other structure) to the treatments and this is often related to the objectives of the study. Then you should not use a multiple comparison test.
Finally we repeat our assertion that the appropriate analysis is one that corresponds to the objectives of the study. We almost never find that a multiple comparison test corresponds to well-written objectives so they should usually be avoided.
Three examples are given below.
- Crop trials with many varieties
- The different varieties have often been developed from a few sources, so it is usually more meaningful to use contrasts to compare the mean responses from these sources, then test whether there is variation between the varieties from each sources.
- Numerical factors
- If the factor levels are numerical, it is more meaningful to test for linearity as a way to describe differences between the factor levels.
- Codein and acupuncture for dental pain relief
- In the example described on the previous page, the four treatments were really combinations of levels of two separate factors — whether or not codeine and acupuncture were used to help relieve dental pain. Rather than using a multiple comparison test to compare all four treatments, it is better to separate out the effect of the two factors and their interaction in an analysis of variance table and test them separately.