Confounding

In a badly designed experiment, the characteristics of the experimental units can distort the apparent relationship between the controlled variable and response.

In the most extreme case, the design makes it impossible to disentangle the effects of the treatment and other characteristics of the experimental units. If the treatment is perfectly correlated with another variable, the effects of the two variables cannot be distinguished. The treatment and variable are then said to be confounded.

It is particularly important that confounding is avoided when data are collected.

Experiment with a new cafe layout

A chain of cafes has decided to update its image with different advertising and a new layout for its outlets. The diagram below shows sales from a random sample of 10 cafes before the change (2013), and sales from another sample of 10 cafes after the change (2014).

Although the mean sales are higher, it should not be concluded that the new layout has improved sales. The two layouts were tried in different years, and there may have been an unrelated changes in the 'cafe culture' between 2013 and 2014 — the layout is confounded with other changes over time.

Perhaps sales are higher because people had more disposable income in 2014.

No conclusions can be drawn about the effect of the layout.

Trial of a new teaching method

An accountancy lecturer writes a web-based tutorial resource to help teach a topic. Students in a large class are told about the resource and allowed access through a login system that records usage. About half of the class use the tutorial.

To assess whether the tutorial helps students to learn the ideas, the lecturer counts the number of correct answers from the 3 questions about this topic in the final multiple-choice exam.

Students who used the resource got a higher average mark, but it is impossible to conclude that it was the tutorial resource that caused it.

Use of the resource is confounded with other characteristics of the students — only the more motivated students use it and they also study harder. It is impossible to distinguish between motivated students performing better and use of the resource improving marks.

The data do not allow you to reach any conclusions about whether the resource is effective.