Need for an objective estimation method

Recapping, any linear equation provides fitted values — predictions of the response for each combination of values of the explanatory variables,

 =  b0 + b1 xi + b2 zi

These fitted values are unlikely to match exactly the observed response values and the prediction 'errors' are called the residuals,

ei  =  yi

'Small' residuals are desirable but adjusting the parameters 'by eye' is neither a scientifically objective nor a practical method to achieve small residuals.

Residual sum of squares

A combined measure of the size of the residuals is their sum of squares,

An objective estimation method estimates the parameters with the values that minimise the residual sum of squares. This is called the method of least squares and is similar to least squares estimation for the simple linear model.

Obtaining the least squares estimates

A display of the residual sum of squares could be a useful guide to manually adjusting the parameters. However the principle of least squares reduces parameter estimation to a relatively straightforward mathematical problem — find the values of b0, b1 and b2 to minimise:

Although the formulae for the least squares estimates (that solve this mathematical problem) are more complicated than those for simple linear regression, they can nonetheless be algebraically derived.

In practice, statistical software should be used to obtain least squares parameter estimates. The formulae are therefore not worth showing here.

(It is actually possible to write the formulae for these least squares estimates concisely, but this requires matrix notation so we will leave it until the next chapter of CAST.)

Illustration

The following diagram shows the same data set that was used in the previous page. A square is drawn beside each residual — its area is the squared residual. The total red area is therefore the sum of the squared residuals.

Again drag the arrows to adjust the parameters. Your aim should be to minimise the red area or, numerically, to minimise the residual sum of squares that is displayed at the top right of the diagram.

Finally, click the button Least squares to see the parameter values that the computer calculates to minimise the residual sum of squares.