Goal of small residuals
Recapping, any linear equation provides fitted values — predictions of the response for each combination of values of the explanatory variables. For example, with two explanatory variables, the fitted values are,
![]() |
= b0 + b1 xi + b2 zi |
These fitted values are unlikely to match exactly the observed response values and the prediction 'errors' are called the residuals,
ei = yi − | ![]() |
'Small' residuals are desirable but adjusting the parameters 'by eye' is neither a scientifically objective nor a practical method to achieve small residuals.
Least squares
A combined measure of the size of the residuals is their sum of squares,
SSResidual | ![]() |
An objective estimation method estimates the parameters with the values that minimise the residual sum of squares. This is called the method of least squares and is similar to least squares estimation for the simple linear model.
The principle of least squares reduces parameter estimation to a relatively straightforward mathematical problem — when there are two explanatory variables, we must find the values of b0, b1 and b2 to minimise:
SSResidual | ![]() |
The solution can be obtained algebraically but the formulae for the least squares estimates are more complicated than those for simple linear regression, and we will not give them here.
In practice, statistical software should always be used
to obtain least squares parameter estimates.
Illustration
The diagram below shows an artificial data set. A square is drawn beside each residual — its area is the squared residual. The total red area is therefore the sum of the squared residuals.
The least squares plane can be moved by dragging the three arrows. Your aim should be to minimise the red area or, numerically, to minimise the residual sum of squares that is displayed at the top right of the diagram.
Finally, click the button Least squares to see the parameter values that the computer calculates to minimise the residual sum of squares.