Covariance
The expected value of one particular function of two variables is especially important. The concept of covariance is closely related to that of the variance of a single random variable.
Definition
The covariance of two random variables, \(X\) and \(Y\), is
\[ \Covar(X,Y) \;=\; E\left[(X - \mu_X)(Y - \mu_Y)\right] \]where \(\mu_X\) and \(\mu_Y\) are the means of the two variables. The covariance is often denoted by \(\sigma_{XY}\).
The following results can be easily proved from the definition of covariance and the general properties of expected values. They are simply stated here.
Properties of covariance
For any random variables, \(X\) and \(Y\), and constant \(a\),
The following result is often useful when finding the covariance of two variables. Note how similar it is to the corresponding result for variances.
Alternative formula for covariance
For any random variables, \(X\) and \(Y\),
\[ \Covar(X, Y) \;=\; E[XY] - E[X]E[Y] \]From the general properties of expected values, and noting that both \(\mu_X\) and \(\mu_Y\) are constants,
\[ \Covar(X, Y) \;=\; E[XY] - \mu_X \mu_Y - \mu_Y \mu_X + \mu_X \mu_Y \;=\; E[XY] - \mu_X \mu_Y \]Linear transformations
The covariance between two random variables is affected in a simple way by linear transformations of the variables.
Covariance of linear transformations of X and Y
For any random variables, \(X\) and \(Y\), and constants \(a\), \(b\), \(c\) and \(d\),
\[ \Covar(a + bX, c+dY) \;=\; bd \Covar(X, Y) \]The means of the two transformed variables are
\[ E[a + bX] = a + b\mu_X \spaced{and} E[c + dY] = c + d\mu_Y \]The covariance can then be written as
\[ \begin{align} \Covar(a + bX, c + dY) \;&=\; E\left[\bigl(a + bX - E[a + bX]\bigr)\bigl(c + dY - E[c + cY]\bigr)\right] \\[0.3em] &=\; E\left[\bigl(a + bX - (a + b\mu_X)\bigr)\bigl(c + dY - (c + d\mu_Y)\bigr)\right] \\[0.3em] &=\; E\bigl[b(X - \mu_X) \times d(Y - \mu_Y)\bigr] \\[0.3em] &=\; bd \times E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] \\[0.3em] &=\; bd \times \Covar(X,Y) \end{align} \]The covariance is therefore unaffected by adding constants (\(a\) and \(c\)) to the variables. Multiplying by constants (\(b\) and \(d\)) simply multiplies their covariance by these values.