What is the Variance of a Vector?

Suppose we collect several random variables into a random vector:

$$\mathbf{X}=\left(\begin{array}{c}X_1 \cr X_2 \cr \vdots \cr X_n\end{array}\right).$$

This gives us a “random vector”, which is simply a vector whose entries are random variables.

For a single random variable, the variance is one number. But for a random vector, its “variance” is instead a whole matrix! That is, the variance-covariance matrix, which contains the variances of each component and the covariances of each pair.

This matrix is formed as follows:

$$\operatorname{Var}(\mathbf{X})=\left(\begin{array}{cccc}\operatorname{Var}(X_1) & \operatorname{Cov}(X_1,X_2) & \cdots & \operatorname{Cov}(X_1,X_n) \cr \operatorname{Cov}(X_2,X_1) & \operatorname{Var}(X_2) & \cdots & \operatorname{Cov}(X_2,X_n) \cr \vdots & \vdots & \ddots & \vdots \cr \operatorname{Cov}(X_n,X_1) & \operatorname{Cov}(X_n,X_2) & \cdots & \operatorname{Var}(X_n)\end{array}\right).$$

The entries on the diagonal are variances, since

$$\operatorname{Cov}(X_i,X_i)=\operatorname{Var}(X_i).$$

The other entries are covariances between different components.

Since

$$\operatorname{Cov}(X_i,X_j)=\operatorname{Cov}(X_j,X_i),$$

the matrix is symmetric.

It is also common to write the variance-covariance matrix using a capital Greek letter sigma:

$$\operatorname{Var}(\mathbf{X})=\Sigma.$$

The Variance of a Linear Combination of Components

Recall that for an ordinary random variable,

$$\operatorname{Var}(aX)=a^2\operatorname{Var}(X).$$

Now let

$$\mathbf{a}=\left(\begin{array}{c}a_1 \cr a_2 \cr \vdots \cr a_n\end{array}\right)$$

be a vector of constants.

Then

$$\mathbf{a}^T\mathbf{X}=a_1X_1+a_2X_2+\dots+a_nX_n.$$

This is now a single random variable: a linear combination of the components of $\mathbf{X}$.

Its variance is

$$\operatorname{Var}(\mathbf{a}^T\mathbf{X})=\mathbf{a}^T\operatorname{Var}(\mathbf{X})\mathbf{a}.$$

If we write $\operatorname{Var}(\mathbf{X})=\Sigma$, this becomes

$$\operatorname{Var}(\mathbf{a}^T\mathbf{X})=\mathbf{a}^T\Sigma\mathbf{a}.$$

Although the expression on the right contains vectors and a matrix, once expanded out the result is just one number: the variance of the random variable $\mathbf{a}^T\mathbf{X}$.

The Variance of a Transformed Random Vector

We can also transform the whole random vector $X$ into another vector, using a matrix.

If $A$ is a constant matrix with $n$ columns, then

$$A\mathbf{X}$$

is another random vector.

Its variance-covariance matrix is

$$\operatorname{Var}(A\mathbf{X})=A\operatorname{Var}(\mathbf{X})A^T.$$

Using $\Sigma$ for the variance-covariance matrix of $\mathbf{X}$,

$$\operatorname{Var}(A\mathbf{X})=A\Sigma A^T.$$

So, this is different from the last case:

• $\mathbf{a}^T\mathbf{X}$ is a single random variable, so its variance is one number.

• $A\mathbf{X}$ is a random vector, so its variance is another variance-covariance matrix.

Notice also that in the second case the transpose “$T$” goes right at the end in the formula.

Background:

• Variance and Standard Deviation

• Covariance and Correlation