STATE 02 · THE GAUSSIAN

Covariance and correlation structures

Symmetry, scale and the geometry of dependence.

Covariance and correlation

The covariance of two variables measures how they move together. It is the expectation of the product of their deviations from their means. A positive covariance signals a tendency to rise and fall together; a negative covariance signals a tendency to move in opposite directions. The covariance of a variable with itself is its variance, so the covariance operation contains the variance as a special case.

Covariance depends on the units in which the variables are measured. Dividing by the two standard deviations removes the units and yields the correlation, a pure number confined between minus one and one. A correlation of one means the two variables are exact increasing linear functions of each other; minus one means exact decreasing linear dependence; zero means no linear association at all. Zero correlation does not rule out non-linear dependence, though for jointly Gaussian variables it does imply independence.

The covariance matrix

For a collection of many variables we gather all pairwise covariances into a square table called the covariance matrix. Its diagonal holds the variances and its off-diagonal entries hold the covariances. The table is symmetric, because the covariance of the pair does not depend on the order in which the pair is named. Dividing each entry by the two relevant standard deviations produces the correlation matrix, whose diagonal is filled with ones.

The covariance matrix is positive semi-definite. This means that for any vector of coefficients, the quadratic form built from the matrix and the vector is never negative. The statement is not an accident of notation; it is the algebraic shadow of the fact that the square of any real quantity is non-negative. Take the variance of a linear combination of the variables, add and subtract the means, and the result appears as exactly that quadratic form.

Figure 1. A symmetric covariance table; the entries above and below the diagonal mirror one another.

Symmetry and positive semi-definiteness in practice

The two structural facts, symmetry and positive semi-definiteness, are not mere formalities. They constrain which tables can possibly arise as covariance matrices of genuine variables. A symmetric table with a negative variance on the diagonal, or a correlation larger than one, can never arise. A table whose three correlations are mutually inconsistent, in the sense that no joint law could produce them, is likewise impossible.

These restrictions matter whenever a covariance matrix is specified in advance and used as input to a model. If the specified table is not positive semi-definite, the model is empty: no family of variables exhibits those covariances. Checking the structural conditions first therefore spares the analyst a great deal of fruitless work.

With the matrix language in hand the reader is ready for sequences of variables rather than single observations. The next movement of the reference turns to stochastic processes, where dependence unfolds along an index such as time.