If we only looked at $XY$, we’d face a problem:

• The values of $X$ and $Y$ can be large or small, but without considering their typical (mean) values, we cannot determine their relative movement.
• A large $XY$ does not necessarily mean a strong relationship; it might just mean one variable is large in absolute terms.
• If $X$ and $Y$ are always positive (e.g., temperature and ice cream sales), $XY$ will always be positive, making it misleading as a measure of co-movement.

By subtracting their respective expected values (i.e., taking $Z = (X - \mathbb{E}[X])(Y - \mathbb{E}[Y])$), we normalize these values, ensuring that we measure relative movement rather than absolute magnitudes:

• If $Z > 0$, both values are either above or below their respective means.

  $\Rightarrow$ They move together.

• If $Z < 0$, one is above its mean while the other is below.

  $\Rightarrow$ They move oppositely.

• The magnitude of $Z$ indicates how strong the relationship is: large deviations imply a strong relationship, small deviations imply a weak one.

By taking the expectation $\mathbb{E}[Z]$, we determine whether these relationships hold on average.

The problem with covariance is that it is not scale-free which means that it is hard to interpret beside whether the variables move together.

To understand this, take a look at the following example:

import numpy as np

height_in_cm = [110, 120, 130, 140, 150, 160, 170, 180, 190, 200]
height_in_m = [1.10, 1.20, 1.30, 1.40, 1.50, 1.60, 1.70, 1.80, 1.90, 2.00]

age = [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]


# Print covariance with two decimals
print(f"Cov(height_in_cm, age) = {np.cov(height_in_cm, age)[0, 1]:.2f}")
print(f"Cov(height_in_m, age) = {np.cov(height_in_m, age)[0, 1]:.2f}")

Cov(height_in_cm, age) = 91.67
Cov(height_in_m, age) = 0.92

See ? We didn’t obtain the same results even though the relationships did not change.

But what if… what if there was a way to make this scale free ?

Well there is and it’s called correlation.

Correlation is defined as:

Let’s modify our previous code to compute correlation

print(f"Corr(height_in_cm, age) = {np.corrcoef(height_in_cm, age)[0, 1]:.2f}")
print(f"Corr(height_in_m, age) = {np.corrcoef(height_in_m, age)[0, 1]:.2f}")

Corr(height_in_cm, age) = 1.0
Corr(height_in_m, age) = 1.0

That’s great, we obtained the same value for the same relation even though our units were not the same. Now we have a scale and unit-free metric.

Another very nice thing about correlation is that it is very easy and intuitive to interpret:

• It is between $-1$ and $1$.
• If $\text{corr}(X, Y) > 1$, it means that both variable move in the same direction (e.g., height and weight).
• If $\text{corr}(X, Y) < 1$, it means that both variable move in opposite directions (e.g., temperature and people skiing).
• If $\text{corr}(X, Y) \approx 0$, it means that variable are not linearly correlated (e.g., shoe size and IQ).

Actually, there are different types of correlations. This one is called Pearson correlation and it measures linear correlation between two random variables.

We won’t go into details but the other one you should know of is the Spearman’s Rank correlation. This one is useful as it is a non-parametric correlation and does not assume a specific distribution.

Until now we’ve only discussed about static data. But what happens when data evolves or fluctuates over time? In the world of time-series data, it’s crucial to understand how signals relate to themselves or to each other over time. This is where auto-covariance and cross-covariance come into play.