133 Lesson 1.3.2: Understanding Joint Uncertainty of Two Unknown Quantities

133.1 Independence and uncorrelation

Independence. Iff jointly distributed random variables are independent, then the joint density factors into the product of marginal densities:

f_X(x_1, x_2, \ldots, x_n) = f_{X_1}(x_1) f_{X_2}(x_2) \cdots f_{X_n}(x_n).

This means the value of one random variable has no effect on the value of another. There is no linear or nonlinear relationship implied between them.

Uncorrelated. Two jointly distributed random variables X_1 and X_2 are uncorrelated iff:

\operatorname{cov}(X_1, X_2) = \mathbb{E}\!\left[(X_1 - \bar{x}_1)(X_2 - \bar{x}_2)^\top\right] = 0.

This implies

\mathbb{E}[X_1 X_2^\top] = \mathbb{E}[X_1]\mathbb{E}[X_2^\top].

The expectation of a product factors into the product of expectations, although this is not generally true.
Uncorrelated variables satisfy \rho_{12} = 0, meaning there is no linear relationship between them.

133.2 Independence versus uncorrelation

In general, uncorrelatedness is a much weaker condition than independence.
If X_1 and X_2 are independent, then they must also be uncorrelated.
However, uncorrelated random variables are not necessarily independent.
A special case occurs for jointly Gaussian random variables: if they are uncorrelated, then they are also independent.
This special Gaussian case is one reason Gaussian assumptions are so useful in sequential probabilistic inference.

133.3 Conditional probability

We can define a conditional probability density function

f_{X \mid Y}(x \mid y) = \frac{f_{X,Y}(x,y)}{f_Y(y)},

which gives the likelihood that X=x given that Y=y has occurred.

The marginal density f_Y(y) may be computed by integrating out the effect of X:

f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dx.

Conditional PDF = joint pdf / marginal pdf of the conditioning variable,
For each fixed y, we integrate out the effect of X to obtain the marginal pdf f_Y(y).

133.4 Bayes’ rule

From the definition of conditional density,

f_{X,Y}(x,y) = f_{X \mid Y}(x \mid y) f_Y(y) = f_{Y \mid X}(y \mid x) f_X(x).

Therefore, we can solve for one conditional PDF in terms of the other:

f_{X \mid Y}(x \mid y) = \frac{f_{Y \mid X}(y \mid x) f_X(x)}{f_Y(y)}.

This is Bayes’ rule.
It is used in a key step in the KF derivation to relate post-measurement probability to pre-measurement probability.

133.5 Conditional expectation

The conditional expectation of X given that Y=y has occurred is

\mathbb{E}[X \mid Y=y] = \mathbb{E}[X \mid Y] = \int_{-\infty}^{\infty} x\, f_{X \mid Y}(x \mid Y)\,dx.

Conditional expectation is critical to Kalman filtering.
The KF computes \mathbb{E}[x_k \mid Z_k] the expected value of the model state vector given the complete set of measurements made up to step k.

133.6 Central limit theorem

If Y = \sum_i X_i, with X_i IID, with finite mean and variance, then Y is approximately normally distributed. The approximation improves as more random variables are summed.
A special case is that a linear combination of Gaussian random variables is exactly Gaussian.
This motivates key assumptions used in the derivation of the Kalman filter:
- the state x_k is Gaussian,
- the process noise w_k is Gaussian,
- the sensor noise v_k is Gaussian,
- and w_k and v_k are uncorrelated with each other.
Under the Gaussian assumption, uncorrelatedness also implies independence.
Even when these assumptions are not exactly true in practice, the Kalman filter often performs well.

133.7 Summary

If two random variables are independent, the joint pdf equals the product of the marginal pdfs.
If two random variables are uncorrelated, the expected value of their product equals the product of their expected values, and there is no linear relationship between them.
In most practical cases, random variables are correlated, so conditional expectation helps predict one variable from another.
The central limit theorem supports the assumption that many random variables can be modeled as Gaussian.
Uncorrelated Gaussian random variables are independent, which is a special and very useful case.

--- title: "Lesson 1.3.2: Understanding Joint Uncertainty of Two Unknown Quantities" subtitle: "Kalman Filter Boot Camp (and State Estimation)" date: "2027-03-23" description: "This lesson builds on the prior lesson to understand the joint uncertainty of two unknown quantities, and how this is captured by the covariance matrix." categories: - "Probability and Statistics" keywords: - "Kalman Filter" - "state estimation" - "linear algebra" --- ## Independence and uncorrelation {#sec-kf-1.3.2-independence-and-uncorrelation} **Independence.** Iff jointly distributed random variables are independent, then the joint density factors into the product of marginal densities: $$ f_X(x_1, x_2, \ldots, x_n) = f_{X_1}(x_1) f_{X_2}(x_2) \cdots f_{X_n}(x_n). $$ - This means the value of one random variable has no effect on the value of another. There is no linear or nonlinear relationship implied between them. **Uncorrelated.** Two jointly distributed random variables $X_1$ and $X_2$ are uncorrelated iff: $$ \operatorname{cov}(X_1, X_2) = \mathbb{E}\!\left[(X_1 - \bar{x}_1)(X_2 - \bar{x}_2)^\top\right] = 0. $$ - This implies $$ \mathbb{E}[X_1 X_2^\top] = \mathbb{E}[X_1]\mathbb{E}[X_2^\top]. $$ - The expectation of a product *factors* into the product of expectations, although this is not generally true. - Uncorrelated variables satisfy $\rho_{12} = 0$, meaning there is no **linear** relationship between them. ## Independence versus uncorrelation {#sec-kf-1.3.2-independence-versus-uncorrelation} - In general, uncorrelatedness is a much weaker condition than independence. - If $X_1$ and $X_2$ are independent, then they must also be uncorrelated. - However, uncorrelated random variables are not necessarily independent. - A special case occurs for jointly Gaussian random variables: if they are uncorrelated, then they are also independent. - This special Gaussian case is one reason Gaussian assumptions are so useful in sequential probabilistic inference. ## Conditional probability {#sec-kf-1.3.2-conditional-probability} We can define a **conditional probability density function** $$ f_{X \mid Y}(x \mid y) = \frac{f_{X,Y}(x,y)}{f_Y(y)}, $$ which gives the likelihood that $X=x$ given that $Y=y$ has occurred. The marginal density $f_Y(y)$ may be computed by integrating out the effect of $X$: $$ f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dx. $$ - Conditional PDF = joint pdf / marginal pdf of the conditioning variable, - For each fixed $y$, we integrate out the effect of $X$ to obtain the marginal pdf $f_Y(y)$. ## Bayes' rule {#sec-kf-1.3.2-bayes-rule} - From the definition of conditional density, $$ f_{X,Y}(x,y) = f_{X \mid Y}(x \mid y) f_Y(y) = f_{Y \mid X}(y \mid x) f_X(x). $$ - Therefore, we can solve for one conditional PDF in terms of the other: $$ f_{X \mid Y}(x \mid y) = \frac{f_{Y \mid X}(y \mid x) f_X(x)}{f_Y(y)}. $$ - This is **Bayes' rule**. - It is used in a key step in the KF derivation to relate post-measurement probability to pre-measurement probability. ## Conditional expectation {#sec-kf-1.3.2-conditional-expectation} - The conditional expectation of $X$ given that $Y=y$ has occurred is $$ \mathbb{E}[X \mid Y=y] = \mathbb{E}[X \mid Y] = \int_{-\infty}^{\infty} x\, f_{X \mid Y}(x \mid Y)\,dx. $$ - Conditional expectation is critical to Kalman filtering. - The KF computes $\mathbb{E}[x_k \mid Z_k]$ the expected value of the model state vector given the complete set of measurements made up to step $k$. ## Central limit theorem - If $Y = \sum_i X_i,$ with $X_i$ IID, with finite mean and variance, then $Y$ is approximately normally distributed. The approximation improves as more random variables are summed. - A special case is that a linear combination of Gaussian random variables is exactly Gaussian. - This motivates key assumptions used in the derivation of the Kalman filter: - the state $x_k$ is Gaussian, - the process noise $w_k$ is Gaussian, - the sensor noise $v_k$ is Gaussian, - and $w_k$ and $v_k$ are uncorrelated with each other. - Under the Gaussian assumption, uncorrelatedness also implies independence. - [Even when these assumptions are not exactly true in practice, the Kalman filter often performs well.]{.mark} ## Summary - If two random variables are independent, the joint pdf equals the product of the marginal pdfs. - If two random variables are uncorrelated, the expected value of their product equals the product of their expected values, and there is no linear relationship between them. - In most practical cases, random variables are correlated, so conditional expectation helps predict one variable from another. - The central limit theorem supports the assumption that many random variables can be modeled as Gaussian. - Uncorrelated Gaussian random variables are independent, which is a special and very useful case.