Appendix E — Appendix: The Law of Large Numbers

Author

Oren Bochman

E.1 Law of large numbers

Suppose we observe data D=\{x_1, \ldots, x_n\} with each x_i \sim F .

By the strong law of large numbers the empirical distribution \hat{F}_n based on data D=\{x_1, \ldots, x_n\} converges to the true underlying distribution F as n \rightarrow \infty almost surely:

\hat{F}_n\overset{a. s.}{\to} F

The Glivenko–Cantelli asserts that the convergence is uniform. Since the strong law implies the weak law we also have convergence in probability:

\hat{F}_n\overset{P}{\to} F

Correspondingly, for n \rightarrow \infty the average \text{E}_{\hat{F}_n}(h(x)) = \frac{1}{n} \sum_{i=1}^n h(x_i) converges to the expectation \text{E}_{F}(h(x)) .

---
title: "Appendix: The Law of Large Numbers"
---

## Law of large numbers {#sec-appendix-law-of-large-numbers}

Suppose we observe data $D=\{x_1, \ldots, x_n\}$ with each $x_i \sim F$ .

By the strong law of large numbers the empirical distribution $\hat{F}_n$ based on data $D=\{x_1, \ldots, x_n\}$ converges to the true underlying distribution $F$ as $n \rightarrow \infty$ almost surely:

$$
\hat{F}_n\overset{a. s.}{\to} F
$$

The [Glivenko--Cantelli](https://en.wikipedia.org/wiki/Glivenko%E2%80%93Cantelli_theorem) asserts that the convergence is uniform. Since the strong law implies the weak law we also have convergence in probability:

$$
\hat{F}_n\overset{P}{\to} F
$$

Correspondingly, for $n \rightarrow \infty$ the average $\text{E}_{\hat{F}_n}(h(x)) = \frac{1}{n} \sum_{i=1}^n h(x_i)$ converges to the expectation $\text{E}_{F}(h(x))$ .