F.1 Central Limit Theorem
The Central Limit Theorem is one of the most important results in statistics, stating that with sufficiently large sample sizes, the sample average approximately follows a normal distribution. This underscores the importance of the normal distribution, as well as most of the methods commonly used which make assumptions about the data being normally distributed.
Let’s first stop and think about what it means for the sample average to have a distribution. Imagine going to the store and buying a bag of your favorite brand of chocolate chip cookies. Suppose the bag has 24 cookies in it. Will each cookie have the exact same number of chocolate chips in it? It turns out that if you make a batch of cookies by adding chips to dough and mixing it really well, then putting the same amount of dough onto a baking sheet, the number of chips per cookie closely follows a Poisson distribution. (In the limiting case of chips having zero volume, this is exactly a Poisson process.) Thus we expect there to be a lot of variability in the number of chips per cookie. We can model the number of chips per cookie with a Poisson distribution. We can also compute the average number of chips per cookie in the bag. For the bag we have, that will be a particular number. But there may be more bags of cookies in the store. Will each of those bags have the same average number of chips? If all of the cookies in the store are from the same industrial-sized batch, each cookie will individually have a Poisson number of chips. So the average number of chips in one bag may be different from the average number of chips in another bag. Thus we could hypothetically find out the average number of chips for each bag in the store. And we could think about what the distribution of these averages is, across the bags in the store, or all the bags of cookies in the world. It is this distribution of averages that the central limit theorem says is approximately a normal distribution, with the same mean as the distribution for the individual cookies, but with a standard deviation that is divided by the square root of the number of samples in each average (i.e., the number of cookies per bag).
Theorem F.1 (Central Limit Theorem) Let X_1, ..., X_n be independent and identically distributed (IID) with \mathbb{E}(X_i) = \mu and Var(X_i) = \sigma^2 <\infty
Then:
\lim_{n\to\infty} \sqrt{n} \sum_{i=0}^{n} \frac{1}{n}\frac{(X_i-\mu)}{\sigma} = \sum_{i=0}^{n} \frac{X_i-\mu}{\sqrt{n} \sigma} = N(0, 1)
That is, \hat{X_n} is approximately normally distributed with mean µ and variance \frac{\sigma}{2/n} or standard deviation \frac{\sigma}{\sqrt{n}}.