Sample Means

Suppose

$$X_1,X_2,X_3,\dots$$

is a sequence of IID random variables with population mean

$$\mathbb{E}(X_i)=\mu.$$

For example, we might imagine measuring the heights of randomly selected men from the UK. Since we will consider what happens as our sample size gets bigger and bigger, there is no upper limit on how many we’ll ask.

Let $Y_n$ be the average of the first $n$ observations:

$$Y_n=\frac{X_1+\dots+X_n}{n}.$$

This gives us a sequence of sample means

$$Y_1,Y_2,Y_3,\dots$$

Of course, this is a different sequence from the sequence of heights themselves.

The Weak and Strong Laws of Large Numbers both describe what happens to these sample means as the sample size grows, but they make different claims.

The Weak Law of Large Numbers

The Weak Law of Large Numbers says that, for a large sample size $n$, the sample mean $Y_n$ is very likely to be near $\mu$.

That is, for every $\varepsilon>0$,

$$\lim_{n\to\infty}\mathbb{P}(|Y_n-\mu|>\varepsilon)=0.$$

So, for one large value of $n$, the probability that the corresponding sample mean is far away from $\mu$ is small.

The Strong Law of Large Numbers

The Strong Law of Large Numbers makes a stronger statement.

Once an outcome $\omega$ has been realised, we obtain an entire sequence of numerical sample means

$$Y_1(\omega),Y_2(\omega),Y_3(\omega),\dots$$

The Strong Law says that, with probability $1$, this whole sequence converges to $\mu$. That is,

$$\mathbb{P}(\{\omega\in\Omega:\lim_{n\to\infty}Y_n(\omega)=\mu\})=1.$$

This is not about a sample of fixed size, but about the whole sequence!

Comparing the Weak and Strong Laws

The Weak Law says that a sample mean at one large value of $n$ is probably near $\mu$.

In contrast, the Strong Law says that, for almost every realised outcome, the entire sequence of sample means eventually settles towards $\mu$.

The distinction can be summarised as follows:

• Weak Law: one large-$n$ sample mean is probably close to $\mu$.

• Strong Law: with probability $1$, the whole realised sequence of sample means converges to $\mu$.

Background:

• Random Variables (Semi-Formal)