Introduction

Suppose we have a sequence of random variables

$$X_1,X_2,X_3,\dots$$

and another random variable $X$, which will act as our target.

All of these random variables are defined on the same sample space $\Omega$.

Intuitively, a random variable is an uncertain quantity which depends on the outcome of an experiment. But formally, it is a function, defined on this set of possible outcomes:

$$X:\Omega\to\mathbb{R}.$$

So, when the experiment produces some particular outcome $\omega\in\Omega$, the random variable $X$ turns that outcome into an ordinary number:

$$X(\omega).$$

Likewise, once the outcome $\omega$ of the experiment has been produced, it is fed into every random variable in our sequence to find their realised values too:

$$X_1(\omega),X_2(\omega),X_3(\omega),\dots$$

Once $\omega$ has been selected, this is no longer a sequence of random variables, but rather simply an ordinary sequence of numbers.

Almost sure convergence asks whether this numerical sequence converges to the number $X(\omega)$.

What Is Almost Sure Convergence?

Once $\omega$ is fixed, we are interested in whether the sequence of numbers

$$X_1(\omega),X_2(\omega),X_3(\omega),\dots$$

converges to the value of the target random variable, $X(\omega)$, in the ordinary sense of convergence for a sequence of real numbers.

It might be that it converges for some outcomes $\omega$, but not for others.

Almost sure convergence says that the probability of selecting an outcome $\omega$ for which the sequence does converge is equal to $1$.

Formally, this set of outcomes is

$$\{\omega\in\Omega:\lim_{n\to\infty}X_n(\omega)=X(\omega)\}.$$

So, for almost sure convergence, we require

$$\mathbb{P}(\{\omega\in\Omega:\lim_{n\to\infty}X_n(\omega)=X(\omega)\})=1.$$

We then write

$$X_n\overset{a.s.}{\longrightarrow}X.$$

Why Is It Called “Almost Sure”?

The phrase almost sure means that the probability of convergence is equal to $1$.

It does not necessarily mean that convergence happens for every single possible outcome $\omega$.

That is, there may still be some theoretically possible outcomes for which $X_n(\omega)$ does not converge to $X(\omega)$.

But the set of all outcomes $\omega$ where convergence fails must have probability zero in total. That is, we must be “almost sure” that we will not get such an outcome.

How Does Almost Sure Convergence Compare with Other Notions?

The three main notions of convergence we are considering satisfy

$$\text{Almost Sure Convergence}\Rightarrow\text{Convergence in Probability}\Rightarrow\text{Convergence in Distribution}.$$

Almost sure convergence is therefore the strongest of the three.

The reason is that almost sure convergence makes a very strong statement about what happens after an outcome $\omega$ has actually been chosen. It says that, for almost every possible outcome $\omega$, the entire sequence of numbers

$$X_1(\omega),X_2(\omega),X_3(\omega),\dots$$

converges as an infinite sequence of real numbers.

Convergence in probability asks for less. As $n$ gets larger, it only requires that the probability of selecting an outcome $\omega$ for which the two numbers $X_n(\omega)$ and $X(\omega)$ are far apart tends to zero.

Example: The Strong Law of Large Numbers

An especially important example again comes from sample averages.

Suppose

$$X_1,X_2,X_3,\dots$$

are IID random variables with

$$\mathbb{E}(X_i)=\mu$$

and finite variance

$$\operatorname{Var}(X_i)=\sigma^2.$$

Let

$$\bar{X}_n=\frac{X_1+X_2+\dots+X_n}{n}$$

be the average of the first $n$ observations.

The Strong Law of Large Numbers says that

$$\bar{X}_n\overset{a.s.}{\longrightarrow}\mu.$$

So, with probability $1$, the sequence of sample averages

$$\bar{X}_1,\bar{X}_2,\bar{X}_3,\dots$$

converges to the population mean $\mu$.

This is stronger than merely saying that each $\bar{X}_n$ becomes increasingly unlikely to be far away from $\mu$. It says that, for almost every outcome $\omega$ of the experiment, the entire sequence of realised sample averages

$$\bar{X}_1(\omega),\bar{X}_2(\omega),\bar{X}_3(\omega),\dots$$

converges to $\mu$ as an ordinary sequence of real numbers.

Background:

• Random Variables (Semi-Formal)