What Is a Moment Generating Function?

The moment generating function (MGF) of a random variable $X$ packages all of its moments into a single function.

It is defined by

$$M_X(t)=\mathbb{E}(e^{tX}).$$

To see where this comes from, first consider a sequence

$$(a_0,a_1,a_2,\dots).$$

Its exponential generating function is the power series

$$f(t)=a_0+\frac{a_1}{1!}t+\frac{a_2}{2!}t^2+\frac{a_3}{3!}t^3+\dots$$

The coefficient of $t^n$ is therefore

$$\frac{a_n}{n!}.$$

Now let our sequence be the moments of $X$:

$$1,\ \mathbb{E}(X) , \ \mathbb{E}(X^2), \ \mathbb{E}(X^3) , \ \dots$$

The corresponding exponential generating function is

$$M_X(t)=1+\frac{\mathbb{E}(X)}{1!}t+\frac{\mathbb{E}(X^2)}{2!}t^2+\frac{\mathbb{E}(X^3)}{3!}t^3+\dots$$

This is the moment generating function of $X$. It is a function that depends on $t$ only. Often we can sum the series to give a concise formula.

The MGF uniquely determines the probability distribution of $X$ (when the MGF exists for values of $t$ around $0$). This means we can sometimes identify the distribution of a complicated random variable simply by recognising its MGF.

What Are Moment Generating Functions For?

As the name suggests, we can use an MGF to generate the moments of a random variable.

Suppose we want the second moment.

Starting from

$$M_X(t)=1+\frac{\mathbb{E}(X)}{1!}t+\frac{\mathbb{E}(X^2)}{2!}t^2+\frac{\mathbb{E}(X^3)}{3!}t^3+\dots,$$

differentiate once:

$$M’_X(t)=\mathbb{E}(X)+\frac{\mathbb{E}(X^2)}{1!}t+\frac{\mathbb{E}(X^3)}{2!}t^2+\frac{\mathbb{E}(X^4)}{3!}t^3+\dots$$

Differentiate again:

$$M’’_X(t)=\mathbb{E}(X^2)+\frac{\mathbb{E}(X^3)}{1!}t+\frac{\mathbb{E}(X^4)}{2!}t^2+\frac{\mathbb{E}(X^5)}{3!}t^3+\dots$$

Now set $t=0$:

$$M’’_X(0)=\mathbb{E}(X^2).$$

In general,

$$M_X^{(n)}(0)=\mathbb{E}(X^n).$$

So we differentiate $n$ times, then set $t=0$ to recover the $n$th moment.

Example: Finding the Mean of a Binomial Distribution

Suppose

$$X\sim\operatorname{Binomial}(n,p).$$

Its MGF is

$$M_X(t)=(1-p+pe^t)^n.$$

To find the mean, differentiate once:

$$M’_X(t)=n(1-p+pe^t)^{n-1}pe^t.$$

Now set $t=0$:

$$M’_X(0)=n(1-p+p)^{n-1}p=np.$$

Hence

$$\mathbb{E}(X)=np,$$

as expected.

We could find the second moment by differentiating again, and from this obtain the variance.

How Do We Find Moment Generating Functions?

Recall the exponential series

$$e^y=1+y+\frac{y^2}{2!}+\frac{y^3}{3!}+\dots$$

Set

$$y=tX.$$

Then

$$e^{tX}=1+tX+\frac{(tX)^2}{2!}+\frac{(tX)^3}{3!}+\dots$$

Taking expectations gives

$$\mathbb{E}(e^{tX})=1+\mathbb{E}(X)t+\frac{\mathbb{E}(X^2)}{2!}t^2+\frac{\mathbb{E}(X^3)}{3!}t^3+\dots$$

This is exactly the power series for the MGF:

$$M_X(t)=\mathbb{E}(e^{tX}).$$

In practice, this expected-value formula is often the easiest way to find an MGF directly. For a discrete random variable we calculate a sum; for a continuous random variable we calculate an integral. However, as with PGFs, once we know the MGFs of some basic random variables, we can often find the MGFs of new ones more easily by combining them.

MGFs of Sums and Transformations

MGFs behave especially nicely when random variables are added together.

If $X$ and $Y$ are independent, then

$$M_{X+Y}(t)=\mathbb{E}(e^{t(X+Y)})=\mathbb{E}(e^{tX}e^{tY}).$$

Independence allows us to split the expectation:

$$M_{X+Y}(t)=\mathbb{E}(e^{tX})\mathbb{E}(e^{tY})=M_X(t)M_Y(t).$$

So, just as with PGFs, adding independent random variables means multiplying their generating functions.

MGFs also behave nicely when we add a number to a random variable or multiply it by a number.

For

$$Y=aX+b,$$

we have

$$M_Y(t)=M_{aX+b}(t)=\mathbb{E}(e^{t(aX+b)})=e^{bt}M_X(at).$$

An especially common situation in statistics is a random sample

$$X_1,X_2,\dots,X_n,$$

where the variables are IID (independent and identically distributed).

For their sum,

$$S_n=X_1+\dots+X_n,$$

we have

$$M_{S_n}(t)=(M_X(t))^n.$$

The sample average is

$$\bar{X}=\frac{X_1+\dots+X_n}{n}.$$

Using the transformation rule,

$$M_{\bar{X}}(t)=\left(M_X\left(\frac{t}{n}\right)\right)^n.$$

This makes MGFs particularly useful, since taking an average of some independent random variables is something we do all the time!

Background:

• Expected Values

• Binomial Distribution

• Probability Generating Functions