What Is a Probability Generating Function?

A probability generating function (PGF) packages the probabilities of a non-negative integer-valued random variable into a power series. Often, this power series can be summed to give a convenient closed-form expression.

To see the idea, first recall that a power series looks like an infinite polynomial.

Suppose we have a sequence

$$(a_0,a_1,a_2,\dots).$$

We can form the power series

$$a_0+a_1z+a_2z^2+a_3z^3+\dots$$

where the coefficient of $z^n$ is $a_n$. This power series is the generating function of the sequence.

For example,

$$1+z+z^2+z^3+\dots$$

is the generating function of the sequence

$$(1,1,1,1,\dots).$$

It is also the familiar geometric series. When $-1<z<1$, the power series can be summed to give

$$\frac{1}{1-z}.$$

Now suppose that $X$ takes values in

$${0,1,2,\dots}.$$

We have the sequence of probabilities

$$\mathbb{P}(X=0),\mathbb{P}(X=1),\mathbb{P}(X=2),\dots$$

The probability generating function of $X$ is the generating function of this sequence:

$$P_X(z)=\sum_{n=0}^{\infty}\mathbb{P}(X=n)z^n.$$

Written out,

$$P_X(z)=\mathbb{P}(X=0)+\mathbb{P}(X=1)z+\mathbb{P}(X=2)z^2+\mathbb{P}(X=3)z^3+\dots$$

The coefficient of $z^n$ is precisely

$$\mathbb{P}(X=n).$$

Since the probabilities add to $1$, this series converges for every $z\in[-1,1]$, and sometimes on a larger domain as well.

The same PGF can also be written neatly as an expected value:

$$P_X(z)=\mathbb{E}(z^X).$$

As we’ll see shortly, if we know $P_X$, we can recover every probability $\mathbb{P}(X=n)$. The PGF of $X$ therefore pins down its entire probability distribution.

In a sense, we have simply represented the same information in a different form: instead of a sequence of probabilities, we now have a single function.

Extracting Probabilities from a PGF

The probabilities are stored in the power series as coefficients, but in practice we may only have the closed-form expression obtained after summing the series.

Differentiation gives us a way to recover the original probabilities.

Setting $z=0$ immediately gives

$$P_X(0)=\mathbb{P}(X=0),$$

because every other term contains a positive power of $z$.

Suppose instead that we want to recover $\mathbb{P}(X=2)$.

Differentiate once:

$$P’_X(z)=\mathbb{P}(X=1)+2\mathbb{P}(X=2)z+3\mathbb{P}(X=3)z^2+\dots$$

Differentiate again:

$$P’’_X(z)=2\mathbb{P}(X=2)+6\mathbb{P}(X=3)z+12\mathbb{P}(X=4)z^2+\dots$$

Now set $z=0$:

$$P’’_X(0)=2\mathbb{P}(X=2).$$

Dividing by $2$ gives

$$\mathbb{P}(X=2)=\frac{P’’_X(0)}{2}.$$

The same trick works for any value $n$. Differentiate $n$ times to move $\mathbb{P}(X=n)$ into the constant position, set $z=0$, and divide by the factorial:

$$\mathbb{P}(X=n)=\frac{P_X^{(n)}(0)}{n!}.$$

This is where the word generating comes from: the function allows us to generate the individual probabilities whenever we need them.

The algebra above explains why the method works. Once we have a closed-form expression for the PGF, we can apply exactly the same procedure directly to that expression.

Example: The Poisson Distribution

Suppose

$$X\sim\operatorname{Poisson}(\lambda).$$

Its probability generating function is

$$P_X(z)=e^{\lambda(z-1)}.$$

Suppose we want to recover $\mathbb{P}(X=2)$.

Differentiating twice gives

$$P’_X(z)=\lambda e^{\lambda(z-1)}$$

and

$$P’’_X(z)=\lambda^2e^{\lambda(z-1)}.$$

Setting $z=0$ gives

$$P’’_X(0)=\lambda^2e^{-\lambda}.$$

Dividing by $2!$ gives

$$\mathbb{P}(X=2)=\frac{P’’_X(0)}{2!}=\frac{e^{-\lambda}\lambda^2}{2!},$$

which is exactly the usual Poisson probability.

PGFs Can Also Generate Expected Values

Differentiating a PGF can reveal more than individual probabilities.

Recall that

$$P’_X(z)=\mathbb{P}(X=1)+2\mathbb{P}(X=2)z+3\mathbb{P}(X=3)z^2+\dots$$

Setting $z=1$ gives

$$P’_X(1)=\mathbb{P}(X=1)+2\mathbb{P}(X=2)+3\mathbb{P}(X=3)+\dots$$

This is exactly the definition of the expected value:

$$P’_X(1)=\mathbb{E}(X).$$

Differentiating again gives

$$P’’_X(1)=\mathbb{E}(X(X-1)).$$

From this we can recover the variance:

$$\operatorname{Var}(X)=P’’_X(1)+P’_X(1)-(P’_X(1))^2.$$

A PGF does not merely store the probabilities of $X$. Its derivatives also contain useful information about quantities such as its mean and variance.

How Do We Find a Probability Generating Function?

One direct method is to start from the probabilities and sum the resulting power series.

For example, suppose $X$ has a geometric distribution with parameter $p$, where $X$ counts the total number of trials required to obtain the first success.

Then

$$\mathbb{P}(X=n)=(1-p)^{n-1}p,\qquad n=1,2,\dots$$

and hence

$$P_X(z)=\sum_{n=1}^{\infty}(1-p)^{n-1}pz^n.$$

Pulling out $pz$ and reindexing gives

$$P_X(z)=pz\sum_{m=0}^{\infty}((1-p)z)^m.$$

Using the geometric-series formula,

$$P_X(z)=\frac{pz}{1-(1-p)z}.$$

At first, this may seem circular: the PGF was supposed to generate the probabilities, but here we needed to know the probabilities in order to find the PGF!

The real advantage appears once we have PGFs for familiar distributions available to us. We can then use simple rules for combining and transforming PGFs to find the PGFs of new random variables. Once we have those new PGFs, we can recover their probabilities and expected values.

PGFs of Combined Random Variables

One central result concerns sums.

If $X$ and $Y$ are independent non-negative integer-valued random variables, then

$$P_{X+Y}(z)=P_X(z)P_Y(z).$$

Multiplying two functions is often much easier than calculating every probability of $X+Y$ separately.

This result is closely related to convolution, and we explore it in more detail on the next page.

More generally, if $X_1,\dots,X_n$ are IID copies of $X$, then

$$P_{X_1+\dots+X_n}(z)=(P_X(z))^n.$$

This is particularly useful because sums of IID random variables appear everywhere in probability and statistics.

For example, the sample average is

$$\bar{X}=\frac{X_1+\dots+X_n}{n}.$$

The average itself will not usually be integer-valued, so it will not usually have a PGF of the kind defined here. We can still use the PGF to find the distribution of the numerator

$$X_1+\dots+X_n,$$

and then divide the resulting values by $n$ to obtain the distribution of the average.

This is an important connection. Many central results in probability and statistics concern sums or averages of IID random variables, including the Central Limit Theorem.

PGFs also behave neatly when we add a number to a random variable or multiply it by a number.

If $a$ and $b$ are non-negative integers, then

$$P_{aX+b}(z)=z^bP_X(z^a).$$

We can see this directly from the expected-value formula:

$$P_{aX+b}(z)=\mathbb{E}(z^{aX+b})=z^b\mathbb{E}((z^a)^X).$$

Summary

A probability generating function represents an entire sequence of probabilities using a single function.

This representation can make some problems much easier. We can differentiate a PGF to recover probabilities and expected values, multiply PGFs to study sums of independent random variables, and substitute into them to study simple transformations.

The underlying probabilities have not changed. We have simply represented them in a form that is sometimes easier to manipulate.

Later in this section, moment generating functions will use a closely related idea for more general random variables.

Background:

• Expected Values

• Geometric Distribution

• Poisson Distribution