Probability generating functions

Probability Generating Functions (PGFs) are useful tools for dealing with sums and limits of
random variables. For some stochastic processes, they also have a special role in telling us
whether a process will ever reach a particular state.
By the end of this topic, you should be able to:
• find the sum of Geometric, Binomial, and Exponential series;
• know the definition of the PGF, and use it to calculate the mean, variance, and probabilities;
• calculate the PGF for Geometric, Binomial, and Poisson distributions;
• calculate the PGF for a randomly stopped sum;
• calculate the PGF for first reaching times in the random walk;
• use the PGF to determine whether a process will ever reach a given state.

Common Sums

1
1. Geometric Series: 1  r  r 2  r 3    x 0 r x 

, where r  1 . This formula
1 r

proves that  (   x)  1 .
x 0
 
When X ~ Geometric (p): (   x)  p(1  p ) x   (   x)   p(1  p) x
x 0 x 0

 p  (1  p) x
x 0

p
 (because 1  p  1)
1  (1  p )
= 1

n
2. Binomial Theorem: For any p, q   , and integer n, ( p  q ) n   p x q n  x .
x 0

 n n! 
Note that    . The Binomial Theorem proves that  (   x)  1
 x  (n  x)!x! x 0

 n
when X ~ Binomial (n, p): (   x)    p x (1  p) n x for x  0, 1, 2,  , n , so
 x
n n
 n x

x 0
(   x )     p (1  p) n x
x 0  x 

 ( p  (1  p)) n  1n  1
 
x 
3. Exponential Power series: For any   ,  x!  e  . This proves that  (  x)  1
x 0 x 0

 x
when X ~ Poisson(  ): (   x)  e  for x  0, 1, 2,  , so
x!
 
x 
x
 (  x)   x! e   e   x!
x 0 x 0
 

x 0

= e e  1


 
n

Note: Another useful identity is: e  lim 1   for    .


n
 n

The probability generating function (PGF) is a useful tool for dealing with discrete random
variables taking values 0, 1, 2, . . .. Its particular strength is that it gives us an easy way of
characterizing the distribution of X +Y when X and Y are independent. In general it is difficult to
find the distribution of a sum using the traditional probability function. The PGF transforms a
sum into a product and enables it to be handled much more easily. Sums of random variables are
particularly important in the study of stochastic processes, because many stochastic processes are
formed from the sum of a sequence of repeating steps. The PGF can be used to generate all the
probabilities of the distribution. This is generally tedious and is not often an efficient way of
calculating probabilities. However, the fact that it can be done demonstrates that the PGF tells us
everything there is to know about the distribution.

Definition 1: Let X be a discrete random variable taking values in the non-negative integers {0,
1, 2, . . .}. The probability generating function (PGF) of X is G  ( s )  ( s  ) , for all s ∈ ℜ for
which the sum converges.

Calculating the probability generating function


G  ( s )  ( s  )   s   (   x )
x 0

Properties of the PGF

1. G (0)  (   0)
G (0)  0 0  (   0)  01  (   1)  0 2  (   2)  
 G (0)  (   0)
 
2. G  (1)  1 : G  (1)  1x (   x)   (   x)  1
x 0 x 0

e.g. (a) Binomial Distribution:

 n
Let X ~ Binomial (n, p), so (   x)    p x (1  p) n x for x  0, 1, 2,  , n
 x
 n
 n n
n
G (s)   s x   p x q n x    ( ps) x q n x
x 0  x x 0  s 

 ( ps  q) n by the Binomial Theorem true for all s.

Thus, G ( s)  ( ps  q) n for all s   .

G (0)  ( p  0  q) n 200 X ~ Bin(n = 4, p = 0.2)

 qn
150
 (   0)
G(s)
G (1)  ( p  1  q) n 100

 1n
50
1
0
-20 -10 0 10 s

x
(b) Poisson Distribution: Let X ~ Poisson ( ) , so (   x)  e  for x  0, 1, 2, 
x!

x 
( s ) x
G (s)   s x e   e   40 X ~ Poisson(4)
x 0 x! x 0 x!

 e  e ( s ) for all s   30
G(s)

Thus G ( s)  e  ( s 1) for all s   20

10
0
-1.0 -0.5 0.0 0.5 1.0 1.5 2.0 s

(c) Geometric Distribution: Let X ~ Geometric (p), so (   x)  p(1  p) x  pq x for

x  0, 1, 2,  where q = 1- p
 
G  ( s )   s x pq x  p  ( qs) x
x 0 x 0

p
 for all s such that qs  1 . G(s)
1  qs
 5
to Infinity
4
p 1
Thus G  ( s )  for s  . 3 X ~ Geom(0.8)
1  qs q 2
1
0
-5 0 5 s

Using the probability generating function to calculate probabilities

The probability generating function gets its name because the power series can be expanded and
differentiated to reveal the individual probabilities. Thus, given only the PGF G  ( s )  ( s  ) ,
we can recover all probabilities P(X = x). For shorthand, write p x  (   x) . Then

G  ( s )  ( s  )   p x s x  p 0  p1 s  p 2 s 2  p3 s 3  p 4 s 4  
x 0

Thus p 0  (   0)  G  (0) .

First derivative: G  ( s )  p1  2 p 2 s  3 p3 s 2  4 p 4 s 3  

Thus p1  (   1)  G (0) .

Second derivative: G  ( s )  2 p 2  (3  2) p3 s  (4  3) p 4 s 2  

1
Thus p 2  (   2)  G (0) .
2

Third derivative: G ( s)  (3  2  1 p3  (4  3  2) p 4 s  

1
Thus p3  (   3)  G (0) .
3!

1 1 d
n
In general: p n  (   n)   G ( n ) (0)    n G  ( s )  s 0
 n!   n!  ds

Example 1

s
Let X be a discrete random variable with PGF G ( s)  (2  3s 2 ) . Find the distribution of X.
5
Solution
 2 3
G (s)  s  s 3 : G  (0)  (   0)  0
5 5
2 9 2
G  ( s )   s 2 : G  (0)  (   1) 
5 5 5
18
G  ( s )  s : G  (0)  (   2)  0
5
18 3
G  ( s )  : G  (0)  (   3) 
5 5

1 (r )
G( r ) ( s)  0  r  4 : G  ( s )  (   r )  0  r  4
r!

 2
 1 with probabilit y 5
Thus, 
 3 with probabilit y 3
 5

1
Uniqueness of the PGF: The formula p n  (   n)   G (n ) (0) shows that the whole
 n! 
sequence of probabilities p0 , p1 , p 2 , is determined by the values of the PGF and its
derivatives at s = 0. It follows that the PGF specifies a unique set of probabilities.

Fact: If two power series agree on any interval containing 0, however small, then all terms of the
two series are equal.

Formally: Let A(s) and B(s) be PGFs with (s)  n0 an s n , (s)  n0 bn s n . If there exists
 

some R′ > 0 such that A(s) = B(s) for all −R′ < s < R′ , then a n  bn for all n.

Practical use: If we can show that two random variables have the same PGF in some interval
containing 0, then we have shown that the two random variables have the same distribution.

Another way of expressing this is to say that the PGF of X tells us everything there is to know
about the distribution of X.

Expectation and moments from the PGF

As well as calculating probabilities, we can also use the PGF to calculate the moments of the
distribution of X. The moments of a distribution are the mean, variance, etc.

Theorem 1: Let X be a discrete random variable with PGF G (s) . Then:

 (i) ( )  G (1)
d k G ( s)
(ii) {(   1)(   2)  (   k  1)}  G (k )
 (1)  s 1
ds k
This is the kth factorial of X
Proof

(i) G (s)   s x p x
x 0

So G  ( s)   xs x 1 p x
x 0

 G  (1)   xp x  (  )
x 0

d G ( s) 
k
(ii) G ( k ) ( s )    x( x  1)( x  2)  ( x  k  1) p x
ds k xk

 {(  1)(  2)(  k  1)}

Example 2

Let X ~ Poisson (  ). The PGF of X is G ( s)  e  ( s 1) . Find (a) E(X) and (b) Var(X).

Solution

(a) G ( s)  e  ( s 1)

 ( )  G (1)  

(b) Consider (  1)  G (1)  2 e  ( s 1) s 1  2

So var()  (  2 )  [( )] 2
 (   1)  ( )  [( )]2
 2    2


Probability generating function for a sum of independent random variables

One of the PGF’s greatest strengths is that it turns a sum into a product: s ( 1   2 )   ( s 1 s  2 ) .
This makes the PGF useful for finding the probabilities and moments of a sum of independent
random variables.
Theorem 2: Suppose that 1 ,  2 ,,  n are independent random variables, and let
n
  1   2     n Then G (s)   Gi (s) .
i 1

Proof

G ( s )  ( s ( 1   2   n ) )

 ( s 1 s  2  s  n )
 ( s 1 )( s  2 ) ( s  n ) (because 1 ,  2 ,,  n ) are independent )
n
  Gi ( s)
i 1

Example 3

Suppose that X and Y are independent with X ∼ Poisson(  ) and Y ∼ Poisson(  ). Find the
distribution of X + Y.
Solution

G    ( s )  G  ( s )  G ( s )

 e  ( s 1) e  ( s 1)
 e (    )(s 1)
But this is the PGF of the Poisson(    ) distribution. So, by the uniqueness of PGFs,

X + Y ∼ Poisson(    ).

Summary: Properties of the PGF

Definition: G  ( s )  ( s  )

Used for: Discrete random variables with values 0, 1, 2, . . .

Moments: ( )  G (1) {(   1)(   2)  (   k  1)}  G (k ) (1)

1 (n )
Probabilities: (   n)  G (0)
n!

Sums: G   ( s)  G ( s)G ( s) for independent X, Y.