EE5110: Probability Foundations for Electrical Engineers July-November 2015

Lecture 12: Multiple Random Variables and Independence

Instructor: Dr. Krishna Jagannathan Scribes: Debayani Ghosh, Gopal Krishna Kamath M, Ravi Kolla

12.1 Multiple Random Variables

In this lecture, we consider multiple random variables defined on the same probability space. To begin with,
let us consider two random variables X and Y, defined on the probability space (, F, P). It is important to
understand that the realizations of X and Y are governed by the same underlying randomness, namely .
For example, the underlying sample space could be something as complex as the weather on a particular
day; the random variable X could denote the temperature on that day, and another random variable Y, the
humidity level. Since the same underlying outcome governs both X and Y, it is reasonable to expect X and
Y to posses a certain degree of interdependence. In the above example, a high temperature on a given day
usually says something about the humidity.
In Figure (12.1), the top picture shows two random variables X and Y, each mapping to R. These two
random variables are measurable functions from the same probability space to the real line. The bottom
picture in Figure (12.1) shows (X(), Y ()) mapping to R2 . Indeed, the bottom picture is more meaningful,
since it captures the interdependence between X and Y.
Now, an important question arises: is the function (X(), Y ()) : R2 measurable, given that X
and Y are measurable functions? In order to pose this question properly and answer it, we first need
2 2
to define

the Borel -algebra on R . The Borel -algebra on R is the -algebra generated by the class
2
R , {(, x] (, y] | x, y R}. That is,

B(R2 ) = R2 .

The following theorem asserts that whenever X and Y are random variables, the function (X, Y ) : R2
is F-measurable, in the sense that the pre-images of Borel sets on R2 are necessarily events.

Theorem 12.1 Let X and Y be two random variables on (, F, P). Then, (X(), Y ()) : R2 is F-
measurable, i.e., the pre-images of Borel sets on R2 under (X(), Y ()) are events.

Proof: Let G be the collection of all subsets of
R2 whose pre-images under (X(), Y ()) are events. To prove
the theorem, it is enough to prove that B R2 G.
Claim 1: G is a - algebra of subsets of R2 .

Next, note that {|X() x}, {|Y () y} F, x, y R, since X and Y are random variables. Thus,
{|X() x} {|Y () y} F, x, y R, since F is a - algebra.
So, {|X()

x, Y ()


y} F, x, y R

(, x] (, y] G, x, y R (from the definition of G)
R2 G R2 (G) B R2 G.
Since the pre-images of Borel sets on R2 are events, we can assign probabilities to them. This leads us to
the definition of the joint probability law.

12-1
12-2 Lecture 12: Multiple Random Variables and Independence

X ()
Y () X ( )

Y ( )

Y
( X (), Y ())

( X ( ), Y ( ))

Figure 12.1: Illustration of Multiple Random Variables

Definition 12.2 The joint probability law of the random variables X and Y is defined as:

PX,Y (B) = P ({ | (X(), Y ()) B}), B B(R2 ),

where B(R2 ) is the Borel -algebra on R2 .

In particular, when B = (, x] (, y], we have

PX,Y ((, x] (, y]) = P ({|X() x, Y () y}). (12.1)

The LHS in (12.1) is well defined, and hence the RHS in (12.1) is well defined and is called as the joint CDF
of X and Y.

12.2 Joint CDF

Definition 12.3 Let X and Y be two random variables defined on the probability space (, F, P). The joint
CDF of X and Y is defined as follows:

FX,Y (x, y) = P ({|X() x, Y () y}), x, y R.

In short hand, we write FX,Y (x, y) = P (X x, Y y) .

Lecture 12: Multiple Random Variables and Independence 12-3

12.2.1 Properties of joint CDF:

1. x
lim FX,Y (x, y) = 1, lim FX,Y (x, y) = 0.
x
y y
Proof: Let {xn } and {yn } be two unbounded, monotone-increasing sequences. We have

lim F
x X,Y
lim P (X x, Y y) ,
(x, y) = x
y y

= lim P (X xn , Y yn ) ,
n

!
(a) [
= P { : X() xn , Y () yn } ,
n=1
= P () ,
= 1,

where (a) is due to continuity of probability measures (Lecture #5, Property 6). Proof of the other
part follows on the similar lines and is left as an exercise to the reader.
Note that the order of the two limits does not matter here.
2. Monotonicity: For any x1 x2 , y1 y2 , FX,Y (x1 , y1 ) FX,Y (x2 , y2 ) .
Proof: Let x1 x2 and y1 y2 . Clearly, events {X x1 , Y y1 } {X x2 , Y y2 }. Then,
P (X x1 , Y y1 ) P (X x2 , Y y2 ) FX,Y (x1 , y1 ) FX,Y (x2 , y2 ).
3. FX,Y is continuous from above, i.e., lim+ FX,Y (x + u, y + v) = FX,Y (x, y) , x, y R.
u0
v0+
Exercise: Prove this.
4. lim FX,Y (x, y) = FX (x)
y
Proof: Let {yn } be an unbounded, monotone-increasing sequence. Then, lim FX,Y (x, y) = lim FX,Y (x, yn ).
y n
Hence,

lim FX,Y (x, yn ) = lim P (X x, y yn ) ,

n n

!
(a) [
= P { : X() x, Y () yn } ,
n=1
= P ( : X() x) ,
= FX (x),

where (a) is due to continuity of probability measure.(Lecture #5, Property 6).

Using the above property, we can calculate the marginal CDFs from joint CDF. However, the joint CDF
cannot be obtained from the marginals alone, since the marginals do not capture the inter-dependence
of X and Y.

12.3 The -algebra generated by a random variable

Before we proceed to define the independence of random variables, it is useful to understand the notion of
the -algebra generated by a random variable. We first state an elementary result that holds for any arbitrary
function.
12-4 Lecture 12: Multiple Random Variables and Independence

Proposition 12.4 Let and S be two non-empty sets and let f : S be a function. If H is a -algebra
of subsets of S, then G , {A | A = f 1 (B) , B H} is a -algebra of subsets of .

In words, Proposition (12.4) says that the collection of pre-images of all the sets belonging to some -algebra
on the range of a function, is a -algebra on the domain of that function.
Let (, F, P) be a Probability Space and X : R be a random variable. X in turn induces the probability
triple (R, B(R), PX ) on the real line.

Definition 12.5 The -algebra generated by the random variable X is defined as

(X) , {E |E = X 1 (B), B B(R)}. (12.2)

Proposition (12.4) asserts that (X) defined above is indeed a -algebra on .

Proposition 12.6 (X) F, i.e., the -algebra generated by X is a sub--algebra of F.

Figure (12.3) shows a pictorial representation of the -algebra generated by X. Each Borel set B maps back
to an event E. A collection of all such preimages of Borel sets constitutes the -algebra generated by X.
Thus, (X) is a -algebra that consists precisely of those events whose occurrence or otherwise is completely
determined by looking at the realised value X(). To get a more concrete idea of this concept, let us look
at the following examples:

B
(R, B(R))

X 1

Figure 12.2: The collection of the pre-images of all Borel Sets is the -algebra generate by the random
variable X, denoted (X).

Example 1:- Let (, F, P) be a probability space and A F be some event. Consider the Indicator random
variable of event A, IA . It is easy to see that (IA ) = {, A, AC , }. Also, (IA ) F.
Lecture 12: Multiple Random Variables and Independence 12-5

Example 2:- Let ([0, 1], B([0, 1]), ) be the probability space in consideration, and consider a random vari-
able X() = , . It can be seen that (X) = F.

Remark: 12.7 As seen from the above two examples, (X) could either be small (as seen in example 1
above) or as large as the -algebra F itself (as seen in example 2 above).

Now, we introduce the important notion of independence of random variables.

12.4 Independence of Random Variables

Definition 12.8 Random variables X and Y are said to be independent if (X) and (Y ) are independent
-algebras.

In other words, X and Y are independent if, for any two borel sets B1 and B2 on R, the events { : X()
B1 } and { : Y () B2 } are independent i.e.,
 \ 
P { : X() B1 } { : Y () B2 } = P ({ : X() B1 }) P ({ : Y () B2 }) , B1 , B2 B(R).

The following theorem gives a useful characterization of independence of random variables, in terms of the
joint CDF being equal to the product of the marginals.

Theorem 12.9 X and Y are independent if and only if

FX,Y (x, y) = FX (x)FY (y).

Proof: First, we prove the necessary part, which is straightforward. Let X and Y are independent. Consider
B1 B (R), B2 B (R) then the events {|X() B1 } and {|Y () B2 } are independent (due to defi-
nition (12.8)) P (X B1 , Y B2 ) = P (X B1 ) P (Y B2 ) PX,Y (B1 B2 ) = P (X B1 ) P (Y B2 ).
But, this is true for all borel sets in R. In particular, choose B1 = (, x] and B2 = (, y] then we get
FX,Y (x, y) = FX (x)FY (y), x, y R which completes the proof of the necessary part.
The sufficiency part is more involved; refer [1][Section 4.2].

Definition 12.10 X1 , X2 , Xn random variables are said to be independent if -algebras (X1 ) , (X2 ) , ,
(Xn ) are independent i.e., for any Bi B (R) , 1 i n, we have
n
Y
P (X1 B1 , X2 B2 , Xn Bn ) = P (Xi Bi ) .
i=1

Theorem 12.11 X1 , X2 , Xn are independent if and only if

n
Y
FX1 ,X2 , ,Xn (x1 , x2 , xn ) = FXi (xi ) .
i=1

Proof: Refer [1][Section 4.2].

Finally, we define independence for an arbitrary family of random variables.

Definition 12.12 An arbitrary family of random variables, {Xi , I}, is said to be independent if the
algebras { (Xi ) , i I} are independent (Lecture #9, Section 9.2, Definition 9.7).
12-6 Lecture 12: Multiple Random Variables and Independence

12.5 Exercises
1. Prove Claim 1 under Theorem 12.1.

2. For random variables X and Y defined on same probability space, with joint CDF FX,Y (x, y), prove
that lim FX,Y (x, y) = 0.
x
y

3. Prove Propositions (12.4) and (12.6).

4. [Quiz II 2014] Suppose X and Y are independent random variables, and f (X) and g(Y ) are functions
of X and Y respectively. Will the random variables f (X) and g(Y ) be independent? Justify your
answer.

References
[1] David Williams, Probability with Martingales, Cambridge University Press, Fourteenth
Printing, 2011.