508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics

Author: Andrés Hincapié and Linyi Cao

This Version: August 1, 2016

Measure Theory Basics 3

1 Measure
A convenient framework for theoretical statistics.

“Measures”: length, area, volume

A measure is a real-valued set function µ that satisfies

1. µ (A) ≥ 0 for ∀A (as subsets of some universe/space Ω ) (non negativity )
2. µ(∅) = 0 (null empty set)
∞  ∞
S P
3. µ Ai = µ (Ai) for any countable collections of pairwise disjoint sets (sigma-additivity )
i=1 i=1

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 4

Domain of µ is a class of subsets of Ω

Why not the whole set of all subsets of Ω ?

Turns out typically it is not possible to give a satisfactory definition of measure for all subsets of Ω such
that sigma-additivity holds. See later an example with Lebesgue measure.

This is why we need σ -algebra.

A measure is sub-additive, σ − continuous from below and above.

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 5

2 σ -algebra

A σ -algebra Σ over a sample space Ω satisfies

1. φ ∈ Σ

2. For any set A ∈ Σ , AC ∈ Σ

3. Closed under countable unions

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 6

The duple (Ω, Σ) is called a measurable space

A measure µ with domain Σ

Sets in Σ are called measurable (w.r.t. µ )

The triple (Ω, Σ, µ) is called a measure space

Measure is σ -finite if there exists a family of sets Ai ∈ Σ such that their union equals Ω and µ (Ai) < ∞
, ∀i

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 7

EXAMPLE: Counting measure

Let Ω be countable and Σ be the class of all subsets of Ω . For any set A ∈ Σ , define

# of individual elements of A if A finite
µ (A) =
∞ otherwise

Is it σ -finite?

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 8

EXAMPLE: Lebesgue measure

Let Ω be a n-dimensional Euclidian space E n , Σ be the smallest σ -algebra containing all open rectan-
gles
B = {(x1, . . . , xn) : ai < xi < bi} , − ∞ < ai < bi < ∞

There exits a measure µ defined over Σ that assigns to B

µ (B) = (a1 − b1) · . . . · (an − bn)
i.e. its length/area/volume...
P
And assigns to other elements A ∈ Σ: µ(A) = inf { B∈F µ(B) : F is a countable collection of open
rectangles whose union covers A}
Violating sigma-additivity : Ω = (0, 1) , µ is the Lebesgue measure. If we work with all subsets of Ω,
denoted as 2Ω, we won’t even have finite-additivity. Check the construction of Vitali sets.
508-B (Statistics Camp, Wash U, Summer 2016)
Measure Theory Basics 9

Consider the measurable space (Ω, Σ) and the real-valued function f defined f : Ω → R .

We say f is measurable w.r.t. (Ω, Σ) if

f −1 (B) = {ω ∈ Ω : f (ω) ∈ B} ∈ Σ for any Borel set B
If (Ω, Σ) is a Borel space, then we can also call f a Borel function.

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 10

EXAMPLE: for any set A ⊆ Ω , the indicator function is defined as


1 if x ∈ A
IA (x) = I (x ∈ A) =
0 if x ∈
/A

Consider any Borel set B




 ∅ if 0, 1 ∈
/B
Ω if 0, 1 ∈ B

IA−1 (B) =

 AC if 0 ∈ B, 1 ∈ /B
if 0 ∈/ B, 1 ∈ B

 A

Now consider the smallest σ -algebra generated by A: Σ = {∅, Ω, A, AC }

Hence IA (x) is a measurable function w.r.t. (Ω, Σ) , or w.r.t. any other σ -algebras that contains A.

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 11

3 Random Variables
Back to our discussion of random variables, in this framework a random variable is a measurable
function X (ω) : Ω → R

And related to a random variable X (ω), a probability measure is defined

P ({ω ∈ Ω : X (ω) ∈ B}) = P rob(X ∈ B) = PX (B)

Recall a probability measure P satisfies

P (Ω) = 1
After defining r.v. everything becomes numerical: we talk about probabilities of Borel sets: FX (x) =
P rob(X ∈ (−∞, x])

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 12

4 Integration w.r.t. a Measure

We have talked about two main concepts: σ -algebra and measure

Another important concept is integration of a Borel function f w.r.t. a measure µ

In order to do that we need to specify a class of functions.

We define integration in three steps.

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 13

First, we define integration w.r.t. simple functions.

A simple function is a Borel function with finite support a1, . . . , am (real numbers) defined as
m
X
ψ (ω) = aiIAi (ω)
i=1

where sets Ai ∈ Σ for all i . It’s basically a weighted sum of indicator functions

Each set A1, . . . , Am ∈ Σ has an associated real value a1, . . . , am

Integration of ψ w.r.t. a measure µ is defined as

Z m
X
ψ (ω) dµ ≡ aiµ (Ai)
i=1

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 14

Second, for a non-negative Borel function f

Define Sf as the collection of all non-negative simple functions ψ such that ψ (ω) ≤ f (ω) for any ω ∈ Ω
.

Integration of f w.r.t. a measure µ is defined as

Z Z
f (ω) dµ ≡ sup{ ψ (ω) dµ : ψ ∈ Sf }

The intuition is that one can construct a non-decreasing sequence of simple functions s1, s2 . . . such
that it converges from below to f

It is the upper bound of the integrals of simple functions in Sf

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 15

Third, for a general Borel function f

Define the positive part of f as

f+ (ω) = max {0, f (ω)}
and its negative part
f− (ω) = max {0, −f (ω)}
They are both non-negative Borel functions. Hence f = f+ (ω) − f− (ω)

Integration of f w.r.t. a measure µ is given by

Z Z Z
f (ω) dµ = f+ (ω) dµ − f− (ω) dµ

when at least one of the two elements in the rhs is finite (because what is ∞ − ∞ ?)

When both elements in rhs are finite we say f is integrable w.r.t. µ

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 16

EXAMPLE:

If Ω = {ω 1, . . .} and µ is the counting measure we can use the definition of integration for simple
functions to see that

Z X
f (ω) dµ = f (ω i)
i

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 17

FUBINI’S THEOREM. Intuition: the order of integration does not matter

First, define the Cartesian product A × B of any two sets A , B as the set of ordered pairs
{(a, b) : a ∈ A, b ∈ B}
Let (Ω, Σ, µ) and (Y, B, ν) be two measure spaces.

Define Σ × B as the smallest σ -algebra containing all sets A × B with A ∈ Σ and B ∈ B

There exits a unique measure λ , called the product measure, over Σ × B that assigns µ (A) · ν (B) to
any product set A × B , and it is denoted µ × ν

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 18

EXAMPLE

Let Ω and Y be Euclidian spaces E m and E n and Σ and B be the σ -algebras of Borel sets on Ω and Y
, respectively

Then Ω × Y is the Euclidian space E m+n and Σ × B is the σ -algebra of Borel sets of E m+n

In addition, if µ and ν are Lebesgue measures on (Ω, Σ, µ) and (Y, B, ν) , respectively, then µ × ν is
the Lebesgue measure on (Ω × Y, Σ × B)

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 19

FUBINI’S THEOREM:

Let (Ω, Σ, µ) and (Y, B, ν) be measure spaces and let f be a non-negative measurable function on Ω
× Y . Then
Z Z  Z Z 
f (ω, y) dν (y) dµ (ω) = f (ω, y) dµ (ω) dν (y)
Ω Y ZY Ω

= f d (µ × ν)
Ω×Y

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 20

5 Differentiation w.r.t. a Measure

Consider a measure space (Ω, Σ, µ) and let f be a non-negative Borel function

For ∀A ∈ Σ we can define

Z
λ (ω) = f (ω) dµ
A

it can be shown that λ is another measure on (Ω, Σ)

If µ (A) = 0 implies λ (A) = 0 , we say λ is absolutely continuous w.r.t. µ , denoted as λ  µ

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 21

RADON-NIKODYM THEOREM

Let µ and λ be two measures on (Ω, Σ) and µ be σ -finite.

If λ is absolutely continuous w.r.t. µ , then there exists a non-negative Borel function f on Ω s.t. the
following holds
Z
λ (A) = f dµ for ∀A ∈ Σ
A

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 22

Following notation we can define

dλ
f=
dµ
as the Radon-Nikodym derivative or density of λ w.r.t. µ

dP
If λ = P , where P is a probability measure, dµ is the pdf of P w.r.t. µ ( µ can be either the Lebesgue
measure or the counting measure)

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 23

EXAMPLE:

Let ω 1 < ω 2 < . . . are real numbers

P
Let p1, p2, . . . be such that pi > 0 and i pi =1

Define a probability measure P on Ω = {ω 1, ω 2, . . .}

Let µ be the counting measure on (Ω, Σ)

X
P (A) = pi
{i:ω i ∈A}

Now let f (ω i) = pi for i = 1, 2, . . . Then

Z X
P (A) = f dµ = f (ω i)
A {i:ω i ∈A}

where integration is w.r.t. the counting measure.

508-B (Statistics Camp, Wash U, Summer 2016)

Measure Theory Basics 24

Measure theory allows us to deal with both cases, discrete and continuous, within a single framework
(and notation)

DISCRETE CASE: differentiation is w.r.t. the counting measure

CONTINUOUS CASE: differentiation is w.r.t. Lebesgue measure

508-B (Statistics Camp, Wash U, Summer 2016)