Good Review Materials

http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm

Random Variables (Gonzales & Woods review materials)

and
Chapt. 1: Linear Algebra Review
Random Vectors
Chapt. 2: Probability, Random Variables,
Random Vectors

Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Random variables Discrete random variables

Samples from a random variable are real numbers Discrete random variables have
A random variable is associated with a probability distribution
over these real values
a pmf (probability mass function), P
Two types of random variables P(X = a) = P(a)
Discrete
Only finitely many possible values for the random variable:
Example: Coin flip 1
P (a )
X {a1, a2, , an} X = 0 if heads
(Could also have a countable infinity of possible values)
e.g., the random variable could take any positive integer value X = 1 if tails 0.5
Each possible value has a finite probability of occurring.
Continuous
What is the pmf of this
Infinitely many possible values for the random variable random variable?
E.g., X {Real numbers} 0
a
0 1

Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Discrete random variables Continuous random variables

Discrete random variables have Continuous random variables have a
a pmf (probability mass function), P pdf (probability density function), p
P(X = a) = P(a) 1
Example: Uniform distribution
P (a )
Example: Die roll p(x) p(1.3) = ? p(2.4) = ?
X {1, 2, 3, 4, 5, 6} What is the probability
0.5
What is the 0.5 that X = 1.3 exactly:
pmf of this P(X = 1.3) = ?
random variable? x
0 1 2 Probability corresponds to area
0 under the pdf. 1.5
P(1 < X < 1.5) = ? p(x) dx = 0.25
a
1 2 3 4 5 6
1
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

1
 Continuous random variables Random variables
What is the total area under any pdf ? How much change do you have on you?

Asking a person (chosen at random) that question can
p( x)dx = 1?

be thought of as sampling from a random variable.

Example continuous random variable:

Human heights Is the random variable
p(h) p(h) p(h) Amount of change people carry
discrete or continuous?

h h h
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Random variables: Mean & Variance An important type of random variable

These formulas can be used to find the mean and
variance of a random variable when its true
probability distribution is known.
Definition Discrete r.v. Continuous r.v.

Mean
= E(X) = ai P (ai ) = x p(x) dx
i

Variance E ((X ) 2 ) (a )
i
2
P(ai ) (x ) 2
p(x) dx
Var(X) i

Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

The Gaussian distribution Estimating the Mean & Variance

( x )2
After sampling from a random variable n times,
1
p(x) = e 2 2
these formulas can be used to estimate the
2
mean and variance of the random variable.
Samples x1, x2, x3, , xn
X ~ N(, 2)
1 n
Estimated mean: m= xi
n i=1
1 n
Estimated variance: 2 = (x i m)2
n i=1
maximum
likelihood estimate
n
1
2 = (x i m)2 unbiased estimate
n 1 i=1

Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

2
 Finding mean, variance in Matlab Example continuous random variable
Samples x = [x1 x 2 x 3 L x n ] Peoples heights (made up)
Mean Gaussian
>> m = (1/n)*sum(x)
= 67 , 2 = 20
Variance x1 m

What if you went
1 x m to a planet where
= [x1 m
2
x 2 m L x n m] 2
n M heights Gaussian

x n m = 75 , 2 = 5
Method 1: >> v = (1/n)*(x-m)*(x-m) How would they be
Method 2: >> z = x-m different from us?
>> v = (1/n)*z*z
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Example continuous random variable Random vectors

An n-dimensional random vector consists of n
Time people woke up this morning random variables that are all associated with the same
Gaussian events.
= 9 , 2 = 1 Example 2-D random vector:
X where X is random variable of human heights
V =
Y Y is random variable of wake-up times

Sample n times from V. Lets collect some

v1 v 2 L v n samples and graph them:
y
x1 x 2 L x n
(wake-up
y1 y 2 L y n times)
x (heights)
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Random Vectors Mean of a random vector

What will the graph of V look like? Estimating the mean v1 v 2 L v n
of a random vector x1 x 2 L x n

n samples from V y1 y 2 L y n

1 x m x
n n
1
Mean m=
n i=1
v i = i =
n i=1 y i m y
What is mean of V ?
Mean of X is 67 67 To estimate mean of V in Matlab
m =
Mean of Y is 10 10 >> (1/n)*sum(v,2)

Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

3
 Random vector Covariance of two random variables
Example 2-D random vector:
X where X is random variable of human heights Height and wake-up time are uncorrelated, but
V = height and weight are correlated.
Y Y is random variable of human weights
Sample n times from V v1 v 2 L v n Covariance
What will graph x1 x 2 L x n Cov(X, Y) = 0 for X = height, Y = wake-up times

look like? y1 y 2 L y n Cov(X, Y) > 0 for X = height, Y = weight
Definition:
Cov(X,Y ) = E ((X x )(Y y ))

If Cov(X, Y) < 0 for two random variables X, Y , what would a

scatterplot of samples from X, Y look like?
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Estimating covariance from samples Estimating covariance in Matlab

Sample n times: x1 x2 L xn Samples Means
x = [x1 x 2 x 3 L x n ] mx m_x
y1 y2 L yn
y = [y1 y 2 y 3 L y n ] my m_y
1 n
Cov(X,Y ) = (x i mx )(y i my )
n i=1
maximum Covariance y1 m y
likelihood estimate
1 y 2 my
1 n Cov(X,Y ) = [x1 m x x 2 mx L x n mx ]
Cov(X,Y ) = (x i mx )(y i my )
n 1 i=1
unbiased estimate n M

y n my
Cov(X, X) = ?Var(X) Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)
How are Cov(X, Y) and Cov(Y, X) related? Method 2: >> w = x-m_x
Cov(X, Y) = Cov(Y, X) >> z = y-m_y
>> v = (1/n)*w*z
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

Covariance matrix of a
Example covariance matrix
D-dimensional random vector
In 2 dimensions Peoples heights
X (made up)
V = X ~ N(67, 20)
Y
(
Cov(V) = E (V )(V )
T
) Time people woke up
this morning
X X Var(X ) Cov(X,Y ) Y ~ N(9, 1)
= E [X X Y Y ] =
Y Y Cov(X,Y ) Var(Y)
In D dimensions
Cov(V) = E ((V )(V ) )
T
What is the covariance
20 0
When is a covariance matrix symmetric? X

matrix of V = ? 0 1
A. always, B. sometimes, or C. never Y
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department

4
Estimating the covariance matrix from
Gaussian distribution in D dimensions
samples (including Matlab code)
Sample n times and find mean of samples 1-dimensional Gaussian is completely determined
v1 v 2 L v n
mx
by its mean, , and variance, 2 :
x x2 L xn (x )2
V = 1 m= 1
y1 y2 L yn my X ~ N(, 2) p(x) =
2 e 2 2

Find the covariance matrix x1 mx y1 m y

D-dimensional Gaussian is completely determined
1 x1 m x x 2 mx L x n mx x 2 mx y 2 my
Cov(V ) =
y 2 my L y n my M M
by its mean, , and covariance matrix, :
n y1 m y

x n mx

y n my 1 12 (x )T 1 (x )

>> m = (1/n)*sum(v,2)
X ~ N(, ) p(x) =
(2 ) D/2

1/ 2 e
>> z = v - repmat(m,1,n)
>> v = (1/n)*z*z What happens when D = 1 in the Gaussian formula?
Tim Marks, Cognitive Science Department Tim Marks, Cognitive Science Department