Scribd
Upload
73 views8 pages

Maximum Likelihood Method: MLM: pick α to maximize the probability of getting the measurements (the x 's) that we did!

This document provides an overview of the maximum likelihood method (MLM) for estimating parameters from data. It begins by stating the assumptions of MLM - that measurements are made, the probability distribution describing the measurements is known, and the goal is to determine the distribution's parameter(s). It then explains that MLM involves picking the parameter value(s) that maximize the likelihood function, which represents the probability of obtaining the actual measurements. The document provides examples of applying MLM to estimate the mean of a Gaussian and Poisson distribution. It concludes by noting some general properties of MLM, such as its consistency, unbiasedness, and efficiency for large sample sizes.

Uploaded by

Gharib Mahmoud
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
73 views8 pages

Maximum Likelihood Method: MLM: pick α to maximize the probability of getting the measurements (the x 's) that we did!

This document provides an overview of the maximum likelihood method (MLM) for estimating parameters from data. It begins by stating the assumptions of MLM - that measurements are made, the probability distribution describing the measurements is known, and the goal is to determine the distribution's parameter(s). It then explains that MLM involves picking the parameter value(s) that maximize the likelihood function, which represents the probability of obtaining the actual measurements. The document provides examples of applying MLM to estimate the mean of a Gaussian and Poisson distribution. It concludes by noting some general properties of MLM, such as its consistency, unbiasedness, and efficiency for large sample sizes.

Uploaded by

Gharib Mahmoud
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

K.K.

Gan L5: Maximum Likelihood Method 1


Lecture 5
Maximum Likelihood Method
!

x
=
1
N
x
i
i=1
N
"
! Suppose we are trying to measure the true value of some quantity (x
T
).
" We make repeated measurements of this quantity {x
1
, x
2
, x
n
}.
" The standard way to estimate x
T
from our measurements is to calculate the mean value:
# set x
T
=
x
.
# DOES THIS PROCEDURE MAKE SENSE???
# MLM: a general method for estimating parameters of interest from data.
! Statement of the Maximum Likelihood Method
" Assume we have made N measurements of x {x
1
, x
2
, x
n
}.
" Assume we know the probability distribution function that describes x: f(x, !).
" Assume we want to determine the parameter !.
# MLM: pick ! to maximize the probability of getting the measurements (the x
i
's) that we did!
! How do we use the MLM?
" The probability of measuring x
1
is f(x
1
, !)dx
" The probability of measuring x
2
is f(x
2
, !)dx
" The probability of measuring x
n
is f(x
n
, !)dx
" If the measurements are independent, the probability of getting the measurements we did is:
" We can drop the dx
n
term as it is only a proportionality constant
#
!
L = f (x
1
,")dx # f (x
2
,")dx # # # f (x
n
,")dx = f (x
1
,") # f (x
2
,") # # # f (x
n
,")dx
n
!
L = f (x
i
i=1
N
" ,#) Likelihood Function
K.K. Gan L5: Maximum Likelihood Method 2
" We want to pick the ! that maximizes L:
" Both L and lnL have maximum at the same location.
# maximize lnL rather than L itself because lnL converts the product into a summation.
# new maximization condition:
% ! could be an array of parameters (e.g. slope and intercept) or just a single variable.
% equations to determine ! range from simple linear equations to coupled non-linear equations.
! Example:
" Let f(x, !) be given by a Gaussian distribution.
" Let ! = be the mean of the Gaussian.
" We want the best estimate of ! from our set of n measurements {x
1
, x
2
, x
n
}.
" Lets assume that " is the same for each measurement.
" The likelihood function for this problem is:
!
"L
"#
#=#*
= 0
!
ln L = ln f (x
i
,")
i=1
N
#
!
" ln L
"#
#=#*
=
"
"#
i=1
N
$ ln f (x
i
,#)
#=#*
= 0
!
f (x
i
,") =
1
# 2$
e
%
( x
i
%")
2
2#
2

!
L = f (x
i
,")
i=1
n
# =
1
$ 2%
e
&
( x
i
&")
2
2$
2
i=1
n
# =
1
$ 2%
'
(
)
*
+
,
n
e
&
( x
1
&")
2
2$
2
e
&
( x
2
&")
2
2$
2
Le
&
( x
n
&")
2
2$
2
=
1
$ 2%
'
(
)
*
+
,
n
e
&
( x
i
&")
2
2$
2
i=1
n
-
K.K. Gan L5: Maximum Likelihood Method 3
" Find ! that maximizes the log likelihood function:
" If " are different for each data point
# ! is just the weighted average:
!
" ln L
"#
=
"
"#
nln
1
$ 2%
&
'
(
)
*
+
,
(x
i
,#)
2
2$
2
i=1
n
-
.
/
0
1
2
3
= 0
"
"#
(x
i
,#)
2
i=1
n
- = 0
2(x
i
i=1
n
- ,#)(,1) = 0
x
i
i=1
n
- = n#
# =
1
n
x
i
i=1
n
-
!
" =
x
i
#
i
2
i=1
n
$
1
#
i
2
i=1
n
$
Average
Weighted average
K.K. Gan L5: Maximum Likelihood Method 4
! Example
" Let f(x, !) be given by a Poisson distribution.
" Let ! = be the mean of the Poisson.
" We want the best estimate of ! from our set of n measurements {x
1
, x
2
, x
n
}.
" The likelihood function for this problem is:
" Find ! that maximizes the log likelihood function:
Some general properties of the Maximum Likelihood Method
! For large data samples (large n) the likelihood function, L, approaches a Gaussian distribution.
! Maximum likelihood estimates are usually consistent.
# For large n the estimates converge to the true value of the parameters we wish to determine.
! Maximum likelihood estimates are usually unbiased.
# For all sample sizes the parameter of interest is calculated correctly.
! Maximum likelihood estimate is efficient: the estimate has the smallest variance.
! Maximum likelihood estimate is sufficient: it uses all the information in the observations (the x
i
s).
! The solution from MLM is unique.
" Bad news: we must know the correct probability distribution for the problem at hand!
!
L = f (x
i
,")
i=1
n
# =
e
$"
"
x
i
x
i
!
i=1
n
# =
e
$"
"
x
1
x
1
!
e
$"
"
x
2
x
2
!
...
e
$"
"
x
n
x
n
!
=
e
$n"
"
x
i
i=1
n
%
x
1
! x
2
!..x
n
!
!
d ln L
d"
=
d
d"
#n"+ln" $ x
i
i=1
n
% #ln(x
1
! x
2
!..x
n
!)
&
'
(
)
*
+
= #n+
1
"
x
i
i=1
n
% = 0
" =
1
n
x
i
i=1
n
% Average
K.K. Gan L5: Maximum Likelihood Method 5
Maximum Likelihood Fit of Data to a Function
! Suppose we have a set of n measurements:
" Assume each measurement error (") is a standard deviation from a Gaussian pdf.
" Assume that for each measured value y, theres an x which is known exactly.
" Suppose we know the functional relationship between the ys and the xs:
% !, #...are parameters.
# MLM gives us a method to determine !, #... from our data.
! Example: Fitting data points to a straight line:
" Find ! and # by maximizing the likelihood function L likelihood function:
x
1
, y
1
!
1
x
2
, y
2
!
2
...
" x
n
, y
n
!
n
!
y = q(x,",#,...)
!
q(x,",#,...) = "+#x
L = f (x
i
,",#)
i=1
n
$ =
1
%
i
2&
e
'
( y
i
'q( x
i
,",#))
2
2%
i
2
i=1
n
$ =
1
%
i
2&
e
'
( y
i
'"'#x
i
)
2
2%
i
2
i=1
n
$
!
" ln L
"#
=
"
"#
ln
1
$
i
2%
&
'
(
)
*
+
,
(y
i
,# ,-x
i
)
2
2$
i
2
.
/
0
1
2
3
i=1
n
4 = ,
2(y
i
,# ,-x
i
)(,1)
2$
i
2
.
/
0
1
2
3
i=1
n
4 = 0
" ln L
"-
=
"
"-
ln
1
$
i
2%
&
'
(
)
*
+
,
(y
i
,# ,-x
i
)
2
2$
i
2
.
/
0
1
2
3
i=1
n
4 = ,
2(y
i
,# ,-x
i
)(,x
i
)
2$
i
2
.
/
0
1
2
3
i=1
n
4 = 0
two linear equations
with two unknowns
K.K. Gan L5: Maximum Likelihood Method 6
" Assume all "s are the same for simplicity:
" We now have two equations that are linear in the two unknowns, ! and #.
% We will see this problem again when we talk about least squares (chi-square) tting.
! EXAMPLE:
" A trolley moves along a track at constant speed. Suppose the following measurements of the
time vs. distance were made. From the data nd the best value for the velocity (v) of the trolley.
" Our model of the motion of the trolley tells us that:
!
y
i
i=1
n
" # $
i=1
n
" # %x
i
i=1
n
" = 0
y
i
x
i
i=1
n
" # $x
i
i=1
n
" # %x
i
2
i=1
n
" = 0
!
y
i
i=1
n
" = n#+$ x
i
i=1
n
"
y
i
x
i
i=1
n
" = # x
i
i=1
n
" +$ x
i
2
i=1
n
"
!
" =
y
i
i=1
n
# x
i
2
$ y
i
x
i
x
i
i=1
n
#
i=1
n
#
i=1
n
#
n x
i
2
$( x
i
i=1
n
# )
2
i=1
n
#
and % =
n x
i
y
i
i=1
n
# $ y
i
x
i
i=1
n
#
i=1
n
#
n x
i
2
$( x
i
i=1
n
# )
2
i=1
n
#
61 49 40 33 19 11 Distance d (mm)
6.0 5.0 4.0 3.0 2.0 1.0 Time t (seconds)
!
d = d
0
+vt
!
y
i
i=1
n
"
y
i
i=1
n
" x
i
#
$
%
%
%
%
&
'
(
(
(
(
=
n x
i
i=1
n
"
x
i
i=1
n
" x
i
2
i=1
n
"
#
$
%
%
%
%
&
'
(
(
(
(
)
*
#
$
%
%
%
&
'
(
(
(
Matrix form
Taylor Eqs. 8.10-12
K.K. Gan L5: Maximum Likelihood Method 7
" We want to find v, the slope (#) of the straight line describing the motion of the trolley.
" We need to evaluate the sums listed in the above formula:
!
x
i
i=1
n
" = t
i
i=1
6
" = 21 s
y
i
= d
i
i=1
6
"
i=1
n
" = 213 mm
x
i
y
i
i=1
n
" = t
i
d
i
i=1
6
" = 919 s # mm
x
i
2
i=1
n
" = t
i
2
i=1
6
" = 91 s
2
d
0
= 0.8 mm
!
v =
n x
i
y
i
i=1
n
" # y
i
x
i
i=1
n
"
i=1
n
"
n x
i
2
#( x
i
i=1
n
" )
2
i=1
n
"
=
6$919 #21$213
6$91#21
2
= 9.9mm/ s best estimate of the speed
best estimate of the starting point
K.K. Gan L5: Maximum Likelihood Method 8
0
10
20
30
40
50
60
70
0 1 2 3 4 5 6 7
time (sec)
_____ d = 0.8 + 9.9t
d
i
s
t
a
n
c
e

(
m
m
)
" The line best represents our data.
" Not all the data points are "on" the line.
" The line minimizes the sum of squares of the deviations between the line and our data (d
i
):
MLM fit to the data for d = d
0
+ vt
!
" = data
i
#prediction
i
[ ]
2
i=1
n
$ = d
i
#(d
0
+vt
i
)
[ ]
2
i=1
n
$
Least square fit

You might also like