Scribd
Upload
8 views4 pages

Problem 1: Otherwise, 0 X 0 1), 0 ( ) (

The document outlines several problems related to statistical estimation, including Maximum Likelihood estimation for various probability densities such as Exponential, Rayleigh, and Beta distributions. It also discusses the Expectation Maximization (EM) algorithm for estimating parameters of a distribution using incomplete data, and provides a nonparametric classification problem involving Parzen window and K Nearest Neighbor methods. Each problem requires analytical solutions or graphical representations to illustrate the concepts discussed.

Uploaded by

Elaheh Lotfi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
8 views4 pages

Problem 1: Otherwise, 0 X 0 1), 0 ( ) (

The document outlines several problems related to statistical estimation, including Maximum Likelihood estimation for various probability densities such as Exponential, Rayleigh, and Beta distributions. It also discusses the Expectation Maximization (EM) algorithm for estimating parameters of a distribution using incomplete data, and provides a nonparametric classification problem involving Parzen window and K Nearest Neighbor methods. Each problem requires analytical solutions or graphical representations to illustrate the concepts discussed.

Uploaded by

Elaheh Lotfi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

‫ﺑﺎﺳﻤﻪ ﺗﻌﺎﻟﻲ‬

‫داﻧﺸﮕﺎه ﺗﻬﺮان‬
‫داﻧﺸﻜﺪه ﻣﻬﻨﺪﺳﻲ ﺑﺮق و ﻛﺎﻣﭙﻴﻮﺗﺮ‬
‫درس ﺑﺎزﺷﻨﺎﺧﺖ اﻟﮕﻮ‬
‫آﺑﺎنﺁﺑﺎﻧﻤﺎﻩ‬
1389 2310‫ﺩﻭﺷﻨﺒﻪ‬
‫ دوﺷﻨﺒﻪ‬:‫ﻣﻮﻋﺪ ﺗﺤﻮﻳﻞ‬ ‫ﺗﻤﺮﻳﻦ ﺳﺮي دوم‬

Problem 1
Let {x k }, k  1,2,..., N denote independent training samples from one of the following densities.
Obtain the Maximum Likelihood estimate of θ in each case.
a) f ( x k ; )   exp(x k ) xk  0  0 Exponential Density
2
xk x
b) f ( x k ; )  2 exp(- k 2 ) xk  0  0 Rayleigh Density
 2
 1
c) f ( x k ; )   x k 0  xk  1  0 Beta Density

Problem 2
Let x have a uniform density
 1 0  x 
f X ( x  ) ~ U (0,  )   
0 otherwise,
a) Suppose that n samples D  {x1 , x 2 ,..., x n } are drawn independently according to
f X ( x  ) . Show that the maximum likelihood estimate for  is max[D], i.e., the value of
the maximum element in D.
b) Suppose that n=5 points are drawn from the distribution and the maximum value of
which happens to be max k xk = 0.6. Plot the likelihood f X ( D  ) in the range 0    1.
Explain in words why you do not need to know the values of the other four points.
Problem 3
Let us review the Expectation Maximization (EM) algorithm . We want to estimate a
parametric pdf / y(y;O) using observations on X. The random vector X is re lated to Y
through a many to one mappin g g( . ) , where g : R n ~ R I and /I > ! . This is in deed
estimation of di stribution of Y using incomplete data X = {x ... } i_I'
where x ... = g(y ... ) ,
k ~ 1 , 2, ... , N
Now, let use the idea of Maximum Likelihood (ML) estimation, i.e. fin ding the
parameters 0 in such a way that the observed outcome become the most likely outcome of
the random experiment. Therefore, we should maximize the fo llowing joint distribution
which is ca lled the likelihood function
N
1(0) ~ f( {y ,}: _,; OI X) ~ IT f , (y ,; 0 I x, )
,-,
or equivalently we may maximize the lOll-likelihood function
N
L(O) ~ ln f( {y , }:_,;OI X) ~ L:ln f , (y , ;0 I x, )
,-,
If observation x ... corresponds to a unique y ... (complete data) then L(O) is a deterministic

,
functio n of 0 and the ML estimate of 0 would be 0 ML = arg maxL (O) . However, if

observation x... cOlTesponds to a subset Y (x ... ) = {y Ig (y) = x ... } of R n (incomplete


data) -instead of a unique y ... - then L(O) is a stocha stic functi on of 0 and the ML
estimate would be 0M ,
. L = arg max £ {L(O)} , where the expected value is taken over
non-observed part of y ... 's.
N
E{L (O)} ~ L:E{ln f , (y , ;0 Ix,)}
,-,
In general, such a nonlinear maximization problem demands for iterative optimization
methods like steepest de scent or Newton method. EM is an alternative iterative
optimization method for this particular problem, which offers a rather fa st guaranteed
convergence.
E-step - At (t+ l )th iteration compute the expected value of L(O) , assuming O(t) as the
true value of 0 for the expectation computation
, ,N ~
Q(O:0(1» ~ E {L (O) IO(I)} ~ L: E {ln f , (y ,; 0 Ix,) IO(I)}
,-,

2
-
j\il-step - Compute eCr + 1) by maximizing Q(9;9(t»
-
- -
,
9(1 + 1) ~ argmllxQ(9;9(1))
Now let use EM algorithm in mixhu"e density model
, ,
", (x:9) ~ L f( x:9 1j)p j Pj ~ O L Pj = l
) _1 ) _1

Assume y = (x,}) , then


N N N
L(9) ~ L ln f(y, ;9 1x,) ~ L ln f( x, ,j, ;9) ~ L In(f(x, ;9 1j, )p j ,)
,-, ,-, ,-,
Now the parameters vector for EM algorithm is 0 = [OT,(Pl , P 2' .. , p J )7 y, and
N
Q(0 ;13(1)) ~ L
,-,
E {(Inf(x, ;9 1j, )p j,l 113(1) )
N ,
~ L L p(j,; e (l) 1x, )In(f(x, ; 9 1j,)p j,l
.1"_ 1 ) _1
N ,
~ L L p(j; e (l) 1x,) ln(f(x,; 9 1j)p)
.1"_ 1 ) _1

In case of Gaussian disttibutioll we have


. 1 II x' -~ j ll'
f( x , ; 9 I J) ~ , ,,, exp(- ,)
(2 iTer;) - 2Jj
so
- f f
Q(0 ;0 (1)) ~ L-L- p (j; 0 (1) x,){-
- 1
II x, - ~ II'
"
/ ,
- '2 ln(2lTO"j) + ln p j }
'< _1 j _ 1 2cr;
(a) Find iterative estimation forlllulas for ,.. j ' 0) , and Pj by maximizing Q(0 :0(t» .
Compare your results with page 39 of your text book.
(b) How can p(j; 0 (t) I x.t) be computed at each iteration. Explain and dlive formula.
(c) Specify all the quantities that 0 (1 ) is an explicit fUllction of.

3
Problem 4

Problem 5 – optional
Nonparametric Estimate and Classification
The figure below illustrates two dimensional training samples from two classes.  and  
represent classes 1 and 2, respectively. Classify the test sample (0.5,0) t using:
a) Parzen window approximation using the uniform window function
 1
1 u i  , i  1,2
 (u1 , u 2 )   2
0 otherwise
i- v N is a square with a side of hN  2
ii- v N is a circle with a radius of rN  1
b) Unbiased K Nearest neighbor approximation with k=3, and
i- Euclidean distance: d ( x, y )   (x
i
i  yi ) 2

ii- City Block distance: d ( x, y )  maxi xi  y i

Any point on the border of the volume is considered an inside point.

You might also like