Maximum Likelihood Estruation (MLE)

data set
X training
:

& :
peremeters

MMLE
D
=

arg max p(X/a)

P
(x1a) =

#p(xia)
i I
(MAP)
=

Posteriori Estimation
Maximum an

⑨MAP p(a1X
=
org max (a)
=org max P(x)
 tric Regression
YN+1= ?
:

- xN+1 ->

g(x 1/4)
y f(x) t
N N
+

f(xN 1)
=
+

->
+ 1 =
+
e

with g(x/a)
noise
f(x)
n

underlyrug approximate

[
=>

process E[x] W
2)
=

~N(t ; 0
Assumptions p(t) =N
,

=[x ]
: + c

~Nygly
+

VAR[X] K2
⑪ p(yIx)
=

~ &
VAR(x+C] =
K

y1x7f(x)
t -

E(g(xa) e]
+

=(yIx]
+

E
=

y(x=g(x1a) ur
+

E E(g(x1a)] + E[t]
V
constant R =
e
e
. .

0
- g(x(a) -

VAR[g(x1a) t]
-

x NN(x ; 0 9) VAR[y1x] =
+

, 2

x +
5NN(X ; 5, 9) -
VARCE] =
&
 yi)31tR R

a
:
x
=

2(xi ,
y

(Xi (xi yi)

I
, yi) ~
p -
,
p(x 2),

↳ i .
I d
. .
=

p(x)y)p(y)
p(x y),

) p(yIx)p(x)
3N) px
=

:
y
p(x x2 y2 , XN ,

ye
=
,
---

S
, , ,
-
,

xi
aax
XN
-
+ 1

=
loglikelihood legt (01x) log(#, [p (yi1xi) p(xi)]] constant
- alo
=>
=

="flogp (yixi) logp(xi)] +

e. leg [NI
i
leg plyilxi)
1
=

maximize i
=> maximiz
e
 r exp[-] ·
=
N(x; r ,
w)
constant R V N
-
.
.

↓ ↓
N(y ;
↓
]
"[P(-Se
2

rexx(
=

maximize
-
-

maximize #g(xila 3 I
22 constant
->

2ndorder ->
-
- e

g(xila)
i
-
1

polynomiminimize
al
- =

* (yi g(xia)) g(xila)imear ->

↓ ~

Xi +W2
X: Yi ↓ n
Note
.

↓
.

Wo+ W Xi yi
x23
=

N
& 3 we Gwx0 x13
In 2
, , -
a
E ei
Cy -y-
=
=

,
minimize : :
=
-

1
=

.
-
ei
minimize
E (3i -
g(xia)] g(xi(0) wo+wXi
=

Ew , w
, A 9
O 2 =

Error[aIX] "[ [3 : -(wo+we x=)]

i
=
1

-
r
2
zwe i 1
=

8 Error
-
A -

8w1 i 1
=

⑦E
I Yi I

i =
1
We

+
(yixi) =
iy
Ewo Xi+ .
3 Xi
we .

a i 1
o
i
= =
I
a)
wheN
-Wit
#
Furth
2
z

Oyi

"b
R
N

i
=
1
. Xi
i

A
=

N
1

⑨ M,

↳
-

A
=
=

Gixe nig
when
!b murtible
: .

O Exercise
-is statement
= .

matrit this
wN=1 A
prove

[ii]
2
0
a =
det( A) =

x1 .
-

x1 x1
.
=

↓
not
invertible .
 K th order polynomial ↓!!
miel
Regression e nk
to mony

g(Xilwo we
, ...,
WK) 8
=

ur
Wotwixit ......
+
WKXi exon
,
questrous
lineer constant /me the
regression K 0 = M
1
=

k =
=>
i previous
N years
( 2 xi)

~=
=>

·
N(wo)
=

=
, ,

i = I
N

I Exi -

N
N
Exi
->
-

I
4
x
G:
.

x? WK
i
=
- 1
(x )x
I
i 1 i
=
1
- + 1x 1 +

~+x O b
DTD <= A
 Show that

=I
:

Exercise
A
DT .
D
=

if N < K +
1
,
DTD is not invertible

if N>, K +
1
,
DT D .
is invertible

⑭
0

xi X2 T D I
K N

(8) f I
I
-
2 A
0 1
Exi

I---
1 ...

X
X ,
W
i ↳

+ 1)
( 1)x(k+)
+

1) x(N) Nx(k
(k +

(N 2) (*
=

N
k
=

=
25

3
[Yx5[] 5
2x4
=

< Jx rock-deficient