151 ParameterEstimation

T X 0 map's one or multipleobservation's to 8 0 or also otherparameter É s

upperboundof an interval 6 If
duntftynistic

EX3
9 Tea'qÉ 2
29 fi
t ah
g me asymptoticunabbased
showestimator isunbiased eg me eggaen
consistentEstimator
ayyy xm of andconsistent

III
Asymptoticallyvanishingvariance co oIngvar
unbiased Estimator E Tex xn o fan estimator isbiased wecancreate theunbiased
version by nonmalisingthem
Bias 03 Ex OED O true

Yamana VarCTCXi XN ECT ECT ECT EET

IIIIItion
IIvarCaxtb a'varix
Mean Square Error
Forunbiasedestimate o ECT and hence
Elect EAoy MSE varCt

Bias Variancetrade off ELECT ECT ECT LECT O EvanCT Bias

It at
commonly used PDFs
mean variance
Binomial distribution pe x k E pkg pyn k
np npc p

4
Geometricdistribution Prix k fi p p Hp Xp
Normal distribution f 2 4,02 Har exp 2 te o
a a s
Uniform distribution for a b a ay cby
5
Exponentialdistribution fan XE Ha 4 2

6 Poisson distribution f KIX é x x

Il MaximumLikelihoodEstimation
observed data becomes most
Tmi is Xml angry a ex xn og pausible in terms of statistical
model
same i i d
Discrete
II PoCai IIIFxCai o
contends next Xm
Po x XN fx
LogLikelihoodFunction max log Lexi Xm o
gm angOED
ML ChannelEstimation
Yi I sit Ni EELyi Osi Ily 02112 Cgt stot Cy Os
argomelyIly
O STs sty 05112 0
G d dot 7 0
 hmu is given by pseudo Inverse Sts St ML and LSEstimator are identical
Note biasedEstimator's can providebetter estimates than unbiasedest
Best unbiased Estimator E Taxi xD 0
Um 10 and rare Tex Xn Ivan sext xn

II Fishers Information Inequality anyalternative estimate's

Auniversal lowerbound for variance x o 0

L x O differentiable w.r.to
score Function 9 x D
go cogLex a Aldo Loso
E gex o O Lex o
Cramer Rao Lower Bound
FisherInformation IFCO Varg x o
UaeETCH Z DELTA a

Yar TX I DX IF O

Isunbiased
Ira
Egg cn.io
i
mean curvature
g
of the

PropertiesofFisherInformation
17 A weak curvature of Logical an o withrespect to o at Otome
corr sponds
to little information im see sene and vice versa

27 If 10 is monotonicallyincreasing with the number of independent

observationstatistics
IMO n IEG
ExponentialModels
loggiaexp acosta
fx x o h x expCaco toe with bio dx
Fisher Mt explbco
p delta
F
p do

tix should at least provide a sufficient statistic's for estimating O means no other
Test statistic i e function of observations provides more info for the estimation
of the unknown
soft stat tax takes the value t and the conditional fx It Xlt based on tax
t is independent of o for all O E 0
Given an Exponentialmodel it can be shown that a unbiased estimator T x x toe
IFCO Daco b is a um v0 Estimator
AsymptoticalEfficient Do c Achenes the or Lower Bound
ACTU AN ONETCO O Yar TCU II lo
O is anopen set g
Fxex o is twicedifferentiable cos a
 Fisher'sInformation matrix ICO E o fix D fat xnon on aux o
diff forex
do Lex o en Cfo D do

É É I y
43
la 1 x D

Elected as
II BayesianEstimation ÉnÉInÉ yay M S E E CTX 03
unknownparameter randommanage
ConditionalMean Estimator Tom se El al x se Of 1 02 do
ELELECTA Q 033 ELECT dem t
Fin IIs the mean cost gateaux criterion

Jointpdf ofRandomparameter o
FCO fCO fix o FCO fax o False
PosteriorPDF f ope fco.gg food
off
a
Fx x
Formula on Integral
years abadz JI ebola
d Quadratic cost Function o o
Error measure E o g costrenown b Absolute Cost Function 10 81 10 817
Hit or miss cost function eco o
as Egitto o otherwise

Ifc CO 8fax a dx do

DECO 40 8 fcondo
Bayes 98 min
ommse
conditionmean
summat
8mm
If or conditionalmedian Estimator
TED MedianCQ x se

Conditionmean
Ocm ECO x HotCox re fix
of
Gaussian Standard Model No x m to kit motion o
NOE t5 10 0

501 1 xn 5 10 068
For JointlyGaussian Random Not 2 10 0
feat Co EleGe ter
arable Tom se me a x 23

MMSIE litem 0113

telcolead tr Co CoxÉxCxp
 It LinearEstimation
Tun
Bad B2 x ty go sit m Linear regression
Estimation of y by inference of t and me is based on n pairs ofjointly draws
observations Cai yo where
D and x earned
fly zig
Least Square Estimation
minimization of the sum of squarederrors betweenNpairs of
obse vations
mi and outcomes Yi of the Linear model g sett
E'Era
E Yi att Elba Y T.tt
Ey ly Xt
geometricDonations You sent
D Y StrangeEx XEV
2 Y Ye na Ext EyHy yay tag XXX
3 xt ly 57 0

ÉÉodo averse I CX't Cleft Inverse ofX text

when x non singular
xx's
Yg Egging qq.no gmu
1 NoE

gz1om 0hO99djeawno
y
projector qmsj.AT
y limo8cm mo only poor into
used
8cm Omitonlydata is
Linear Minimum meanSquare Error Estimate
GI
consider a Linear model
zig Effy
tix m
13 ya the m I sett t m
I caff
tea gY I g laytoyxÉbetux
tmmsemsmmse m.my qgyExtlay cymfate
gym
E CY Ex MDT O
E Ly Ex m 23 0 Elly the m112 Cy cyx CI ex y
zero mean R V 0
F ummse Estimator g lynxExa
minimum m se Elly 513 Cy Eaxy y y CIex y
sup y shxt r filter I IT tht thot
desend
SNR variance ofdesiredopp
variance of undesiredofp
Eummse
É m
 I
Randomsequence
Random sequence
sequence ofRandom variables Xn r Xn
b Conditional stochasticalomdependance
f y xCy x a
a Faly x 2 9,2 824 2 9 0
yo y my
p
2
peppy
why
known
Markov sequence
Xne Xna 2 Xnk i X nk k 2 storDensity

Fx
n'slink in xnGenk um p

fx.cn
send
typhoons
seilni
xn see on
zfxilxi d s
Chapman KolmogorovEquation
Given Markov sequence
n n tm nt mtl
Fxntmtextentmtelsen

Entmixtentmbendentm
Figmtefinitetentm

eg Antz xn Gentzlsen
If xntz antient anti fxntyxn anti sen d sent

d Multivariatecase e Gauss Markovsequence

Gaussiansequence that FuereemarkorProper

X fxnlxn.hnsen i Jf luxnlxn i seni

In xn i sen i

Xn SnXn e t Wn Sn China of
Xm na Snkn 1 of yn any Assume Homogeneous markor sequence
Sn S
conditional mean and covariance luxo lux luxn o 65 0 In xn i un to
Tf 02 1 gents 5h convergeofromance xnYenD
i ga 57 diverseof romance Notes

II KalmanFelter processnoise
Xn Gn xp the
un NcoCvn
Knin ElXnlying yen
T
Yh HnXn twnqmeagg.gg gyp Ctn
Gxnlyem
ECCXn Jenin Cen sinin
a stateprediction Sengn I Gnsen i n t gefcxnly.cn yen
b Conditionalstate covariance Cx un i GnCxn ingen t Cun
 and Knin
Inn t kn yn than n s exnin cxnlnt knltncxn.in i
a Predictionstep
KalmanGain kn c Mn Hot HnCxn n int Con
ininig
Grint him
ofnln i ha f Ininghntgo
b correctionstep and Thin
knglfyyf.gg ohh
11 Innovationsequence
Ayn Yn Yn n e Ayn Y
Yn than a Yn Y

Ffpredector corrector step

00
Omnoration co variance matrix
when R V Ayn n 1
zero mean
then
cyun f Caynin
co romance matrix I
Filtered state covariance matrix a
Perfection

samples Examples

mutnante
 ParticleFilter xn gncxn i.vn 9h and hn are non Linearfunction's
Yn hnc xn con Hence we need new approach
Kaman rueter suboptimumnon Lineareater's audits of
MontoCarloIntegration ggygggg.myIuter
b unsentedKalman Futer Gaussian
assumption
I
ga 19
IÉ IÉ
realisation ofgex
Importancesampling
As it is difficult to find sample in false we use inflotancefendsitynqx se o

In I Eggedfifty EItiger's
But Notnormalised
goodchoice
fax E des I
µ
Ig wi
nonagon ÉÉÉ wi 8
In E
xD fxcx
Gel
wig
fanyentenlyin
Ey x postmen

c Sequential Importantsampling gunmen onlyin impose

dens
and fmni ntiiytieit.aicn ily
ecxnlyam yins
E.io in winfxamiy.nu
axon Yinka'en yen axn Xh D Yin senimienDyen
particles
gin axn xn i yncan sein inn
win i axn i xn 2 YaGen itani zayn 1
Example

d DegeneracyProblemand Resampung Axn Xn11YnC 9Th D yenDC Fxnany yen

missingthe newestobservation
a NJGoodvalue causesDegenaryEffect
weight9 the
important
I W the 2resampung
We step
win optimum age yin
g

ResampungStep Prob of eachparticle current Importantweight Plain win

using pmF Yj l n sin pincer ani
ter resampling
importance
pinGen
me corresponds to posterior conditional PDF
Éwns un
weightwi
III fin yen Gen yens panGen
a
degeneracy

f optimalImportantdensity we know got posteriorpdf

and Hence
Suboptimal Important sampling
as the above equation optimalis hard to compute
in real worldapplication we use suboptimal
 4
2. Probability Theory Basics

zygote
Statisticalwander 1.2. Matrizen A 2 K m⇥n
2.7. Relations between fX (x), fX,Y (x, y), fX | Y (x|y)
Signal
ei
* e
A = (a ) 2 Km⇥n hat m Zeilen (Index i) und n Spalten (Index j)
* kann Spuren von Katzen enthalten
nicht für Humorallergiker geeignet
alle Angaben ohne Gewehr

mean
Processing Tranancee (Ae + ijBe )> = Ae > + Be > (Ae · Be )> = Be > · Ae > 2.1. Kombinatorik
Mögliche Variationen/Kombinationen um k Elemente von maximal n El-
fX ,Y (x, y) = fX | Y (x, y)fY (y) = fY | X (y, x)fX (x)

ˆ1
Joint PDF
ˆ1
1 > > 1 1 1 1

ai
(A ) = (A ) (A · B ) =B A ementen zu wählen bzw. k Elemente auf n Felder zu verteilen:

s tea
e e e e e e fX ,Y (x, ⇠) d⇠ = fX | Y (x, ⇠)fY (⇠) d⇠ = fX (x)

a.amp
i.i.d: independently identically distributed zcantuatcican dim K = n = rang A + dim ker A
e en⇥n
rang A = rang A>
e e 1 1
1.2.1. Quadratische Matrizen A 2 K Mit Reihenfolge Reihenfolge egal | {z } | {z }

naa n
regulär/invertierbar/nicht-singulär , det(A) 6= 0 , rang A = n Marginalization Total Probability
n+k 1
1. Math singulär/nicht-invertierbar , det(A) = 0e, rang A 6= n e Mit Wiederholung n k
k
e e
p p orthogonal , A> = A 1 ) det(A) = ±1 Ohne Wiederholung
n! n
⇡ ⇡ 3.141 59 e ⇡ 2.718 28 2 ⇡ 1.414 3 ⇡ 1.732 e e e (n k)! k
symmetrisch: A = A> schiefsymmetrisch: A = A> 2.8. Bedingte Zufallsvariablen
Binome, Trinome e e n⇥n
e e
(a ± b)2 = a2 ± 2ab + b2 a2 b2 = (a b)(a + b) 1.2.2.
" Determinante
# von
" A 2K # : det(A) = |A| Permutation von n mit jeweils k gleichen Elementen: k !·kn! Ereignis A gegeben: FX |A (x|A) = P X  x |A
e e e ⇣ ⌘ ⇣ ⌘ 1 2 !·...
(a ± b)3 = a3 ± 3a2 b + 3ab2 ± b3 A 0 A B n n n! ZV Y gegeben: FX | Y (x|y) = P X x | Y =y
det e e = det e e = det(A) det(D ) Binomialkoeffizient k = n k = k!·(n k)! pX ,Y (x,y)
(a + b + c)2 = a2 + b2 + c2 + 2ab + 2ac + 2bc C D
e f
0 D
e f
e f ⇣ ⌘ ⇣ ⌘ ⇣ ⌘ ⇣ ⌘ ⇣ ⌘ pX | Y (x|y) =
n n 4 5 6 pY (y)
det(A) = det(AT ) det(A 1 ) = det(A) 1 0
=1 1
=n 2
=6 2
= 10 2
= 15
Folgen und Reihen e e e e fX ,Y (x,y) dFX|Y (x|y)
n
P n
P 1
P det(AB ) = det(A) det(B ) = det(B ) det(A) = det(B A) fX | Y (x|y) = =
n(n+1) 1 q n+1 z n = ez fY (y) dx
k= 2
qk = 1 q n! Hat A e 2e linear abhäng.
e e
Zeilen/Spalten e |A| =e 0
) e e
k=1
Aritmetrische Summenformel
k=0
Geometrische Summenformel
n=0
Exponentialreihe
e e 2.2. Der Wahrscheinlichkeitsraum (⌦, F, P)
P 1.2.3. Eigenwerte (EW) und Eigenvektoren (EV) v 2.9. Unabhängigkeit von Zufallsvariablen
Q P P Ergebnismenge ⌦= !1 , !2 , ... Ergebnis !j 2 ⌦
Mittelwerte ( von i bis N )
pQ
(Median: Mitte einer geordneten Liste) X 1 ,···, X n sind stochastisch unabhängig, wenn für jedes x 2 Rn gilt:
1 Px xgeo = N xhm = PN1 Av = v det A = i Sp A = aii = i n
Q
xar = N i xi
e e e Ereignisalgebra F= A1 , A2 , ... Ereignis Ai ✓ ⌦ FX 1 ,···,X n (x1 ,···, xn ) = FX (xi )
Arithmetisches Geometrisches Mittel Harmonisches xi |A| i
Eigenwerte: det(A 1) = 0 Eigenvektoren: ker(A Wahrscheinlichkeitsmaß P : F ! [0, 1] P(A) = |⌦| i=1
i 1) = v i n
Q
Ungleichungen: Bernoulli-Ungleichung: (1 + x)n 1 + nx e
EW von Dreieck/Diagonal e Matrizen sind die Elem. dereHauptdiagonale.
e pX 1 ,···,X n (x1 ,···, xn ) = pX (xi )
i
|x| |y|  |x ± y|  |x| + |y| x> · y  kxk · kyk 1.2.4. Spezialfall 2 ⇥ 2 Matrix A
i=1
n
Dreiecksungleichung " # 1 " # Q
Cauchy-Schwarz-Ungleichung fX 1 ,···,X n (x1 ,···, xn ) = fX (xi )
det(A) = ad bc a b d b 2.3. Wahrscheinlichkeitsmaß P i
= det1 A i=1
Mengen: De Morgan: A \
· B =A]B A ] B = A\
· B Sp(Ae) = a + d c d c a |A|
e s✓ e P(A) = P(A [ B) = P(A) + P(B) P(A \ B)
◆ 2
|⌦|

n 1/2 =
Sp A
2e
±
spA
2e
det A 3. Common Distributions
1.1. Exp. und Log. ex := lim 1+ x
n e ⇡ 2, 71828 e 2.3.1. Axiome von Kolmogorow
n!1 Nichtnegativität: P(A) 0 ) P : F 7! [0, 1]
1.2.5. Di↵erentiation 3.1. Binomialverteilung B(n, p) mit p 2 [0, 1], n 2 N
ax = ex ln a loga x = ln x ln x  x 1
ln a @x> y @y > x @x> Ax Normiertheit: P(⌦) = 1 ! Folge von n Bernoulli-Experimenten
ln(xa ) = a ln(x) ln( x ) = ln x ln a log(1) = 0 = @x =y @xe
= (A + A> )x 1 1
a @x
e e S P p: Wahrscheinlichkeit für Erfolg k: Anzahl der Erfolge
> ⇣ ⌘ Additivität: P Ai = P(Ai ),
@x Ay @ det(B AC ) 1 >
@Ae
= xy > eee = det(B AC ) A i=1 i=1 (⇣ n⌘
k
@A p)n k k 2 {0, . . . , n}
n tricks e e e e e e wenn Ai \ Aj = ;, 8i 6= j k
p (1
pX (k) = Bn,p (k) =
Disterentation 0 sonst
1.2.6. Ableitungsregeln (8 , µ 2 R)
2.4. Bedingte Wahrscheinlichkeit

i
n
Linearität: ( f + µg)0 (x) = f 0 (x) + µg 0 (x0 ) E[X ] = np Var[X ] = np(1 p) GX (z) = (pz + 1 p)
Bedingte Wahrscheinlichkeit für A falls B bereits eingetreten ist:

on
Produkt: (f · g)0 (x) = f 0 (x)g(x) + f (x)g 0 (x) Erwartungswert Varianz Wahrscheinlichkeitserz. Funktion

olazcatzs ⇣ ⌘0 ⇣ ⌘ P(A\B)
f g(x)f 0 (x) f (x)g 0 (x) NAZ ZAN
PB (A) = P(A|B) = P(B)
Quotient: g
(x) = 2 2
g(x) N

.EE
0
2.4.1. Totale Wahrscheinlichkeit
S und Satz von Bayes
Kettenregel f g(x) = f 0 g(x) g (x) 0
Es muss gelten: Bi = ⌦ für Bi \ Bj = ;, 8i 6= j
i2I
3.2. Normalverteilung
P WDF/PDF: KVF/CDF:
Totale Wahrscheinlichkeit: P(A) = P(A|Bi ) P(Bi )
ex dx = ex = (ex )0 1.0 1.0
´
1.3. Integrale i2I φμ,σ (x)
2
μ = 0, σ 2 = 0.2,
μ = 0, σ 2 = 1.0,
μ = 0, σ 2 = 0.2,
μ = 0, σ 2 = 1.0, Φμ,σ (x)
P(A|Bk ) P(Bk ) 2

uw0 = uw
´ 0 P
Satz von Bayes: P(Bk |A) = 0.8 μ = 0, σ 2 = 5.0, 0.8 μ = 0, σ 2 = 5.0,
´
Partielle Integration: u w

É
P(A|Bi ) P(Bi ) μ = −2, σ 2 = 0.5, μ = −2, σ 2 = 0.5,
0 i2I 0.6

t
´ ´ 0.6
Substitution: f (g(x))g (x) dx = f (t) dt
Multiplikationssatz: P(A \ B) = P(A|B) P(B) = P(B|A) P(A) 0.4 0.4

F (x) C f (x) f 0 (x) 0.2 0.2

1 xq+1 xq qxq 1 2.5. Zufallsvariable 0.0 0.0

q+1

togB Bt
0 0 0 −5 −4 −3 −2 −1 0 1 2 3 4 5 −5 −4 −3 −2 −1 0 1 2 3 4 5
p X : ⌦ 7! ⌦ ist Zufallsvariable, wenn für jedes Ereignis A 2 F x x
p pa
2 ax3 ax im Bildraum ein Ereignis A im Urbildraum F existiert,
3 2 ax (x µ)2 µ2R
1 sodass ! 2 ⌦| X (!) 2 A0 2 F 1
x ln(ax) x ln(ax) x WDF: fX (x) = p e 2 2 x2R
2⇡ 2

E
1 eax (ax 1) x · eax eax (ax + 1) >0
cogae
gygax
a2

É ax
ln(a)
cos(x)
ax
sin(x)
ax ln(a)
cos(x)
2.6. Distribution
E(X ) = µ Var(X ) = 2
'X (!) = e
j!µ !2 2
2
Bezeichnung Abk. Zusammenhang Erwartungswert Varianz Charakt. Funktion
cosh(x) sinh(x) cosh(x)
1 dFX (x)
ln | cos(x)| tan(x) Wahrscheinlichkeitsdichte pdf fX (x) =
cos2 (x) dx
x́

msecomi narcoma
Kumulative Verteilungsfkt. cdf FX (x) = fX (⇠) d⇠ 3.3. Sonstiges
´ at a sin(bt)+b cos(bt)
e sin(bt) dt = eat 1 Gammadistribution (↵, ): E[X ] = ↵
a2 +b2

Cy EleymyCynot
p
´ dt 2 at+b ´ 2 at (ax 1)2 +1 at Exponential: f (x, ) = e x
E[X ] = 1
Var[X ] = 2
p = t e dt = e Joint CDF: FX ,Y (x, y) = P({X  x, Y  y})
at+b a a3
2 1 eax2
te dt = at 2 1 eat
´ at
xeax dx = 2a
s Geometricpropertiesofpor
´

sad for riya Ifxicxidxi

a
1.3.1. Volumen und Oberfläche von Rotationskörpern
p um x-Achse
V = ⇡ ab f (x)2 dx O = 2⇡ ab f (x) 1 + f 0 (x)2 dx
´ ´

ECtexia xnDO sumo

Homepage: www.latex4ei.de – Please report mistakes immediately. from LaTeX4EI – Mail: [email protected]
Estimato s
Last revised: 3. September 2018 um 15:43 Uhr (git 20) 1/4
4. Wichtige Parameter 5.2. Quality Properties of Estimators 5.5. Uniformly Minimum Variance Unbiased (UMVU) Estima- 6. Linear Estimation
Consistent: If lim T (x1 , . . . , xN ) = ✓ tors (Best unbiased estimators)
N !1 t is now the unknown parameter ✓, we want to estimate y and
4.1. Erwartungswert (1. zentrales Moment) Bias Bias(T ) := E[T (X 1 , . . . , X N )] ✓ Best unbiased estimator: Lowest Variance of all estimators. x is the input vector... review regression problem y = Ax (we
gibt den mittleren Wert einer Zufallsvariablen an unbiased if Bias(T ) = 0 (biased estimators can provide better estimates Fisher’s Information Inequality: Estimate lower bound of variance if e
solve for x), here we solve for t, because x is known
P ^ than unbiased estimators.) • L(x, ✓) > 0, 8x, ✓ (measured)! Confusing...
h i
´
µX = E[X ] = x · PX (x) = x · fX (x) dx • L(x, ✓) is di↵able for ✓
Variance Var[T ] := E (T E[T ])2 1. Training ! 2. Estimation
x2⌦0 R
Training: We observe y and x (knowing both) and then based
´ @ @ ´ L(x, ✓) dx
diskrete X :⌦!⌦0 stetige X :⌦!R • X @✓ L(x, ✓) dx = @✓ X
Score Function: on that we try to estimate y given x (only observe x) with a
E[↵ X + Y ] = ↵ E[X ] + E[Y ] X  Y ) E[X ]  E[Y ] 5.3. Mean Square Error (MSE) @ log L(x, ✓) = @✓ L(x,✓)
@ linear model ŷ = x> t
h i g(x, ✓) = @✓ E[g(x, ✓)] = 0
E[X 2 ] = Var[X] + E[X]2 2 L(x,✓)
The MSE is an extension of the Variance Var[T ] := E (T E[T ]) : Fischer Information: 
E[X Y ] = E[X ] E[Y ], falls X und Y stochastisch unabhängig
IF (✓) := Var[g(X , ✓)] = E[g(x, ✓)2 ] = @2
Umkehrung nicht möglichich: Unkorrelliertheit 6) Stoch. Unabhängig! h i E
@✓ 2
log L(X , ✓) Estimation: ŷ = x> t + m or ŷ = x> t
MSE: "[T ] = E (T ✓)2 = Var(T ) + (Bias[T ])2 Cramér-Rao Lower Bound (CRB): (if T is unbiased)
4.1.1. Für Funktionen von Zufallsvariablen g(x) ˆ ✓)2 ] Given: N observations (yi , xi ), unknown parameters t, noise m
P ^ =E[(✓ ⇣ ⌘ 2 3 2 >3
@ E[T (X )] 2
´
E[g(X )] = g(x) PX (x) = g(x)fX (x) dx Var[T (X )] 1 Var[T (X )] 1 y1 x1
x2⌦0 R @✓ I (✓) F IF (✓) 6 7 6 7
If ⇥ is also r.v. ) mean over both (e.g. y = 4 :̇ 5 X = 4 :̇ 5 Note: ŷ 6= y!!
h h Bayes est.): ii f
Mean MSE: E[(T (X ) ⇥)2 ] = E E (T (X ) ⇥)2 |⇥ = ✓ (N ) (1) yn x>
n
For N i.i.d. observations: IF (x, ✓) = N · IF (x, ✓)
4.2. Varianz (2. zentrales Moment) Problem: Estimate y based on given (known) observations x and
5.3.1. Minimum Mean Square Error (MMSE) 5.5.1. Exponential Models
ist ein Maß für die Stärke der Abweichung vom Erwartungswert h
2
i unknown parameter t with assumed linear Model: ŷ =!x> t !
Minimizes mean square error: arg min E (✓ˆ ✓) h(x) exp a(✓)t(x) @a(✓) @E[t(X )]
⇥ ⇤ If fX (x) = then IF (✓) = 0 x t
2
E[X ])2 = E[X 2 ] E[X ]2 h i
ˆ
✓ exp(b(✓)) @✓ @✓ Note y = x> t + m ! y = x > t0 with x0 = , t0 =
X = Var[X] = E (X 2 2 1 m
E (✓ˆ ✓) = E[✓ ] 2✓ˆ E[✓] + ✓ˆ2
Some Derivations: (check in exam)
h i Sometimes in Exams: ŷ = x> t , x̂ = T > y
Var[↵ X + ] = ↵2 Var[X ] Var[X ] = Cov[X , X ] d ˆ 2 ! Uniformly: Not di↵able ) no IF (✓)
" # Solution: ˆ E (✓ ✓) = 0 = 2 E[✓] + 2✓ˆ ) ✓ˆMMSE = E[✓] estimate x given y and unknown T e
d✓ (x ✓)
n
P Pn P Normal N (✓, 2 ): g(x, ✓) = 2 IF (✓) = 12 e
Var Xi = Var[X i ] + Cov[X i , X j ] K x
i=1 i=1 j6=i Binomial B(✓, K): g(x, ✓) = x
✓ 1 ✓
IF (✓) = ✓(1K ✓)
p 5.4. Maximum Likelihood
Standard Abweichung: = Var[X ] 6.1. Least Square Estimation (LSE)
Given model {X, F, P✓ ; ✓ 2 ⇥}, assume P✓ (x) or fX (x, ✓) for
observed data x. Estimate parameter ✓ so that the likelihood L(x, ✓) 5.6. Bayes Estimation (Conditional Mean) Tries to minimize the square error for linear Model: ŷLS = x> tLS
" #
or L(✓| X = x) to obtain x is maximized. N
P
4.3. Kovarianz A Priori information about ✓ is known as probability f⇥ (✓; ) with ran-
Least Square Error: min (yi x> 2
Maß für den linearen Zusammenhang zweier Variablen dom variable ⇥ and parameter . Now the conditional pdf fX |⇥ (x, ✓) i t) = min y
t
Xt
Likelihood Function: (Prob. for ✓ given x) i=1 f
is used to find ✓ by minimizing
h the mean MSE instead
i of uniformly MSE.
Cov[X , Y ] = E[(X E[X ])(Y E[Y ])> ] = Discrete: L(x1 , . . . , xN ; ✓) = P✓ (x1 , . . . , xN ) ⇥)2 |⇥ = ✓]
Mean MSE for ⇥: E E[(T(X ) tLS = (X > X ) 1 X > y
Continuous: L(x1 , . . . , xN ; ✓) = fX 1 ,...,X (x1 , . . . , xN , ✓) f f f
= E[X Y > ] E[X ] E[Y ]> = Cov[Y , X ] N Conditional Mean Estimator: ´
If N observations are Identically Independently Distributed (i.i.d.): TCM : x 7! E[⇥| X = x] = ⇥ ✓ · f⇥| X (✓|x) d✓ ŷ = X tLS 2 span(X)
N N LS f
Cov[↵ X + , Y + ] = ↵ Cov[X , Y ] Q Q f (x)f✓ (✓) fX |✓ (x)f✓ (✓)
Cov[X +U, Y +V] = Cov[X , Y ] + Cov[X , V] + Cov[U, Y ] + Cov[U, V]
L(x, ✓) = P✓ (xi ) = fX (xi )
i Posterior f⇥|X (✓|x) = ´ X |⇥ = Orthogonality Principle: N observations xi 2 Rd
i=1 i=1 ⇥ fX ,⇠ (x,⇠) d⇠ fX (x)
Y X T LS ? span[X ] , Y X T LS 2 null[X > ], thus
ML Estimator (Picks ✓): T ML : X 7! argmax{L(X , ✓)} = e fe f e fe e
4.3.1. Korrelation = standardisierte Kovarianz ✓2⇥ Hint: to calculate f⇥|X (✓|x): Replace every factor not containing ✓, X > (Y X T LS ) = 0 and if N > d ^ rang[X ] = d:
Cov[X ,Y ] Cx,y i.i.d. P f e fe
> 1 > f
⇢(X , Y ) = p = ⇢(X , Y ) 2 [ 1; 1] = argmax{log L(X , ✓)} = argmax log L(xi , ✓) such as f 1(x) with a factor and determine at the end such that T LS = (X X ) X Y
Var[X ]·Var[Y ] x· y e
✓2⇥ ✓2⇥ X f f f e
>
´
4.3.2. Kovarianzmatrix für z = (x, y)
" # " # @L(x,✓) d ! ⇥ f⇥|X (✓|x) d✓ = 1
Find Maximum: = log L(x; ✓) =0
CX CX Y Cov[X , X ] Cov[X , Y ] @✓ d✓ ˆ
✓=✓ MMSE: E[Var[X |⇥ = ✓]] 6.2. Linear Minimum Mean Square Estimator (LMMSE)
Cov[z] = C z = = Solve for ✓ to obtain ML estimator function ✓ˆML
e CX Y CY Cov[Y , X ] Cov[Y , Y ]
Multivariate Gaussian: X , ⇥ ⇠ N ) 2
= 2
+ Estimate y with linear estimator t, such that ŷ = t> x + m
X X |⇥=✓ ⇥
Immer symmetrisch: Cxy = Cyx ! Für Matrizen: C xy = C > Note: the Model does not need to be linear! The estimator is linear!
e e
yx Check quality of estimator with MSE T CM : x 7! E[⇥| X = x] = µ + C ⇥,X C X 1 (x µ )
Maximum-Likelihood Estimator is Asymptotically Efficient. However, ⇥ e e X
MMSE:
h i 
there might be not enough samples and the likelihood function is often 2
5. Estimation
2
E kT CM ⇥k2 = tr(C ✓| X ) = tr(C ⇥ C ⇥,X C X 1 C X ,⇥ ) ŷLMMSE = arg min E y (t> x + m)
not known. e e e e e t,m 2
Orthogonality Principle: !
5.1. Estimation T CM (X ) ⇥ ? h(X ) ) E[(TCM (X ) ⇥)h(X )] = 0 x
Statistic Estimation treats the problem of inferring underlying characteris- If Random joint variable z = with
MMSE Estimator: ✓ˆMMSE = arg min MSE y
tics of unknown random variables on the basis of observations of outputs ✓2⇥ ! " #
of those random variables. minimizes the MSE for all estimators µ Cx cxy
µ = x
and C z = e then
z µy e cyx cy
Sample Space ⌦ nonempty set of outputs of experiment
LMMSE Estimation of y given x is
Sigma Algebra F ✓ 2⌦ set of subsets of outputs (events) 5.7. Example: 1 1
Estimate mean ✓ of X with prior knowledge ✓ 2 ⇥ ⇠ N : ŷ = µy + cyx C x 1 (x µ ) = cyx C x x µy + cyx C x µ
Probability P : F 7! [0, 1] e x e } x
X ⇠ N (✓, X 2 2 | {z | {z e }
Random Variable X : ⌦ 7! X mapped subsets of ⌦ |⇥=✓ ) and ⇥ ⇠ N (m, ⇥ )
2 2 =t> =m
Observations: x1 , . . . , xN single values of X N ⇥ X |⇥=✓ 
✓ˆCM = E[⇥|X = x] = 2 ✓ˆML + m (x> t + m)
2
cyx Cx 1 cxy
Observation Space X possible observations of X +N 2 2 +N 2 Minimum MSE: E y = cy
X |⇥=✓ ⇥ X |⇥=✓ ⇥ 2
Unknown parameter ✓ 2 ⇥ parameter of propability function 1 Px
For N independent observations xi : ✓ˆML = N i Hint: First calculate ŷ in general and then set variables according
Estimator T : X 7! ⇥ T (X ) = ✓,ˆ finds ✓ˆ from X Large N ) ML better, small N ) CM better to system equation.
Multivariate: ŷ = T >LMMSE x T>LMMSE = C yx C x
1
e e e e
unknown parm. ✓ estimation of param. ✓ˆ If µ = 0 then
R.V. of param. ⇥ ˆ
estim. of R.V. of parm T(X ) = ⇥ z
Estimator ŷ = cy,x C x 1 x
e
Minimum MSE: E[cy,x ] = cy t> cx,y

00 00 2 0000
Homepage: www.latex4ei.de – Please report mistakes immediately.
INFANTILE
from LaTeX4EI – Mail: [email protected]
__ _____
Last revised: 3. September 2018 um 15:43 Uhr (git 20) 2/4
6.3. Matched Filter Estimator (MF) 7. Gaussian Stu↵ 8.3. Hidden Markov Chains For non linear problems: Suboptimum nonlinear Filters: Extended KF,
Problem: states X i are not visible and can only be guessed indirectly as Unscented KF, ParticleFilter
For channel y = hx + v, Filtered: t> y = t> hx + t> v
khxk 7.1. Gaussian Channel a random variable Y i . 9.2. Extended Kalman (EKF)
Find Filter t> that maximizes SNR = kvk Channel: Y = hsi + N with h ⇠ N , N ⇠ N Linear approximation of non-linear g, h
( h i) n
Q fX 2|X 1 fX 3|X 2 fX n − 1|X n − 2 fX n |X n − 1 fX n +1|X n
E (t> hx)2 fX 1 xn = gn (xn 1 , v n ) vn ⇠ N
tMF = max h i L(y1 , ..., yN ) = fY (yi , h)
i y = hn (xn 1 , wn ) wn ⇠ N
e t E (t> v)2 i=1 ⇣ ⌘ St at es X1 X2 ...... X n−1 Xn ...... n
fY (yi , h) = p 1 exp 1 (y
2 i hsi )2
In the lecture (estimate h): 9 i 2⇡ 2
8 h i 9.3. Unscented Kalman (UKF)
< E ĥH h 2 = 2 s> y
ĥM L = argmin{ y hs }= L ikel ihoods fY 1|X 1 fY 2|X 2 fY n − 1|X n − 1 fY n |X n Approximation of desired PDF fX n |Y n (xn |yn ) by Gaussian PDF.
T MF = max h i h s> s
e T : tr Var[T n] ;
e If multidimensional channel: y = S h + n:
ĥMF = T MF y T MF / C h S H C n 1 ⇣ e1 ⌘
e e e e e L(y, h) = q 1 exp (y S h)> C 1 (y S h) Obser vat ions Y1 Y2 ...... Y n−1 Yn ...... 9.4. Particle-Filter
2 For non linear state space and non-gaussian noise
det(2⇡C ) e e e
⇣ e ⌘
Conditional pdf fX |Y Likelihood pdf fY n | X n Non-linear State space:
6.4. Example l(y, h) = 1
2
log(det(2⇡C ) (y S h)> C 1 (y S h) n n
e e e e State-transision pdf fX n | X xn = gn (xn 1 , v n )
η d
(y S h)> C 1 (y S h) = 2S > C 1 (y S h) Estimation:
n 1
y = hn (xn 1 , wn )
dh n
e e e e e e
Gaussian Covariance: if Y ⇠ N (0, 2 ), N ⇠ N (0, 2 ):
ˆ
likelihood
fX |Y / fY |X · fX |X · fX dxn 1 z }| {
C Y = Cov[Y , Y ] = E[(Y µ)(Y µ)> ] = E[Y Y > ] n n n n X n n 1 n 1 |Yn 1
sn h yn e Posterior Conditional PDF: fX n |Y n (xn |yn ) / fY n |X n (yn |xn ) ·
For Channel Y = Sh + N: E[Y Y > ] = S E[hh> ]S > + E[NN> ] ˆ
· fX n |X (xn |xn 1 ) fX (xn 1 |yn 1 ) dxn 1
K M 9. Recursive Estimation n 1 n 1 |Y n 1
X | {z }| {z }
System Model: y = H sn + ⌘ n 7.2. Multivariate Gaussian Distributions state transition last conditional PDF
n f
with H = (hm,k ) 2 CM ⇥K (m 2 [1, M ], k 2 [1, K]) A vector x of n independent Gaussian random variables xi is jointly Gaus- 9.1. Kalman-Filter i
N random Particles with particle weight wn at time n
LinearfChannel Model y = S h + n with sian. If x ⇠ N (µ , C x ): recursively calculates the most likely state from previous state estimates N
1 P g̃(xi )
x e
h ⇠ N (0, C h ) and n ⇠ Ne (0, C n ) and current observation. Shows optimum performance for Gauss-Markov Monte-Carlo-Integration: I = E[g(X )] ⇡ IN = N
e e Sequences. i=1
fx (x) = fx1 ,...,xn (x1 , ..., xn ) = Importance Sampling: Instead of fX (x) use Importance Density qX (x)
Linear Estimator T estimates ĥ = T y 2 CM K
✓ N i
e e ⇣ ⌘> ⇣ ⌘◆ State space: 1 P w̃ i g(xi ) with weights w̃ i = fX (x )
T MMSE = C hy C y 1 = C h S H (S C h S H + C n ) 1 IN = N
e e e e e ee e e = q 1 exp 1 x
2
µ Cx 1 x µ x n = Gn x n 1 + B u n + v n i=1
iqX (x )
det(2⇡C x ) x e x
T ML = T Cor = (S H C n 1 S ) 1 S H C n 1 e y =H e nx + w e N
P
e e n n n
w g(xi ) with wi =
i w̃i
H e 1e e e e
´
f If fX n (x) dx 6= 1 then IN =
T MF / C h S C n N
P
Affine transformations y = Ax + b are jointly Gaussian with i=1 w̃i
e e e e With gaussian process/measurement noise v n /wn
For Assumption S H S = N s 2
1K⇥M and C n = ⌘ 2
1N ⇥M e i=1
y ⇠ N (Aµ + b, AC x A> ) Short notation: E[xn |y ] = x̂n|n 1 E[xn |y ] = x̂n|n
e e e e e e x e e e n 1 n
All marginal PDFs are Gaussian as well E[y |y ] = ŷ E[y |y ] = ŷ
n n 1 n|n 1 n n n|n
Contour Lines 9.5. Conditional Stochastical Independence
Ellipsoid with central point E[y] and main axis are the eigenvectors of 1. step: Prediction P(A \ B|E) = P(A|E) · P(B|E)
1 Mean: x̂n|n 1 = Gn x̂n 1|n 1
Cy e
e Covariance: C x = Gn C x G> n + Cv
Given Y , X and Z are independent if
e n|n 1 e e n 1|n 1 e e fZ | Y ,X (z|y, x) = fZ | Y (z|y) or
7.3. Conditional Gaussian 2. step: Update fX ,Z | Y (x, z|y) = fZ | Y (z|y) · fX | Y (x|y)
A ⇠ N (µ , C A ), B ⇠ N (µ , C B ) ⇣ ⌘
A e B e Mean: x̂n|n = x̂n|n 1 + K n y H n x̂n|n 1 fZ | X ,Y (z|x, y) = fZ | Y (z|y) or fX | Z ,Y (x|z, y) = fX | Y (x|y)
) (A|B = b) ⇠ N (µ ,C ) f n f
A|B e A|B Covariance: C x = Cx + KnHnCx
e n|n e n|n 1 f f e n|n 1

MMM
Conditional Mean: ⇣ ⌘ correction: E[X n | Y n= yn ] 10. Hypothesis Testing
E[A|B = b] = µ = µ + C AB C BB1 b µ z }| {
A|B=b A e e B ⇣ ⌘ making a decision based on the observations
Conditional Variance: x̂n|n = x̂n|n 1 + Kn y H n x̂n|n 1
| {z } f | n f{z
C A|B = C AA C AB C BB1 C BA }
e e e e e estimation E[X n | Y n 1 =yn 1] innovation: yn

7.4. Misc With optimal Kalman-gain (prediction for xn based on yn ):

6.5. Estimators If CDF of gaussian distribution given (z) ⇠ N (0, 1) then for X ⇠ >
2 Px Kn = Cx H>n (H n C xn|n 1 H n + C wn )
1
Upper Bound: Uniform in [0; ✓] : ✓ˆML = N N (1, 1) the CDF is given as (x µx ) e n|n 1 f
i
x+1
f |f e {zf e }
Probability p for B(p, N ): p̂ML = Nx p̂CM = N +2 C y
e n
Mean µ for N (µ, 2 ) : µ̂2
N
1 P x 8. Sequences Innovation: closeness of the estimated mean value to the real value
ML = N i
i=1 y =y ŷ =y H n x̂n|n 1
n n n|n 1 n f
N
Variance 2
for N (µ, 2 ) : ˆ ML
2 1 P (x
= N µ)2 8.1. Random Sequences
i 2
Sequence of a random variable. Example: result of a dice is RV, roll a Init: x̂0| 1 = E[X 0 ] = Var[X 0 ]
i=1 ´ 0| 1
dice several times is a random sequence. MMSE Estimator: x̂ = xn fX n | Y (xn |y ) dxn
(n) (n)
8.2. Markov Sequence Xn : ⌦ ! Xn
Sequence of memoryless state transitions with certain probabilities.
1. state: fX 1 (x1 )
2. state: fX | X (x2 |x1 )
2 1
n. state: fX n | X (xn |xn 1 )
n 1

DETERNIENIENTE
Homepage: www.latex4ei.de – Please report mistakes immediately.
2
0 7
from LaTeX4EI – Mail: [email protected]
0 no go on is
Last revised: 3. September 2018 um 15:43 Uhr (git 20) 3/4