3.

Conditional Expectations and Discrete-Time Kalman Filtering

Introduction

- Model: the measurement is modeled as the sum of two 2 RVs.

z=x +v
z ≔ measurement
- Known fact

x : a state , f ( x ) , v :noise , f (v)

- Unknown fact (You may have to calculate )

f ( z ) , f ( x , z ) , f ( x|z ) , …
- Given a measure of z , What is an estimator ^x of x ? For example

^x =E [ z ] , ^x =E [ x|z ] ,∨others
 Kim: Example

If you measure your temperature, there are two R.V. Let two random variables, x :
temperature, v : measurement sensor noise.

You measured yours as z 1.Which is the best estimator of your temperature?

1) The simple one may be the estimator is E [ z ]. So how to get E [ z ]?

If you assume they are Gaussians, E [ z ] =E [ x ] + E[v ] . Or you may statistical average.

Is this the best?

2) The answer is no. The best one is

E [x∨z ]
How to get this? Later we will show if they are Gaussian, then

E [ x|z ]=x + A ( z−x )=x + P V

−1
( z−x ) (3.14)
Where this is the BASIC, FUNDAMENTAL LAW…OK… Then the problem is how to find
E [x∨z ] if they are not Gaussian. It is Example 3.8. It is not simple…

1) The minimum variance estimator ^x MV

^x MV =argmin E[ ( x−^x )2 ¿ ¿ z ]¿
^x

1.1) Some textbooks, it is called as the minimum mean square error

https://en.wikipedia.org/wiki/Minimum_mean_square_error#:~:targetText=In
%20statistics%20and%20signal%20processing,values%20of%20a%20dependent
 %20variable.

1.2) Identification : the least (mean) square error

N N
J ( ϑ^ ) =arg min ∑ ε 2 ( t ) =arg min ∑ ( y−ϕT ϑ ) (5.3)
2

ϑ t=1 ϑ t =1

2) The maximum a posteriori estimator ^x MAP ^x MV =argmax f ( x|z )

x

Bayesian probability

f ( x , z ) f ( z|x ) f ( x )
f ( x|z )= =
f (z) f (z)
 f ( x|z ) : = the posteriori PDF

 f ( x ): = the priori PDF

 f ( z|x ) : the likelihood

%% Kim

The posterior : after measurement, estimate the source

The priori : before measurement

%%%%

3.1 Minimum Variance Estimation

 Problem statement – static parameter estimation

z=x +ν
where extimate the states x ∈ R n , z ∈measurement ,h is known , ν noise RVectors .

 The minimum variance estimator

^x MV =argmin E[ ( x−^x )2 ¿ ¿ z ]¿
^x

 Remark 3.1.argmin

argmin ( x +2 ) =0
2

 (Skip): Define convex distance function ρ(x )

ρ ( . ) Is a non-negative and convex distance function.

-. Convex function: a function is convex if for any points, x , y and some scalar α
such that

0 ≤ α ≤ 1, ρ ( αx + (1−α ) y ) ≤ αρ ( x )+ ( 1−α ) ρ ( y )

-. Example of a convex function

 ρ ( x )=√ xT x .
 (Skip): Define a loss function L(.), such that

-. L ( 0 )=0

-. ρ ( x 1 ) ≥ ρ ( x 2 ) ≥ 0=¿ L ( x 1 ) ≥ L ( x 2 ) ≥ 0

(Skip) Theorem 3.2(Sherman’s Theorem). Let x be a random vector with mean, μ, and
density, f x (∙). Let L ( e ) , e=x−^x , be a loss function as defined above. If f x (∙). Is
symmetric about μ and unimodal(i.e., has only one peak), then ^x =μ minimizes
E [L ( e )] .
 Without observation, the minimum variance estimator.

argmin E[ ( X−a )2 ¿ ]=E [ X ]¿

a

Sol:

A=E[ ( X−a ) ]=E [ X 2−2 aX +a 2 ]

2

∂A
=−2 E [ X ] +2 a=0
∂a
 Remark Sherman’s theorem is generalized of the above estimator.

%%%% Kim’s comment

Minimum variance estimator:

^x MV =argmin E[ ( x−^x )2 ¿ ]−→ x^ MV =E [ x]¿

^x

Minimum variance (conditional) estimator: ^x MV =argmin

^x
E ¿¿

 There are two MVE. But E [ x ] is the MVE also, the expectation.

 Usually MVE is the conditional minimum variance estimator E [x∨z ]

And in Linear algebra

Least Square solution = least square error estimator:

2
min ( b−Ax ) , Ax=b
−1
 Solution : ^x = ( A T A ) A T b not to be confused with MVE
%%%

 Theorem 3.6. Given the equation(3.1). if the estimate is a function of z , then the
minimum variance estimate is the conditional mean.

(Skip) Proof: let a (¿ thetext book ^x ( z ) ) be an estimator of x ( that means given z , a is

a constant).
 Then for S ≥ 0.

E [ L ( e ) ]=E [ e Se ]=E [ ( x−a ) S ( x−a ) ]

T T

¿ E[ E [ ( x−μ+ μ−a ) S ( x−μ+ μ−a|z ) ] ]

T

Since the cross term is

E [ ( x−μ ) S ( μ−a )|z ]=E [ ( x−μ ) |z ] S ( μ−a )=0.

T T T

Then

min E ¿ ¿
a

¿ min ¿
a

Since ( x , μ ) are independent of a and is positive semidefinite,

E [ ( x−μ )T S ( x−μ )∨z ]+ min E [ ( μ−a )T S ( μ−a )|z ] ]

a

(a) is equivalent to

min E [ ( μ−a )T S ( μ−a )|z ] ]=μ . QED

a

 Remarks:

z=x +w
1) It means

^x MV =argmin E ¿¿
^x

2) It is unbiased

E [ ^x MV ]=E [ E [ X|Z ] ]=E [ X ]

%% Kim, comment

It is important that

E [ ^x MV ]=E [ E [ X|Z ] ]=E [ X ]

dx
For example, =−2 x , x ( 0 ) is a R .V .
dt
 The solution is
−2 t
x ( t )=e x ( 0 ): which is a R.V. dependent of the initial point R.V. x (0)
−2 t
What is MVE? Yap ^x MV =x ( t )=e x ( 0 )=E [x (t)∨x ( 0 )]

Now what is E [ ^x MV ]=E [ E [ x (t )| x(0) ] ]=E[ x (t )]

 %%%%

The following example by M.Idan should be carefully understood. The main goal is to
verify

var (E [ x|z ] )≤ var ( x )

%%% kim’s comment

Let tow random variables X , Y which are independent with pdf as f ( x ) , g ( y ) . Define Z

Z=X +Y ;
Find pdf of Z

Solution:
z z ∞ z−x
P ( Z ≤ z )= ∫ f ( z ) dz= ∫ f (x , y )dz =∫ ∫ f ( x , y ) dy dx
−∞ −∞ −∞ −∞

∞ z− x
¿ ∫ f (x ) ∫ g ( y ) dy dx ∵ f ( x , y ) =f ( x ) g ( y )
−∞ −∞

Now

( ) ( )
∞ z− x ∞ z− x
df ( z ) d P ( Z ≤ z ) d d dy
dz
=
dz
=
dz
∫ f ( x) ∫ g ( y ) dy dx =∫ f (x )
dz
∫ g ( y ) dy dx
−∞ −∞ −∞ −∞

∞
¿ ∫ f ( x ) g ( z−x ) dx
−∞

= ∫ f ( z − y ) g ( y ) dy
−∞

 This integral is called the convolution integral.

Examp.

{
f ( x )=g ( y )= 1 , if 0 ≤ x ≤1 , 0 ≤ y ≤ 1 ( a .1)
0 otherwise

Define Z=X +Y Find f Z (z )

∞

f( z )= ∫ f ( x ) g ( z−x ) dx
−∞

a) For z ≤ 0, i.e., x + y ≤ 0 , since (a.1), f( x )=0 ,

∞
f Z ( z ) =∫ f ( x ) g ( z−x ) dx =0
−∞

b) For 0 ≤ z ≤ 1, i.e., 0 ≤ x+ y ≤ 1, from (a.1)

 z
f Z ( z ) =∫ f ( x ) g ( z−x ) dx=z
0

c) For 1 ≤ z ≤2 , from (a.1)

1
f Z ( z ) = ∫ dx=2−z
z −1

d) For 2 ≤ z from (a.1)

z
f Z ( z ) =∫ 0 0 dx=0
2

e) In conclusion

{
z0≤ z≤1
f ( z )= 2−z 1 ≤ z ≤2
0 otherwise

Or you may plot f ( x ) , g ( y ) then calculated.

z = x+y

∞
f( z )= ∫ f ( x ) g ( z−x ) dx
−∞

1. Plot each pdf

2. Change the graph one of them

g(-x+z)
g(x) g(-x)

-1+z z
1
 -1
3. Increase z from negative infinity to positive infinity to integrate the convolution integral

g(-x+z) f(x)

-1+z z
z

1) If z <0, there is no overlapped portion, so that f( z )= ∫ f ( x ) g ( z−x ) dx=0

−∞

2) If z >0∧−1+ z< 0−→ 0< z <1

There is overlapped portion so that

z z
f( z )= ∫ f ( x ) g ( z −x ) dx=∫ ( 1 ) ( 1 ) dx=z
0 0

3) If −1+ z <1 , z >1−→ 1< z <2, there is a overlapped portion so that

1 1

f( z )= ∫ 1dx= ∫ 1 dx=2−z
−1 +z −1+ z

4) If If −1+ z >1, there is no overlapped portion so that

f( z )= ∫ 0 dx=0
2

f(x)
g(-x+z) f(x) g(-x+z)

-1+z z -1+z z
%%%%%%

 Ex.3.8: conditional mean and variance of the sum of two RV’s

In tutorial We well consider it.