1

Object Tracking Using Kalman Filter

Shahin Khobahi

I. I NTRODUCTION we subtract each observation ui from the estimated signal x

to obtain an estimation of the noise at each frame:
In this project, we are proposing an adaptive filter ap-
proach to track a moving object in a video. Currently, object vi = ui x (2)
tracking is an important issue in many applications such as
video survelance, traffic management, video indexing, machine Given two deterministic random variables {x, y}, we define
learning, artificial intelligence and many other related fields. the least-mean-squares estimator (l.m.s.e) of x given y as the
As we will discuss in the following sections, moving object conditional expectation of x given y:
tracking can be interpreted as an estimation problem. Kalman x = E(x|y) = E[x|u0 , u2 , ..., uN 1 ] (3)
filter is a powerful algorithm that can be used in the state
estimation problems and that is the reason we used this method For simplicity, we assume that we model x as an unknown
to estimate and predict the position of a moving object. In the constant instead of a random variable. Further we assume
first stage of this project we use the background subtraction that we are given N frames of the video and that is all the
method to detect the moving object in the video and then we information we have. Now, we model our problem as follows:
use the Kalman filter ro predict and estimate the next state of
y(i) = x + v(i), i = 0, 1, . . . , N 1 (4)
the object.
and we define the column vector 1 = [1, 1, . . . , 1]T . Then,
II. P ROBLEM F ORMULATION y = 1x + v (5)
A. Object Detection Using Background Subtraction if this is the case, according to Gauss-Markov theorem, the
A video is composed of a series of frames each which can optimal linear estimator (m.v.u.e) of x is:
be considered as a 2D signal. So, a video can be seen as a N 1 N 1
1 X 1 X
two-dimensional (2D) signal through time. Moreover, there are xmvue = y(i) = ui (6)
two types of objects in a video: steady and moving objects. N i=0 N i=0
Steady objects do not change from frame to frame and can be Namely, (6) means that the optimal linear estimator of x given
considered as the background scene. The goal is to detect a {y(i)}, is simply the mean of the samples(measurements).
moving objects from the steady ones. First let us provide an So, in order to obtain an estimation of the background of
example: consider that you are looking at a wall and suddenly the video, we take the average of all frames and store it as
a birdy fly over this wall. The steady object in this scene is the background scene. Fig. 1 illustrates 4 frames of a sample
the wall and the moving object is the birds. The bird is in fact video, these samples are in fact 4 noisy measurements of our
disturbing your observation of the wall(background), so in the signal, and that yellow ball which we are trying to track acts
context of signal processing, this bird(moving object) can be as the disturbance to the background of the video(the door
seen as a noise to that background. In other words, in a video, a and the wall). Fig. 2 provides the background scene that is
moving object is like a noise to the background scene which the result of averaging over all of the frames. Please note
is a fixed signal, and this moving object is adding noise to that in this project we are assuming that the background does
our observation of that background. Consequently, each frame not change, so sample-mean estimator is a good estimation of
of the video can be interpreted as a noisy observation of the the background. However, in the case of real-time tracking
background. Therefore, the problem is just simply the noise where the background is not changing, one can feed the
detection in a signal. The following model can be used for our average estimator as the frames arrives; evidently, in this case,
problem: estimation improves with time.
y =x+v (1) Now that we obtained x, we can extract the noise from the
signal(video) by subtracting each frame from the background.
Where y is our noisy measurement of x (background signal), Fig. 3 provides four realization of the noise(moving object) at
and v denotes the disturbance which in fact is our moving ob- different frames. Due to the fact that in our problem we do
ject disturbing that background signal x. As it was mentioned not care about the energy of the noise - the gray level of an
earlier, we need to extract the noise v from our noisy signal image(pixels) is proportional to the energy or the amount of
y (the video). Each frame of the video is a noisy realization information it contains (entropy of the image) - we can use
of the signal y and we refer to the i-th frame of the video one-bit Compressed Sensing method to store the noise. That
as ui . Further we assume that the video has N frames. Our is, we use the following model to store the noise :
approach of extracting the noise from the observations ui is
to first obtain an estimation of the background signal x, then zi = sgn(vi i ), i = 0, 1, . . . , N 1 (7)
 2

Where zi denotes the quantized measurement of vi with Background

respect to the threshold i . Namely, instead of saving the

real value of the measurement vi , we only save the sign of it
with respect to the defined threshold i at that measurement.
In practice, one may use an adaptive threshold but in our
application we use a fixed threshold at all of the measurements.
Also, it is worth to mention that in the 1-bit compressed
sensing model (7) we lose all of the information about the
magnitude of vi but as we mentioned earlier we do not care
about the energy of the noise, so this model can be used to
store the measurements and improve the speed of the tracking
process and also it lowers the dimension of the calculations(an
image typically has the intensity information of the colors:
Red, Green, and Blue and after quantizing the measurements
based on (7) we only has one dimension which is known as
binary image). Fig. 2. An estimation of the background signal resulted from averaging over
Fig. 4, illustrates one estimation of the noise after quantization. all frames.
It can be observed that we have an anomaly at the bottom right
corner of the Fig. 4, which emphesized the fact that this is not
Frame No. : 3 Frame No. : 15
an exact realization of the noise(moving object) but instead it
is an estimation of the noise and is prone to some errors.
Now that we estimated the moving object, we can easily find
the center of the object by inspecting the binary image of each
frame and find the area that contains more 1 and choose the
larger area as the object. Then we can estimate the center of
the area, and store it as the position of the moving object at
each frame. Eventually, in Fig. 5, you can see that we detected Frame No. : 45 Frame No. : 80

the moving object at each frame. The yellow circle denotes our
detection.

Frame No. : 3 Frame No. : 15

Fig. 3. Four realizations of the noise (moving object) at different frames.

Frame No. : 45 Frame No. : 80 Frame No. : 15

Fig. 1. Four sample frames of the video. In other words, these are four noisy
measurements of the background (which is a 2D signal).

B. Kalman Filter
In this section we describe the formulation and system
Fig. 4. One realization of the noise after quantization.
model for Kalman filter.
Intutitively, Kalman filter takes the current state of your
system, and makes a prediction based on the current state
and current uncertainty of our measurements, and make a Then, it compares its prediction with the received input and
prediction for the next state of the system with anuncertainty. correct it self upon the error.
 3

Frame No. : 15 So, the following model can be used to define the state
transition:
1 2
xt 1 0 T 0 xt1 2T
yt 0 1 0 T yt1 1 T 2
=
xt 0 0 1 0 xt1 + T .a+ Wt1 (11)
2

yt 0 0 0 1 yt1 T
We can formulate (11) as follows:
Frame No. : 45
Xt = AXt1 + But1 (12)
Where But1 can be seen as the noise(or external force on
the acceleration).
In this project, we are observing the position of the moving
object. Therefore, we define the following measurement matrix
H:  
1 0 0 0
H= (13)
0 1 0 0
Fig. 5. Four sample frames of the video. In other words, these are four noisy The measurement matrix is:
measurements of the background (which is a 2D signal).
    xt
xt 1 0 0 0
yt + Vt

= (14)
yt 0 1 0 0 xt
First we need to define our state for the Kalman filter. We yt
want to predict the position of a moving object based on the
Where V = [N (0, 12 ), N (0, 22 )]T is the measurement noise.
current information of the object. For simplicity we assume
Basically, Kalman filter has three noise covariance matrices:
a constant velocity model for our problem. The dynamics of
a moving object in one-dimension can be described as follows:
Dynamic Noise: During transition from one state to
1 2 another, the system can be disturbed by an external force
xt = aT + vt1 T + xt1 (8) and add noise to the system. An external force can be
2
modeled as a disturbance to the object acceleration in
vt = aT + vt1 (9) our problem. It contributes to the prediction of the next
error covarinace matrix.
Where xt and vt denotes the position and velocity at time t, Measurement Noise: All of our sensors are prone to
and a denotes the acceleration. So, the dynamics of a moving noise and consequently will lead to a corruption of
object in one-dimension can be modeled by the position our measurements. We refer to this disturbance as the
and the first derivation of it. Without losing the generality, Measurement Noise.
we can extend the one-dimensional case to a 2D object and Covariance of State Variables
conclude that the dynamics of a two-dimensional object can Assuming that the state variables are independent, we initialize
be described by x, y, x, and y. the covariance matrix of state variables as follows. Please
We define the state Xt with the following variables of interest: note that we can also consider this matrix as posteriori error
covariance matrix.
2
x 0 0 0
xt 0 y2 0
yt 0
St = 2
(15)
Xt =
xt
(10) 0 0 x 0
0 0 0 y2
yt
Also, we further assume that the measurement noises are
Next, we need to see what is the expected behaviour of our independent, then the covariance matrix of V can be described
variables when we are going from one state to another. Based as:  2 
on Eq. (8) and (9), we define the following behaviour for the 1 0
cov(V) = R = (16)
system variables: 0 22
1 Finally we need to define the covariance matrix of dynamic
xt = xt1 + xt1 T + aT 2 noise. As it was described earlier, this noise represents the
2
1 2 disturbance during transition from one state to another. It can
yt = yt1 + yt1 T + aT be written as:
2 2
xt = xt1 T + aT x 0 xx 0
yt = yt1 T + aT
0 y2 0 yy
Q= (17)
xx 0 x2 0
0 yy 0 y2
 4

From (11), we can define Q as: Frame No. : 3 Frame No. : 15

1 4 1 3

4T 0 2T 0
0 1 4 1 3
T 0 2T
Q= 1T3
4 (18)
2 0 T2 0
1 3
0 2 T 0 T2
We assume that our original tracker (section II.a) is used as
the input to the Kalman filter. We define the input vector as:
  Frame No. : 45 Frame No. : 80
x
Yt = t (19)
yt
We defined all of the required matrices for Kalman filter. Now
we can use the Kalman filter based on the following algorithm
to predict the position of the moving object based on our
original tracker (section II.a) as the input to the filter. Kalman
filter, has two stages: prediction and correction :
P rediction : Fig. 6. Scenario 1 in which the Kalman filter tracks the moving object when
Xt = AXt1 + Bu it is feeded every three samples.

St = ASt1 AT + Q
Correction : Frame No. : 17 Frame No. : 21

Kt1 = St1 H T (HSt1 H T + R)1

Xt+1 = Xt + Kt1 (Yt HXt )
St+1 = (I Kt H)St

III. R ESULTS
In order to observe the behaviour of Kalman filter under
Frame No. : 36 Frame No. : 100
different circumstances, we considered three different cases to
examine the Kalman filter in object tracking. In the following
subsections, we examine each of these cases.

A. Scenario 1: Prediction
The first scenario is the case that we are sensing the position
of the object every 3 frames and we want to have a good
prediction of the position of moving object based on these
samples. Fig. 6, illustrates the result in four different frames. Fig. 7. Scenario 2 in which the Kalman filter tracks the moving object in the
The yellow circle is our main tracker(which is used as the presence of a large noise.
input to the Kalman filter every 3 frames) and the black circle
is the prediction of Kalman filter. It can be observed that the
Kalman filter is tracking the moving object with a very good C. Scenario 3: Blind Prediction
accuracy.
In this case, we let the Kalman filter to learn for half of the
B. Scenario 2: Prediction In The Presence of Noise frames and then we did not update the input for the filter. In
(10) we defined the dynamic of the system for the constant
In this scenario, we add a large noise to the input of the velocity object. That is, we are not capturing the acceleration
Kalman filter. It turns out that the Kalman filter is more of the system. So, we should expect that the Kalman filter can
robust to the noise than the original tracker. So, if we have not track the trajectory of the ball because the object is under
our measurements aren corrupted by noise, one can use the the gravity and has a negative vertical acceleration. If we want
Kalman filter to obtain a better estimation than each of the to track the trajectory of the without the input, we must use a
sensors (data fusion) because this algorithm is an adaptive more complex system model as follows:
filter and is more robust to the noise than each of the sensors.
Fig. 7, illustrates this scenario. It can be seen that, the yellow

x
circle is jumping around and is far from the object. However, y
the Kalman filter has a better estimation of the position. Please

x
note that, a low gain will smooth out the noise but also lowers X= y
(20)
the speed of Kalman filter (it will detect the changes more

x
slowly). y
 5

Fig. 8 provides the result of this scenario. As you can see,

Kalman filter is not able to track the moving object after
cutting the input and it tracks a linear path after that.

Frame No. : 68 Frame No. : 72

Frame No. : 81 Frame No. : 87

Fig. 8. Scenario 3 in which the Kalman filter blindly track the moving object.

IV. C ONCLUSION
In this project we designed a Kalman filter to track a moving
object in a video. In fact, as it was mentioned earlier, a moving
object in a video can be seen as a noise to the background
scene. So, this project was simply a noise detection based on
Kalman filter. The same approach can be used to estimate and
cancel out the noise of other signals. As we saw in the scenario
1 and 2, Kalman filter can be used whenever we need to predict
the next state of a system based on some noisy measurements.
Also, it can be used for sensor fusion as well. It must be
mention that this algorithm is defined for linear systems(we
used linear algebra). In the case if nonlinear systems, the
extended Kalman filter (EKF) which is a nonlinear version
of Kalman filter can be used.