Lecture 7: Tracking

Dr Christos Kyrkou
KIOS Research and Innovation Center of Excellence,
Electrical and Computer Engineering Department, University of Cyprus
[email protected]
 ECE 627

Today: Tracking Objects

❑Goal: Locating a moving object/part across video frames

❑Examples and Applications

❑Overview of probabilistic tracking
❑Kalman Filter
❑Particle Filter

Dr Christos Kyrkou Lecture 7: Tracking 2

 ECE 627

Applications
❑Motion capture
❑Augmented Reality
❑Action Recognition
❑Security, traffic monitoring
❑Video Compression
❑Video Summarization
❑Medical Screening

Dr Christos Kyrkou Lecture 7: Tracking 3

 ECE 627

Tracking Examples
❑Traffic: https://www.youtube.com/watch?v=DiZHQ4peqjg
❑Soccer: http://www.youtube.com/watch?v=ZqQIItFAnxg
❑Face: http://www.youtube.com/watch?v=i_bZNVmhJ2o
❑Body: https://www.youtube.com/watch?v=_Ahy0Gh69-M
❑Eye: http://www.youtube.com/watch?v=NCtYdUEMotg
❑Gaze: http://www.youtube.com/watch?v=-G6Rw5cU-1c

Dr Christos Kyrkou Lecture 7: Tracking 4

 ECE 627

Things that make visual tracking difficult

❑Small, few visual features
❑Erratic movements, moving very quickly
❑Occlusions, leaving and coming back
❑Surrounding similar-looking objects

Dr Christos Kyrkou Lecture 7: Tracking 5

 ECE 627

Strategies for tracking

❑Tracking by repeated detection
▪ Works well if object is easily detectable (e.g., face or colored glove)
and there is only one
▪ Need some way to link up detections
▪ Best you can do, if you can’t predict motion

Dr Christos Kyrkou Lecture 7: Tracking 6

 ECE 627

Tracking with dynamics

❑Key idea: Based on a model of expected motion, predict where
objects will occur in next frame, before even seeing the image
▪ Restrict search for the object
▪ Measurement noise is reduced by trajectory smoothness
▪ Robustness to missing or weak observations

Dr Christos Kyrkou Lecture 7: Tracking 7

 ECE 627

Strategies for tracking

❑Tracking with motion prediction
▪ Predict the object’s state in the next frame
▪ Kalman filtering: next state can be linearly predicted from current
state (Gaussian)
▪ Particle filtering: sample multiple possible states of the object (non-
parametric, good for clutter)

Dr Christos Kyrkou Lecture 7: Tracking 8

 ECE 627

General model for tracking

❑State 𝑋: The actual state of the moving
object that we want to estimate
▪ State could be any combination of position,
pose, viewpoint, velocity, acceleration, etc.

❑Observations 𝑌: Our actual measurement

or observation of state 𝑋. Observation can
be very noisy

❑At each time 𝑡, the state changes to 𝑋𝑡 and

we get a new observation 𝑌𝑡
Dr Christos Kyrkou Lecture 7: Tracking 9
 ECE 627

Steps of tracking
❑Prediction: What is the next state of the object given past
measurements?
P(X t Y0 = y0 ,, Yt −1 = yt −1 )

Dr Christos Kyrkou Lecture 7: Tracking 10

 ECE 627

Steps of tracking
❑Prediction: What is the next state of the object given past
measurements?
P(X t Y0 = y0 ,, Yt −1 = yt −1 )

❑Correction: Compute an updated estimate of the state from

prediction and measurements
P(X t Y0 = y0 ,, Yt −1 = yt −1 , Yt = yt )

Dr Christos Kyrkou Lecture 7: Tracking 11

 ECE 627

Simplifying assumptions
❑Only the immediate past matters
P(X t X 0 ,, X t −1 ) = P(X t X t −1 )

dynamics model

Dr Christos Kyrkou Lecture 7: Tracking 12

 ECE 627

Simplifying assumptions
❑ Only the immediate past matters
P(X t X 0 ,, X t −1 ) = P(X t X t −1 )

dynamics model
❑Measurements depend only on the current state

P(Yt X 0 , Y0 , X t −1 , Yt −1 , X t ) = P(Yt X t )

❑ Markovian Process: observation model

▪ A Markov chain is a stochastic model describing a sequence of possible
events in which the probability of each event depends only on the state
attained in the previous event.

Dr Christos Kyrkou Lecture 7: Tracking 13

 ECE 627

Simplifying assumptions
❑Only the immediate past matters
P(X t X 0 ,, X t −1 ) = P(X t X t −1 )

dynamics model

❑Measurements depend only on the current state

P(Yt X 0 , Y0 , X t −1 , Yt −1 , X t ) = P(Yt X t )
observation model
X0 X1 X2 … Xt-1 Xt

Y0 Y1 Y2 Yt-1 Yt
Dr Christos Kyrkou Lecture 7: Tracking 14
 ECE 627

Problem statement
❑We have models for
▪ Likelihood of next state given current state: P ( X t X t −1 )
▪ Likelihood of observation given the state: P (Yt X t )

❑We want to recover, for each 𝑡: P(X t y0 ,, yt )

Dr Christos Kyrkou Lecture 7: Tracking 15

 ECE 627

Probabilistic tracking
❑Base case:
▪ Start with initial prior that predicts state in absence of any
evidence: 𝑃(𝑋0 )
▪ For the first frame, correct this given the first measurement: 𝑌0 = 𝑦0

Dr Christos Kyrkou Lecture 7: Tracking 16

 ECE 627

Probabilistic tracking
❑Base case:
▪ Start with initial prior that predicts state in absence of any
evidence: 𝑃(𝑋0 )
▪ For the first frame, correct this given the first measurement: 𝑌0 = 𝑦0
Likelihood Prior

P( y0 | X 0 ) P( X 0 )
P( X 0 | Y0 = y0 ) =  P( y0 | X 0 ) P( X 0 )
P( y0 )
Posterior
Normalizing Factor

Dr Christos Kyrkou Lecture 7: Tracking 17

 ECE 627

Probabilistic tracking
❑Base case:
▪ Start with initial prior that predicts state in absence of any
evidence: 𝑃(𝑋0 )
▪ For the first frame, correct this given the first measurement: 𝑌0 = 𝑦0
❑Given corrected estimate for frame t-1:
▪ Predict for frame t ➔ P ( X t y 0 ,, y t −1 )
▪ Observe 𝑦𝑡 ; Correct for frame 𝑡 ➔ P ( X t y 0 ,, y t −1 , y t )

predict correct

Dr Christos Kyrkou Lecture 7: Tracking 18

 ECE 627

Prediction
❑Prediction involves representing P(X t y0 ,, yt −1 )
given P(X t −1 y0 ,, yt −1 )

P(X t y0 ,, yt −1 )
=  P(X t , X t −1 y0 ,, yt −1 )dX t −1
=  P( X t | X t −1 , y0 ,, yt −1 )P( X t −1 | y0 ,, yt −1 )dX t −1
Law of total probability

=  P( X t | X t −1 )P( X t −1 | y0 ,, yt −1 )dX t −1

Dr Christos Kyrkou Lecture 7: Tracking 19

 ECE 627

Prediction
❑Prediction involves representing P(X t y0 ,, yt −1 )
given P(X t −1 y0 ,, yt −1 )

P(X t y0 ,, yt −1 )
=  P(X t , X t −1 y0 ,, yt −1 )dX t −1
=  P( X t | X t −1 , y0 ,, yt −1 )P( X t −1 | y0 ,, yt −1 )dX t −1
=  P(Conditioning
X t | X t −1 )P( Xont −1X|t–1
y0 ,, yt −1 )dX t −1

Dr Christos Kyrkou Lecture 7: Tracking 20

 ECE 627

Prediction
❑Prediction involves representing P(X t y0 ,, yt −1 )
given P(X t −1 y0 ,, yt −1 )

P(X t y0 ,, yt −1 )
=  P(X t , X t −1 y0 ,, yt −1 )dX t −1
=  P( X t | X t −1 , y0 ,, yt −1 )P( X t −1 | y0 ,, yt −1 )dX t −1
=  P( X t | X t −1 )P( X t −1 | y0 ,, yt −1 )dX t −1
Independence assumption
Dr Christos Kyrkou Lecture 7: Tracking 21
 ECE 627

Prediction
❑Prediction involves representing P(X t y0 ,, yt −1 )
given P(X t −1 y0 ,, yt −1 )

P(X t y0 ,, yt −1 )
=  P(X t , X t −1 y0 ,, yt −1 )dX t −1
=  P( X t | X t −1 , y0 ,, yt −1 )P( X t −1 | y0 ,, yt −1 )dX t −1
=  P( X t | X t −1 )P( X t −1 | y0 ,, yt −1 )dX t −1
dynamics corrected estimate
model from previous step
Dr Christos Kyrkou Lecture 7: Tracking 22
 ECE 627

Correction
❑Correction involves computing P(X t y0 ,, yt )
given predicted value P(X t y0 ,, yt −1 )

Dr Christos Kyrkou Lecture 7: Tracking 23

 ECE 627

Correction
❑Correction involves computing P(X t y0 ,, yt )
given predicted value P(X t y0 ,, yt −1 )
P(X t y0 ,, yt )
P ( y t | X t , y 0 ,, y t −1 )
= P ( X t | y 0 ,, y t −1 )
P ( y t | y 0 ,, y t −1 )
P ( y t | X t )PBayes’
( X t | yRule
0 ,, y t −1 )
=
P ( y t | y 0 ,, y t −1 )
P ( y t | X t )P ( X t | y 0 ,, y t −1 )
=
 P( X t | y 0 ,, y t −1 )P ( y t | X t )dX t
Dr Christos Kyrkou Lecture 7: Tracking 24
 ECE 627

Correction
❑Correction involves computing P(X t y0 ,, yt )
given predicted value P(X t y0 ,, yt −1 )
P(X t y0 ,, yt )
P( yt | X t , y0 ,, yt −1 )
= P( X t | y0 ,, yt −1 )
P( yt | y0 ,, yt −1 )
P( yt | X t )P( X t | y0 ,, yt −1 )
=
P( yt | y0 ,, yt −1 )
( yt | X t )P( X t assumption
PIndependence | y0 ,, yt −1 )
=
P( X yt |directly

(observation
t 0 )P( y only
y ,, ydepends | X )on
t −1 dX state Xt)
t t t

Dr Christos Kyrkou Lecture 7: Tracking 25

 ECE 627

Correction
❑Correction involves computing P(X t y0 ,, yt )
given predicted value P(X t y0 ,, yt −1 )
P(X t y0 ,, yt )
P ( y t | X t , y 0 ,, y t −1 )
= P ( X t | y 0 ,, y t −1 )
P ( y t | y 0 ,, y t −1 )
P ( y t | X t )P ( X t | y 0 ,, y t −1 )
=
P ( y t | y 0 ,, y t −1 )
P ( y t | X t )P ( X t | y 0 ,, y t −1 )
=
 P( y t | X t )P ( X t | y 0 ,, y t −1 )dX t
Conditioning on Xt
Dr Christos Kyrkou Lecture 7: Tracking 26
 ECE 627

Correction
❑Correction involves computing P(X t y0 ,, yt )
given predicted value P(X t y0 ,, yt −1 )
P(X t y0 ,, yt )
P ( y t | X t , y 0 ,, y t −1 )
= P ( X t | y 0 ,, y t −1 )
P ( y t | y 0 ,, y t −1 )
P ( y t | X t )P ( X t | y 0 ,, y t −1 )
=
P ( y t | y 0 ,, y t −1 )
observation predicted
model P ( y t | X t )P ( X t | y 0 ,, y t −1 ) estimate
=
 P( y t | X t )P ( X t | y 0 ,, y t −1 )dX t
normalization factor
Dr Christos Kyrkou Lecture 7: Tracking 27
 ECE 627

Summary: Prediction and correction

❑Prediction:
P( X t | y0 ,, yt −1 ) =  P( X t | X t −1 )P( X t −1 | y0 ,, yt −1 )dX t −1

dynamics corrected estimate

model from previous step

❑Correction:
observation predicted
model estimate

P( yt | X t )P( X t | y0 ,, yt −1 )
P( X t | y0 ,, yt ) =
 P( y t | X t )P( X t | y0 ,, yt −1 )dX t
Dr Christos Kyrkou Lecture 7: Tracking 28
 ECE 627

The Kalman filter

❑Linear dynamics model: state undergoes linear transformation
plus Gaussian noise

❑Observation model: measurement is linearly transformed state

plus Gaussian noise

❑The predicted/corrected state distributions are Gaussian

▪ You only need to maintain the mean and covariance
▪ The calculations are easy (all the integrals can be done in closed
form)

Dr Christos Kyrkou Lecture 7: Tracking 29

 ECE 627

Example: Kalman Filter

Ground Truth

Observation

Correction

Next Frame Update Location,

Prediction Velocity, etc.
Dr Christos Kyrkou Lecture 7: Tracking 30
 ECE 627

Comparison

Ground Truth Observation Correction

Dr Christos Kyrkou Lecture 7: Tracking 31

 ECE 627

Propagation of Gaussian densities

Current state
Expected change

Decent model if there is just one object,

but localization is imprecise

Uncertainty

Observation
and Correction
Dr Christos Kyrkou Lecture 7: Tracking 32
 ECE 627

Particle filtering
❑Represent the state distribution non-parametrically
❑Prediction: Sample possible values 𝑋𝑡−1 for the previous state
❑Correction: Compute likelihood of 𝑋𝑡 based on weighted samples
and 𝑃(𝑦𝑡 |𝑋𝑡 )

M. Isard and A. Blake, CONDENSATION -- conditional density propagation for

visual tracking, IJCV 29(1):5-28, 1998
Dr Christos Kyrkou Lecture 7: Tracking 33
 ECE 627

Particle filtering
Start with weighted
samples from previous
time step
Sample and shift
according to dynamics
model
Spread due to
randomness; this is
predicted density P(Xt|Yt-1)
Weight the samples
according to observation
density
Arrive at corrected density
estimate P(Xt|Yt)

M. Isard and A. Blake, CONDENSATION -- conditional density propagation for

visual tracking, IJCV 29(1):5-28, 1998
Dr Christos Kyrkou Lecture 7: Tracking 34
 ECE 627

Propagation of non-parametric densities

Current state
Expected change

Good if there are multiple, confusable

objects (or clutter) in the scene

Uncertainty

Observation
and Correction
Dr Christos Kyrkou Lecture 7: Tracking 35
 ECE 627

Particle filtering results

❑People: http://www.youtube.com/watch?v=wCMk-pHzScE
❑Hand: http://www.youtube.com/watch?v=tIjufInUqZM

Localization (similar model): https://www.youtube.com/watch?v=rAuTgsiM2-8

http://www.cvlibs.net/publications/Brubaker2013CVPR.pdf

Good informal explanation: https://www.youtube.com/watch?v=aUkBa1zMKv4

Dr Christos Kyrkou Lecture 7: Tracking 36
 ECE 627

Particle vs. Kalman

❑In certain situations particle filters scale exponentially, and it
would be a mistake to represent particle filters over anything
more than four dimensions.

❑Particle Filters:
▪ Non-linear dynamics and non-gaussian noise

❑Kalman Filter
▪ Linear dynamics and gaussian noise

Dr Christos Kyrkou Lecture 7: Tracking 37

 ECE 627

Tracking issues
❑Initialization
▪ Manual
▪ Background subtraction
▪ Detection

❑Getting observation and dynamics models

▪ Observation model: match a template or use a trained detector
▪ Dynamics model: usually specify using domain knowledge

Dr Christos Kyrkou Lecture 7: Tracking 38

 ECE 627

Tracking issues
❑Initialization
❑Getting observation and dynamics model
❑Uncertainty of prediction vs. correction
▪ If the dynamics model is too strong, will end up ignoring the data
▪ If the observation model is too strong, tracking is reduced to
repeated detection

Too strong dynamics model Too strong observation model

Dr Christos Kyrkou Lecture 7: Tracking 39
 ECE 627

Tracking issues
❑Initialization
❑Getting observation and dynamics models
❑Prediction vs. correction
❑Data association
▪ When tracking multiple objects, need to
assign right objects to right tracks (particle
filters good for this)

Dr Christos Kyrkou Lecture 7: Tracking 40

 ECE 627

Tracking issues
❑Initialization
❑Getting observation and dynamics models
❑Prediction vs. correction
❑Data association
❑Drift
▪ Errors can accumulate over time

Dr Christos Kyrkou Lecture 7: Tracking 41

 ECE 627

Drift

D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their

Appearance. PAMI 2007.
Dr Christos Kyrkou Lecture 7: Tracking 42
 ECE 627

Things to remember
❑Tracking objects = detection + prediction

❑Probabilistic framework
▪ Predict next state
▪ Update current state based on observation

❑Two simple but effective methods

▪ Kalman filters: Gaussian distribution
▪ Particle filters: multimodal distribution

Dr Christos Kyrkou Lecture 7: Tracking 43