Object detection
CV3DST | Prof. Leal-Taixé 1
Task definition
• Object detection problem
Bounding box. (x,y,w,h)
(x,y)
CV3DST | Prof. Leal-Taixé 2
Task definition
• Object detection problem
Bounding box. (x,y,w,h)
+
class
CV3DST | Prof. Leal-Taixé 3
A bit of history
CV3DST | Prof. Leal-Taixé 4
Traditional object detection methods
• 1. Template matching + sliding window
Template
Image
CV3DST | Prof. Leal-Taixé 5
Traditional object detection methods
• 1. Template matching + sliding window
Image
CV3DST | Prof. Leal-Taixé 6
Traditional object detection methods
• 1. Template matching + sliding window
For every position
you evaluate how
much do the pixels
in the image and
LOW template correlate
correlation
Image
CV3DST | Prof. Leal-Taixé 7
Traditional object detection methods
• 1. Template matching + sliding window
For every position
you evaluate how
much do the pixels
in the image and
template correlate
Image HIGH
CV3DST | Prof. Leal-Taixé
correlation 8
Traditional object detection methods
• Problems of 1. Template matching + sliding window
For every position
you evaluate how
much do the pixels
in the image and
template correlate
Image LOW
CV3DST | Prof. Leal-Taixé
correlation 9
Traditional object detection methods
• Problems of 1. Template matching + sliding window
– Occlusions: we need to see the WHOLE object
– This works to detect a given instance of an object but not
a class of objects
Appearance and
shape changes
Pose changes
CV3DST | Prof. Leal-Taixé 10
Traditional object detection methods
• Problems of 1. Template matching + sliding window
– Occlusions: we need to see the WHOLE object
– This works to detect a given instance of an object but not
a class of objects
– Objects have an unknown position, scale and aspect
ratio, the search space is searched inefficiently with
sliding window
CV3DST | Prof. Leal-Taixé 11
Traditional object detection methods
• 2. Feature extraction + classification
CV3DST | Prof. Leal-Taixé 12
Viola-Jones detector
• 2. Feature extraction + classification
– Learning multiple weak learners to build a strong
classifier
– That is, make many small decisions and combine them
for a stronger final decision
Viola and Jones. Rapid object detection using a boosted cascade of simple features. CVPR 2001.
CV3DST | Prof. Leal-Taixé 13
Viola-Jones detector
• 2. Feature extraction + classification
Haar features
Viola and Jones. Rapid object detection using a boosted cascade of simple features. CVPR 2001.
CV3DST | Prof. Leal-Taixé 14
Viola-Jones detector
• 2. Feature extraction + classification
– Step 1: Select your Haar-like features
– Step 2: Integral image for fast feature evaluation
• I can evaluate which parts of the image have highest cross-
correlation with my feature (template)
– Step 3: AdaBoost for to find weak learner
• I cannot possibly evaluate all features at test time for all
image locations
• Learn the best set of weak learners
• Our final classifier is the linear combination of all weak
learners
Viola and Jones. Rapid object detection using a boosted cascade of simple features. CVPR 2001.
CV3DST | Prof. Leal-Taixé 15
Viola-Jones detector
Viola and Jones. Rapid object detection using a boosted cascade of simple features. CVPR 2001.
CV3DST | Prof. Leal-Taixé 16
Histogram of Oriented Gradients
• 2. Feature extraction + classification
Gradient: blue arrows show the
gradient, i.e., the direction of
greatest change of the image.
Average gradient image over training samples à gradients provide
shape information. Let us create a descriptor that exploits that.
CV3DST | Prof. Leal-Taixé Dalal and Triggs. Histogram of oriented gradients for human detection. CVPR 2005. 17
Histogram of Oriented Gradients
• 2. Feature extraction + classification
HOG descriptor à Histogram of oriented gradients.
Compute gradients in dense grids, compute gradients and create a
histogram based on gradient direction.
CV3DST | Prof. Leal-Taixé Dalal and Triggs. Histogram of oriented gradients for human detection. CVPR 2005. 18
Histogram of Oriented Gradients
• 2. Feature extraction + classification
– Step 1: Choose your training set of images that contain
the object you want to detect.
– Step 2: Choose a set of images that do NOT contain that
object.
– Step 3: Extract HOG features on both sets.
– Step 4: Train an SVM classifier on the two sets to detect
whether a feature vector represents the object of interest
or not (0/1 classification).
CV3DST | Prof. Leal-Taixé Dalal and Triggs. Histogram of oriented gradients for human detection. CVPR 2005. 19
Histogram of Oriented Gradients
• 2. Feature extraction + classification
HOG features weighted by the positive SVM weights – the ones
used for the pedestrian object classifier.
CV3DST | Prof. Leal-Taixé Dalal and Triggs. Histogram of oriented gradients for human detection. CVPR 2005. 20
Deformable Part Model
• Also based on HOG features, but based on body part
detection à more robust to different body poses
CV3DST | Prof. Leal-Taixé Felzenszwalb et al. A discriminatively trained, multiscale, deformable part model. CVPR 2008. 21
How to move
towards general
object detection?
CV3DST | Prof. Leal-Taixé 22
What defines an object?
• We need a generic, class-agnostic objectness
measure: how likely it is for an image region to
contain an object
Very likely to be
an object
Maybe it is an
object
CV3DST | Prof. Leal-Taixé 23
What defines an object?
• We need a generic, class-agnostic objectness
measure: how likely it is for an image region to
contain an object
• Using this measure yields a number of candidate
object proposals or regions of interest (RoI) where to
focus.
+ classifier
CV3DST | Prof. Leal-Taixé 24
Object proposal methods
• Selective search: van de Sande et al. Segmentation
as selective search for object recognition. ICCV 2011.
• Edge boxes: Zitnick and Dollar. Edge boxes: locating
object proposals from edges. ECCV 2014.
CV3DST | Prof. Leal-Taixé 25
Do we want all proposals?
• Many boxes trying to explain one object
• We need a method to keep only the “best” boxes
CV3DST | Prof. Leal-Taixé 26
Non-Maximum Suppression (NMS)
• Many boxes trying to explain one object
• We need a method to keep only the “best” boxes
CV3DST | Prof. Leal-Taixé 27
Non-Maximum Suppression (NMS)
Start with anchor box i
For another box j
If they overlap
Discard box i if the
score is lower than
the score of j
Overlap = to be defined Score = depends on the task
CV3DST | Prof. Leal-Taixé 28
Region overlap
• We measure region overlap with the Intersection
over Union (IoU) or Jaccard Index:
|A \ B|
J(A, B) =
|A [ B|
B B B
A A A
Intersection Union
CV3DST | Prof. Leal-Taixé 29
Non-Maximum Suppression (NMS)
Start with anchor box i
For another box j
If they overlap
Discard box i if the
score is lower than
the score of j
Overlap = to be defined Score = depends on the task
CV3DST | Prof. Leal-Taixé 30
NMS: the problem
Ground truth positions
CV3DST | Prof. Leal-Taixé Hosang, Benenson and Schiele. A Convnet for Non-Maximum Suppression. 2015 31
NMS: the problem
• Choosing a narrow threshold
Ground truth positions
False positives
Low Precision
CV3DST | Prof. Leal-Taixé Hosang, Benenson and Schiele. A Convnet for Non-Maximum Suppression. 2015 32
NMS: the problem
• Choosing a wider threshold
Ground truth position
False negative
False positive
Low Recall
CV3DST | Prof. Leal-Taixé Hosang, Benenson and Schiele. A Convnet for Non-Maximum Suppression. 2015 33
Non-Maximum Suppression (NMS)
• NMS will be used at test time. Most detection
methods (even Deep Learning ones) use NMS!
CV3DST | Prof. Leal-Taixé 34
Learning-based
detectors
CV3DST | Prof. Leal-Taixé 35
Types of object detectors
• One-stage detectors
Class score (cat,
Classification
dog, person)
Feature
Image
extraction Bounding box
Localization
(x,y,w,h)
• Two-stage detectors
Class score (cat,
Classification
Extraction of dog, person)
Feature
Image object
extraction
proposals Localization Refine bounding box
(Δx, Δy, Δw, Δh)
CV3DST | Prof. Leal-Taixé 36
Types of object detectors
• One-stage detectors
– YOLO, SSD, RetinaNet
– CenterNet, CornerNet, ExtremeNet
• Two-stage detectors
– R-CNN, Fast R-CNN, Faster R-CNN
– SPP-Net, R-FCN, FPN
CV3DST | Prof. Leal-Taixé 37
Object detection
CV3DST | Prof. Leal-Taixé 38
