Scale-space image processing

◼ Corresponding image features can appear at different scales

◼ Like shift-invariance, scale-invariance of image processing algorithms

is often desirable.
◼ Scale-space representation is useful to process an image in a manner
that is both shift-invariant and scale-invariant

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 1
 Scale-space image processing

◼ Scale-space theory
◼ Laplacian of Gaussian (LoG) and Difference of Gaussian (DoG)
◼ Scale-space edge detection
◼ Scale-space keypoint detection
⚫ Harris-Laplacian
⚫ SIFT detector
⚫ SURF detector

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 2
 Scale-space representation of a signal
Parametric family of signals f t (x) where fine-scale information is successively attenuated

f t (x) scale t

f (x )= f 0 (x )

Successive smoothing Zero-crossings of 2nd derivative f t (x)

with a Gaussian filter Fewer edges at coarser scales

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 3
 Scale-space representation of images
◼ Parametric family of images smoothed by Gaussian filter

 x2 + y2  Coarser
f t (x, y )= g t (x, y )* f (x, y ); t  0 with g (x, y )=
1 t
t
exp − scales
2 t  2t 

( ) ( )(
F t  x , y = G t  x , y F  x , y ) with G ( , )= exp − 2t (
t
x y
2
x )
2 
+y 
 Original
image f (x,y)
◼ Shift-invariance

f t (x − x, y − y )= g t (x, y )* f (x − x, y − y )

◼ Rotation-invariance

( ) ( ) (
f t xcos − y sin , x sin + y cos = gt x, y * f xcos − y sin , x sin + y cos )

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 4
 Scale-space representation of images (cont.)

◼ Commutative semigroup property

( ) ( ) ( ) ( )
f t1+t 2 x, y = gt1 x, y * f t 2 x, y = g t 2 x, y * f t1 x, y ( )
= g t1 (x, y )* g (x, y )* f (x, y )
t2

◼ Separability
2 2
 +   
g t (x, y )=
2 2
1 x y 1 x 1 y
exp  −  = exp  −   exp  − 
2 t  2t  2 t  2t  2 t  2t 

( )  t
 2
( 

)  t 
 2 
 t 
G t  x , y = exp  −  x2 +  2y  = exp  −  x2  exp  −  2y 
 2 

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 5
 Scale-space representation of images (cont.)

◼ Non-creation of local extrema (for f (x,y) and all of its partial derivatives)
( )
since gt x, y  0 and unimodal.
◼ Solution to diffusion equation (heat equation)

f t (x, y )= 2 f t (x, y )
1
t 2
 t
t
( 
) ( )(
F  x ,  y = G t  x , y F  x ,  y
t
)
  t 2
(
2 
) (
= exp  −  x +  y  F  x , y
t  2 
)
=− (
1 2
2
)  t
 2
( ) (
2 
 x +  2y exp  −  x2 +  y  F  x , y

)
=− (
1 2
2
) (
 x +  2y F t  x , y )
Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 6

( ) 1
( )
f t x, y = 2 f t x, y
t 2
 LoG vs. DoG
Laplacian of Gaussian Difference of Gaussians
t = σ2 = 1  t t = σ2 = 1, k = 1.1
1 2 t
2
 f x, y =( )
t
f x, y ( )

0.05 0.05
0 0
-0.05 -0.05
-0.1 -0.1
-0.15 -0.15
-0.2 -0.2
-0.25 -0.25

4 4
2 4 2 4
0 2 0 2
0 0
-2 -2 -2 -2
Y -4 -4 X Y -4 -4 X
− x 2 + y2
1 
LoG (x, y ) = − 2  1 −
t 
x 2 + y2 
2t 
e 2t DoG (x, y ) =
1
(k − 1)t (
g k t (x, y) − g t (x, y)
2
)
Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 8
 LoG vs. DoG (cont.)
Laplacian of Gaussian Difference of Gaussians
t = σ2 = 1 t = σ2 = 1, k = 1.1

0.8 0.8

0.6 0.6

0.4 0.4
|H|

|H|
0.2 0.2

0 0

2 2
2 2
0 0
0 0
-2 -2 -2 -2
y x y x

( ) (
H  x , y = −  +  G  x , y
2
x
2
y ) (
t
) (
H  x , y = ) 1  k 2t
(k − 1)t 
( ) ( )
G  x ,  y − G t  x ,  y 

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 9
 Scale space: Laplacian images

f t (x, y )

t 2 f t (x, y )

t=1 t=4 t = 16 t = 64

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 10
 Scale space: binarized Laplacian images

( )
f t x, y

sign t 2 f t (x, y )



t=1 t=4 t = 16 t = 64

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 11
 Scale space: edge detection
Zero crossings of Laplacian images

t=1 t=4 t = 16 t = 64

Low-gradient-magnitude edges removed

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 12
Laplacian zero-crossings
 Keypoint detection with automatic scale selection
◼ Scale-space representation provides all scales;
which scale is best for keypoint detection?
Harris
◼ Harris-Laplacian scale t
1. Detect Harris corners at some initial scale
2. For each Harris corner xh , yh Harris
detect characteristic scale
th = argmax t  2 f t (xh , yh ) y Harris
t
3. Apply Harris detector in a spatial neighborhood
at scale th to refine keypoint location xh , yh x
4. Repeat 2. and 3. until convergence

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 14
 Keypoint detection with automatic scale selection
Harris-Laplacian example (150 strongest peaks)

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 15
 Keypoint detection with automatic scale selection
Harris-Laplacian example (200 strongest peaks)

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 16
 SIFT keypoint detection
…
◼ SIFT - Scale-Invariant Feature Transform
t = 16
◼ Decompose image into DoG scale-space representation
t=8 2
Scale ◼ Detect minima and maxima locally and across scales
t=8
(next ◼ Fit 3-d quadratic function to localize extrema with sub-
octave) t=4 2 pixel/sub-scale accuracy [Brown, Lowe, 2002]
t=4 ◼ Eliminate edge responses based on Hessian

t=4

t=2 2

Scale t=2
(first Scale
octave) t= 2

t =1
Difference of
[Lowe, 1999, 2004]
Gaussian Gaussian (DoG)
Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 17
 SIFT scale space pyramid: octave 1

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 18
 SIFT scale space pyramid: octave 2

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 19
 SIFT scale space pyramid: octave 3

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 20
 SIFT scale space pyramid: octave 4

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 21
 SIFT scale space pyramid: octave 5

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 22
 SIFT keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 23
 SIFT keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 24
 Robustness against scaling

[Mikolajczyk, Schmid, 2001]

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 25
 Hessian keypoints in scale space

 2 2 

 x
f t
( x, y ) x 
f t
(x, y ) 

Ht (x, y )= 
2
y
− g(x, y) 
 
f (x, y ) f t (x, y )
2 2
 t 
 x  y y 2


 
g(x, y) g(x, y)
x y

2 2
g(x, y) g(x, y)
x2  y2

2
g(x, y)
x  y

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 26
 SURF keypoint detection
◼ SURF – Speeded Up Robust Features [Bay, Tuytelaars, Van Gool, ECCV 2006]
◼ No subsampling – all resolution levels at full spatial resolution
◼ Simple approximation of scale space Gaussian derivatives using integral images

Dtyy Dtxy
◼ Determinant of Hessian

( )
det H  D D − 0.9D
t t
xx
t
yy ( t
xy )
2

◼ Non-maximum suppression in 3x3x3 [x,y,t] neighborhood

◼ Interpolation of maximum of det(H) in image space x,y and scale t

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 27
 SURF keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 28
 SIFT keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 29
 SURF keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 30
 SIFT keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Scale Space 31