Multimedia Retrieval

Multimedia Retrieval
Ch 5 Image Processing
Ch 5 Image Processing
Anne Ylinen
Anne Ylinen
Agenda
Agenda

Types of image processing
Types of image processing

Application areas
Application areas

Image analysis
Image analysis

Image features
Image features
Types of Image Processing
Types of Image Processing

Image Acquisition
Image Acquisition

Camera
Camera

Scanners
Scanners

X
X
-
-
ray imagers
ray imagers

Computer tomography (CT)
Computer tomography (CT)

Magnetic resonance scanners (MR)
Magnetic resonance scanners (MR)

Ultra sound devices (US)
Ultra sound devices (US)
Types of Image Processing
Types of Image Processing

Image Restoration
Image Restoration

Geometric distortions
Geometric distortions

Noise
Noise

Unsharpness
Unsharpness
Types of Image Processing
Types of Image Processing

Image Reconstruction
Image Reconstruction

using models
using models

different viewpoint
different viewpoint

another imaging device
another imaging device
Types of Image Processing
Types of Image Processing

Image Enhancement
Image Enhancement

Contrast enhancement
Contrast enhancement
amplitude scaling amplitude scaling
contrast modification contrast modification

Histogram normalization
Histogram normalization
nonadaptive nonadaptive histogram modification histogram modification
adaptive histogram modification adaptive histogram modification

Edge enhancement
Edge enhancement
linear edge linear edge crispening crispening
statistical differencing statistical differencing
Types of Image Processing
Types of Image Processing

Image Registration
Image Registration

Rigid registration
Rigid registration

Non
Non
-
-
rigid registration
rigid registration

Used in medical applications, cartography, face
Used in medical applications, cartography, face
recognition, etc.
recognition, etc.
Types of Image Processing
Types of Image Processing

Image Compression, Storage and Transmission
Image Compression, Storage and Transmission

Lossless
Lossless
image can be exactly reconstructed image can be exactly reconstructed

Lossy
Lossy
approximate reconstruction approximate reconstruction
Types
Types
of
of
Image Processing
Image Processing

Image Analysis
Image Analysis

Image analysis aims to generate a description of the
Image analysis aims to generate a description of the
image or of objects present in the image.
image or of objects present in the image.
Application Areas
Application Areas

Medical Imaging
Medical Imaging
MR, CT, US MR, CT, US

Geo Information Systems, Satellite, Aerial photography
Geo Information Systems, Satellite, Aerial photography
and Cartography
and Cartography

Biometry
Biometry
Face and fingerprint recognition, Face and fingerprint recognition, handpalm handpalm recognition, recognition,
tracking people tracking people
feature feature- -based and holistic approaches based and holistic approaches

Optical Character Recognition
Optical Character Recognition

Industrial Vision
Industrial Vision

Multimedia and Image Databases
Multimedia and Image Databases
Image Analysis
Image Analysis

extract information from an image
extract information from an image

detection
detection

classification
classification

parameter estimation
parameter estimation

structural analysis
structural analysis
Image Analysis
Image Analysis
comparison
feature
extraction
model
selection
observed
data
database of
models
model features
observed
features
match criterion
selected
model
Image Analysis
Image Analysis

Image analysis task
Image analysis task

the selection of the features
the selection of the features

the representation of the models
the representation of the models

the matching criterion
the matching criterion

the selection strategy
the selection strategy
Image Features
Image Features

Image
Image

2
2
-
-
dimensional signal
dimensional signal

represented by a matrix F of pixels of N rows and M
represented by a matrix F of pixels of N rows and M
columns
columns

A pixel value
A pixel value
f(n,m
f(n,m
) is an intensity or a vector of 3
) is an intensity or a vector of 3
RGB components
RGB components

mathematical operations are possible e.g. derivative
mathematical operations are possible e.g. derivative
and Fourier transformation
and Fourier transformation
Image Features
Image Features

Pixel Features
Pixel Features

Neighborhood and Image filtering
Neighborhood and Image filtering
each pixel an individual feature each pixel an individual feature
neighboring pixels grouped together neighboring pixels grouped together
used to obtain higher level features used to obtain higher level features
Image Features
Image Features

Scale space and derivatives
Scale space and derivatives
scale at which objects are seen in an image depends on the scale at which objects are seen in an image depends on the
distance between object and camera distance between object and camera
scale space theory for handling image structures at scale space theory for handling image structures at
different scale different scale
derivatives important for edge detection, point feature derivatives important for edge detection, point feature
detection, and so on detection, and so on
Image Features
Image Features

Texture
Texture
small elementary pattern repeated periodically or quasi small elementary pattern repeated periodically or quasi- -
periodically periodically
geometric or radiometric pattern geometric or radiometric pattern
important clues for segmenting the image important clues for segmenting the image
typified by typified by
the distance over which the patter is repeated the distance over which the patter is repeated
the direction in which the pattern is repeated the direction in which the pattern is repeated
the properties of the elementary pattern the properties of the elementary pattern
co co- -occurrence matrices occurrence matrices
Image Features
Image Features

Point Features
Point Features

Interest points
Interest points
corner points and spots corner points and spots
video tracking, stereo matching, object recognition video tracking, stereo matching, object recognition
Harris corner detector Harris corner detector
Image Features
Image Features
Harris corner detector Harris corner detector
image image I(x,y I(x,y) and sifted image ) and sifted image I(x+u I(x+u, , y+v y+v) )
Gaussian window function Gaussian window function w(x,y w(x,y) )
E(u,v E(u,v) should change fast for small sifts of ( ) should change fast for small sifts of (u,v u,v) )
[ ]

+ + =
y x
y x I v y u x I y x w v u E
,
2
) , ( ) , ( ) , ( ) , (
cuv bv au v u E 2 ) , (
2 2
+ +
Image Features
Image Features
2 1
2 1
2
2 1
,
2
2
det
) ( det
M of s eigenvalue ,
) , (
] , [ ) , (


+ =
=
=

traceM
M
traceM k M R
I I I
I I I
y x w M
where
v
u
M v u v u E
y x
y y x
y x x
Image Features
Image Features

2
Corner
Edge
Edge
Flat
R depends only on
eigenvalues of M
R is large for a corner
R is negative with large
magnitude for an edge
|R| is small for a flat
region
R > 0
R < 0
R < 0 |R| small
sourse(www.wisdo sourse(www.wisdom m.weiz .weizm mann.ac.il/~deniss/vision_sp ann.ac.il/~deniss/vision_spr ring04/files/Inva ing04/files/Invar riantFeatu iantFeatur res.ppt es.ppt )
Image Features
Image Features
Line elements Line elements
line segments have a width in the image equal to the scale of line segments have a width in the image equal to the scale of
the image, Gaussian like profile across the line the image, Gaussian like profile across the line
calculate the second derivative in the direction orthogonal to calculate the second derivative in the direction orthogonal to
the gradient vector the gradient vector
more stable result is obtained by approximating the more stable result is obtained by approximating the
neighborhood of each candidate line element by quadratic neighborhood of each candidate line element by quadratic
surface: surface:
( (n,m n,m) is the position of the candidate line element ) is the position of the candidate line element
ckl bl ak m n f l m k n f 2 ) , ( ) , (
2 2
+ + +
Image Features
Image Features
using Taylor expansion using Taylor expansion

1 1
, ,
2 2
are are eigenvalues eigenvalues of H of H
for true line element, one for true line element, one eigenvalue eigenvalue should be large and should be large and
the other small the other small
[ ]

+
yy xy
xy xx
f f
f f
H
where
l
k
H l k m n f l m k n f ) , ( ) , (
Image Features
Image Features

Edge elements
Edge elements
stepwise transition in intensities stepwise transition in intensities
neighboring edge elements linked to neighboring edge elements linked to gether gether form an edge form an edge
segment segment
gradient is large at the position of an edge gradient is large at the position of an edge
Gradient Gradient- -based methods based methods
Laplacian Laplacian- -based methods based methods
Canny Canny s s method method
Image Features
Image Features
Canny Canny s s method method
1. 1. Smooth the image with Gaussian filter Smooth the image with Gaussian filter
g(x,y g(x,y)= )=g g
c c
(x,y (x,y) )
* *f(x,y f(x,y) )
where where
where where represents the width of the Gaussian distribution represents the width of the Gaussian distribution
2. 2. Compute the second derivative in the gradient direction Compute the second derivative in the gradient direction
3. 3. Find zero crossings of the second derivative Find zero crossings of the second derivative
|
|

\
| +
=
2
2 2
2
exp
2
1
) , (

y x
y x g
c
2 2
2 2
2
2
2
y x
yy y xy y x xx x
g g
g g g g g g g
n
g
+
+ +
=

Image Features
Image Features
Pros: Pros:
One pixel wide edges One pixel wide edges
Edges are grouped together Edges are grouped together
(often good for (often good for
segmentation) segmentation)
Robust against noise! Robust against noise!
Cons: Cons:
Complicated to understand Complicated to understand
and implement and implement
Slow Slow
References
References

Blanken
Blanken
et al, Multimedia Retrieval, 2007, Springer
et al, Multimedia Retrieval, 2007, Springer

Pratt, W: Digital Image Processing, 2001, John Wiley
Pratt, W: Digital Image Processing, 2001, John Wiley
& Sons INC
& Sons INC

Bovik
Bovik
, A: Handbook of Image & Video Processing,
, A: Handbook of Image & Video Processing,
2000, Academic Press
2000, Academic Press

Castelman
Castelman
, K: Digital Image Processing, 1996, Prentice
, K: Digital Image Processing, 1996, Prentice
Hall
Hall

Harris, C: A Combined Corner and Edge Detector,
Harris, C: A Combined Corner and Edge Detector,
1988,
1988,