Image formation and fundamentals

1
 IP framework
Natural scene Digital image

15 25
44 100

capture
sampling
quantization
color space
filtering
transforms
coding
Is this good What is the best I ....
quality can get over my
phone line?
How can I
protect Image
my data? Network
Network Processing
System

How much Image rendering

will it cost?

2
 Analog image IP: basic steps

(capturing device)
A/D conversion

Sampling (2D) Quantization

Digital image

{15,1,2}
{25,44,1}
….

3
Digital Image Acquisition
Sensor array

• When photons strike, electron-hole pairs are generated on

sensor sites.
• Electrons generated are collected over a certain period of time.
• The number of electrons are converted to pixel values. (Pixel is
short for picture element.)

4
VM1

Image capture
Object Light source
(surface element)

theta
N

Irradiance
Surface reflectance Radiance CAMERA

Optical axis
Sensors

5
Slide 5

VM1 - radianza: energia che viene emessa dall'elemento di superficie

- irradianza: energia che colpisce la camera e dipende da lo spettro della luce, la riflettanza della superficie (che cambia lo spettro) e la sensibilità spettrale
del sensore
swan; 14/01/2004
Digital Image Acquisition

Two types of discretization:

1. There are finite number of pixels.
(sampling → Spatial resolution)
2. The amplitude of pixel is
represented by a finite number of
bits. (Quantization → Gray-scale
resolution)

6
 Digital Image Acquisition

Take a look at
this cross
section

7
Digital Image Acquisition
• 256x256 - Found on very cheap cameras,
this resolution is so low that the picture
quality is almost always unacceptable. This
is 65,000 total pixels.
• 640x480 - This is the low end on most "real"
cameras. This resolution is ideal for e-
mailing pictures or posting pictures on a
Web site.
• 1216x912 - This is a "megapixel" image size
-- 1,109,000 total pixels -- good for printing
pictures.
• 1600x1200 - With almost 2 million total
pixels, this is "high resolution." You can print
a 4x5 inch print taken at this resolution with
the same quality that you would get from a
photo lab.
• 2240x1680 - Found on 4 megapixel
cameras -- the current standard -- this
allows even larger printed photos, with good
quality for prints up to 16x20 inches.
• 4064x2704 - A top-of-the-line digital camera
with 11.1 megapixels takes pictures at this
resolution. At this setting, you can create
13.5x9 inch prints with no loss of picture
quality.

8
Basics: graylevel images

100 100 200 90

50 0 50 200
100 200 100 50
100 0 200 100

Images : Matrices of numbers

Image processing : Operations among numbers
bit depth : number of bits/pixel
N bit/pixel : 2N-1 shades of gray (typically N=8)

9
 Matrix Representation of Images
• A digital image can be written as a matrix

⎡ x[0, 0] x[0,1] " x[0, N − 1] ⎤

⎢ x[1, 0] x[1,1] " x[1, N − 1] ⎥⎥
x[ n1 , n2 ] = ⎢
⎢ # # % # ⎥
⎢ ⎥
⎣ x[ M − 1, 0] " " x[ M − 1, N − 1]⎦ MxN

⎡ 35 45 20 ⎤
⎢ 43 64 52 ⎥⎥
⎢
⎢⎣10 29 39 ⎥⎦

10
 Digital images acquisition
• Analog camera+A/D converter
• Digital cameras
– CCDs (Charge Coupled Devices)
– CMOS technology
• In both cases: optics
– lenses, diaphrams

Matrices of photo Features of the capture devices:

sensors collecting • Size and number of photosites
photons of given • Noise
wavelength • Transfer function of the optical filter

11
 Color images
C1 C2 C3

• Each colored pixel corresponds to a vector of three values {C1,C2,C3}

• The characteristics of the components depend on the chosen colorspace (RGB,
YUV, CIELab,..)

12
 Digital Color Images
• x R [ n1 , n2 ]
xG [ n1 , n2 ]
x B [ n1 , n2 ]

13
 Color channels

Green Blue
Red

14
 Color channels

Red Green Blue

15
The physical perspective

16
 The perceptual perspective
Simultaneous contrast

17
 Color
• Chromatic induction

18
 Color
• Human vision • Colorimetry
– Color encoding (receptor level) – Spectral properties of radiation
– Color perception (post-receptoral – Physical properties of materials
level)
– Color semantics (cognitive level)
Color categorization and
naming (understanding
colors)

MODELS

Colorimetry
(Measuring colors)
Color vision
(Seeing colors)

19
 Bayer matrix

Typical sensor topology in CCD devices. The green is twice as numerous as red and blue.

20
 Displays
CRT

LCD

21
 Color Displays

CRT

LCD

Polarize to control the amount of light passed.

22
 Color imaging
• Color reproduction
– Printing, rendering
• Digital photography
– High dynamic range images
– Mosaicking
– Compensation for differences in illuminant (CAT: chromatic adaptation
transforms)
• Post-processing
– Image enhancement
• Coding
– Quantization based on color CFSs (contrast sensitivity function)
– Downsampling of chromatic channels with respect to luminance

23
 Some definitions

• Digital images
– Sampling+quantization

• Sampling
– Determines the graylevel value of each pixel
• Pixel = picture element

• Quantization
– Reduces the resolution in the graylevel value to that set by the
machine precision

• Images are stored as matrices of unisigned chars

24
 Resolution
• Sensor resolution (CCD): Dots Per Inch (DPI)
– Number of individual dots that can be placed within the span of one linear inch
(2.54 cm)
• Image resolution
– Pixel resolution: NxM
– Spatial resolution: Pixels Per Inch (PPI)
VM2 – Spectral resolution: bandwidth of each spectral component of the image
• Color images: 3 components (R,G,B channels)
• Multispectral images: many components (ex. SAR images)
– Radiometric resolution: Bits Per Pixel (bpp)
• Graylevel images: 8, 12, 16 bpp
• Color images: 24bpp (8 bpp/channel)
– Temporal resolution: for movies, number of frames/sec
• Typically 25 Hz (=25 frames/sec)

25
Slide 25

VM2 da book Shapiro

swan; 10/04/2003
Example: pixel resolution

26
 Image Resolution
Don’t confuse image size and resolution.

27
 Bit Depth – Grayscale Resolution

8 bits

7 bits

6 bits 5 bits

28
Bit Depth – Grayscale Resolution
4 bits

3 bits

1 bit
2 bits

29
 File format
• Many image formats (about 44)
• BMP, lossless
• TIFF, lossless/lossy
• GIF (Graphics Interchange Format)
– Lossless, 256 colors, copyright protected
• JPEG (Joint Photographic Expert Group)
– Lossless and lossy compression
– 8 bits per color (red, green, blue) for a 24-bit total
• PNG (Portable Network Graphics)
– Freewere
– supports truecolor (16 million colours)

30
Sampling in 2D

31
 Sampling in 1D
Continuous time signal
f(t)

t
Discrete time signal
f [k ] = f (kTs ) = f (t )∑ δ ( t − kTs )
k

f(t)
comb

k
32
Nyquist theorem (1D)

At least 2 sample/period are needed to

represent a periodic signal

1 2π
Ts ≤
2 ωmax
2π
ωs = ≥ 2ωmax
Ts

33
Delta pulse

34
Dirac brush

35
Comb

36
Brush

37
 Nyquist theorem
2D spatial domain
• Sampling in p-dimensions Tsx
G G Tsy
sT ( x ) = ∑ − kT )
δ ( x
k∈Z p
G G G
fT ( x ) = f ( x ) sT ( x )

2D Fourier domain

ωy
• Nyquist theorem
ωymax
⎧ s 1
T
⎪ x ≤ 2π
⎧⎪ω xs ≥ 2ω x max ⎪ 2ω x max ωxmax ω x
⎨ s ⇒⎨
⎪⎩ω y ≥ 2ω y max ⎪Tys ≤ 2π
1
⎪⎩ 2ω y max

38
Spatial aliasing

39
 Resampling
• Change of the sampling rate
– Increase of sampling rate: Interpolation or upsampling
• Blurring, low visual resolution
– Decrease of sampling rate: Rate reduction or downsampling
• Aliasing and/or loss of spatial details

40
Downsampling

41
Upsampling

nearest neighbor (NN)

42
Upsampling

bilinear

43
Upsampling

bicubic

44
Quantization

45
 Quantization
• A/D conversion Ö quantization
f in L2(R) discrete function
f in L2(Z)
Quantizer

uniform perceptual
fq=Q{f} fq=Q{f}

rk

tk tk+1 f f
The sensitivity of the eye decreases
increasing the background intensity
(Weber law)

46
 Quantization
Signal before (blue) and after quantization (red) Q

Equivalent noise: n=fq- f

additive noise model: fq=f+n

47
 Quantization

original 5 levels

50 levels
10 levels

48
 Distortion measure
• Distortion measure
[ ) ]= ∑ ∫ ( f
t k +1

D = Ε ( fQ − f )
K
− f
2 2
Q p ( f ) df
k = 0 tk

– The distortion is measured as the expectation of the mean square error (MSE)
difference between the original and quantized signals.
• Lack of correlation with perceived image quality
– Even though this is a very natural way for the quantification of the quantization
artifacts, it is not representative of the visual annoyance due to the majority of
common artifacts.

• Visual models are used to define perception-based image quality

assessment metrics

49
 Example
• The PSNR does not allow to distinguish among different types of
distortions leading to the same RMS error between images

• The MSE between images (b) and (c) is the same, so it is the PSNR.
However, the visual annoyance of the artifacts is different

50