Chapter Two

Image formation and representation

Computer Vision 2-Image formation and representation

2-1
 Image formation
Idea 1: Put a piece of film in front of an object
◆ Film/sensor: A material that is sensitive to light and can be
used to record the light that impresses it.

It is not a reasonable image.

Computer Vision 2-Image formation and representation
2-2
 Pinhole Camera
Idea 2: The simplest camera system can be designed
by placing a barrier with a small opening between the
3D object and a photographic film or sensor.
◆ The opening known as the aperture.
◆ Only one (or a few) of rays of light passes through the
aperture and hit the film.

Computer Vision 2-Image formation and representation

2-3
 Pinhole Camera
An approximated 1-to-1 mapping can be established
between the 3D object and the film.
The result is that the film gets exposed by an image of
the 3D object by means of this mapping.
◆ This simple camera model is called pinhole camera.

Computer Vision 2-Image formation and representation

2-4
 Pinhole Camera

Computer Vision 2-Image formation and representation

2-5
 Pinhole camera
The film is commonly called image or retinal plane.
The aperture is called pinhole O or center of the camera.
The distance between the image plane and O is the focal
length f.
The line from the camera centre perpendicular to the
image plane is called the principal axis or principal ray
of the camera.
◆ the point where the principal axis meets the image plane is
called the principal point.
Sometimes, the retinal plane is placed between O and
the 3D object at a distance f from O.
◆ In this case it is called virtual image or virtual retinal plane.

Computer Vision 2-Image formation and representation

2-6
 Pinhole camera

Computer Vision 2-Image formation and representation

2-7
 How does the aperture size affect the image?
The larger is the size of the aperture, the greater is the
number of rays of lights that pass through the
aperture.
◆ Thus, each point on the film may corresponds to multiple
points on the object.
◆ As a result, the image of the object becomes blurred.

Computer Vision 2-Image formation and representation

2-8
 How does the aperture size affect the image?
Shrinking the aperture

As the aperture goes from 2mm to 0.35mm, the letters

look more focused but less bright.
Computer Vision 2-Image formation and representation
2-9
 How does the aperture size affect the image?
Why not we make the aperture as small as possible?
◆ When light passes through a small hole, it does not travel in
a straight line and is scattered in many directions (i.e.,
diffraction)

Computer Vision 2-Image formation and representation

2-10
 Problems with pinhole cameras
Generally, pinhole cameras produce dark images,
because a very small set of rays from a particular
point hits the screen.
Pinhole size (aperture) must be very small to obtain a
clear image.
◆ However, as pinhole size is made smaller, less light is
received by image plane.
◆ If pinhole is comparable to wavelength of incoming light,
diffraction effects blur the image.
To overcome this problem we can replace the aperture
with lenses.

Computer Vision 2-Image formation and representation

2-11
 Cameras and lenses
Idea 3: If a lens is properly placed and has proper size,
it satisfies the following property:
◆ All rays of light that are emitted by a point P are refracted
by the lens such that they converge to a single point P’ in
the image plane.
◆ Therefore, the problem of the majority of the light rays
blocked due to a small aperture is removed.

Computer Vision 2-Image formation and representation

2-12
 Cameras and lenses
The above property is true for all the points on the object
that are equidistant from the image plane.
This no longer true for points that are closer or further to
the image plane than P.
◆ The corresponding projection into the image is now blurred
or out of focus (the blue lines).

Circle of
Confusion

Computer Vision 2-Image formation and representation

2-13
 Cameras and lenses
Lenses have a specific distance for which objects are in
focus.
◆ Other points project to a “circle of confusion” in the image
This property is also related to a photography and
computer graphics concept known as depth of field,
which is the effective range at which cameras can take
clear photos.

Computer Vision 2-Image formation and representation

2-14
 Rules of ray diagrams for lenses
1) An incident ray parallel to the principal axis will
refract through the focus.
2) An incident ray through the center of the lens will go
straight through the lens.

Computer Vision 2-Image formation and representation

2-15
 Thin lenses

Thin lens equation:

Any object point satisfying this equation is in focus.

Computer Vision 2-Image formation and representation

2-16
 Thin lens equation
Assume an object & its image are at distance u and v
from the lens plane respectively.

Computer Vision 2-Image formation and representation

2-17
 Thin lens equation

Computer Vision 2-Image formation and representation

2-18
 Thin lens equation

Computer Vision 2-Image formation and representation

2-19
 Digital image space
Let us go back to pinhole model ( the concept holds
for thin lens model too) .
In the pinhole camera model, a point P in 3D (in the
camera reference system) is mapped (projected) into a
point P’ in the image plane Π’.
This R3 to R 2 mapping is referred to as a projective
transformation.

Computer Vision 2-Image formation and representation

2-20
 Pin hole camera
Pin hole camera

Computer Vision 2-Image formation and representation

2-21
 Digital image space
The projection of 3D points into the image plane does
not directly correspond to what we see in actual
digital images (i.e., what we see in a display or
monitor):
◆ points in the digital display are, in general, in a different
reference system than those in the image plane
◆ digital images are divided into pixels whereas points in the
image plane are measured in centimeters;
Next we introduce a number transformations that
allow to map any 3D point from the 3D world directly
into a point in pixel coordinates in the digital image.

Computer Vision 2-Image formation and representation

2-22
 Digital image space

Computer Vision 2-Image formation and representation

2-23
 Transformations in 2D
Let us discuss 2D geometrical transformations :
◆ Translation
◆ Scaling
◆ Rotation

Computer Vision 2-Image formation and representation

2-24
 Translation
Use vector for the notation:
◆ makes things simpler
A point is a vector: P=  x 
y 
 
A translation is merely a vector sum:
P’ = P + T

Before After
Computer Vision 2-Image formation and representation
2-25
 Scaling
Coordinates multiplied by the scaling factor:
• x’ = sx x
• y’ = sy y

Before After
Computer Vision 2-Image formation and representation
2-26
 Scaling
Scaling is a matrix multiplication:

P’ = SP
x  sx 0  x 
y  =  0   
   sy  y 

Computer Vision 2-Image formation and representation

2-27
 Rotating in 2D

New coordinates depend on both x and y

• x’ = cosq x - sinq y
• y’ = sinq x + cosq y

Before After
Computer Vision 2-Image formation and representation
2-28
 Rotating in 2D, matrix notation
A rotation is a matrix multiplication:
P’=RP

x  cosq − sin q  x 
y  = sin q  
cosq  y  
  

Computer Vision 2-Image formation and representation

2-29
 2D transformations, summary
Vector-matrix notation simplifies writing:
◆ translation is a vector sum
◆ rotation and scaling are matrix-vector multiplication
We would like a consistent notation:
◆ that expresses all three identically
◆ that expresses combination of these also identically
How to do this?
◆ Homogeneous coordinates

Computer Vision 2-Image formation and representation

2-30
 Homogeneous Coordinates
Multiply the coordinates by a non-zero scalar and add
an extra coordinate equal to that scalar.

W is an arbitrary non-zero scalar, usually we choose 1.

Computer Vision 2-Image formation and representation

2-31
 Back to Cartesian Coordinates
Divide by the last coordinate and eliminate it. For
example,

Computer Vision 2-Image formation and representation

2-32
 Example
1) What is the 2D Cartesian coordinates of the points
with homogeneous coordinates?
6 7 6𝜔
◆ 𝑥1 = −3 𝑥2 = −2 𝑥3 = −3𝜔
3 1 3𝜔
2) What is the interpretation of the point with
homogeneous coordinates?
6
𝑥4 = −3
0

Computer Vision 2-Image formation and representation

2-33
 2D Translation using Homogeneous Coordinates

𝟏 𝟎 𝒕𝒙
𝑻= 𝟎 𝟏 𝒕𝒚
𝟎 𝟎 𝟏

Before P’ After
Computer Vision 2-Image formation and representation
2-34
 2D scaling using Homogeneous Coordinates

𝒔𝒙 𝟎 𝟎
𝑺 = 𝟎 𝒔𝒚 𝟎
𝟎 𝟎 𝟏

Before After
Computer Vision 2-Image formation and representation
2-35
 2D rotation using Homogeneous Coordinates
𝑥 cos 𝜃 − y sin 𝜃 𝑥
𝑃′ → 𝑥 sin 𝜃 + 𝑦 cos 𝜃 = ? ∙ 𝑦
1 1
𝑋 𝑋
= 𝑹′ 𝟎 ∙ 𝑌 = 𝑹 ∙ 𝑌
𝟎 1
1 1
cos 𝜽 −sin 𝜽 𝟎
𝑹 = sin 𝜽 cos 𝜽 𝟎
𝟎 𝟎 𝟏

Before After
Computer Vision 2-Image formation and representation
2-36
 Rotation in 3D
Rotate around each coordinate axis:

Combine the three rotations:

Computer Vision 2-Image formation and representation

2-37
 Four Coordinate Frames

Computer Vision 2-Image formation and representation

2-38
 World reference system
So far we have described a mapping between a point
P in the 3D camera reference system to a point P’ in
the 2D image plane.
But what if the information we have about points in
3D is available in the world coordinate system?
◆ Then, we need to include an additional transformation that
allows to relate points from the world reference system to
the camera reference system.

Computer Vision 2-Image formation and representation

2-39
 World to Camera Coordinate
The transformation that allows to relate points from
the world reference system to the camera reference
system is regulated by a translation R and a rotation T
in 3D.

Computer Vision 2-Image formation and representation

2-40
 World to Camera Coordinate

Computer Vision 2-Image formation and representation

2-41
 Pinhole camera (Perspective Projection )
From camera coordinate to image plane coordinate
(i.e from 3D to 2D)

Computer Vision 2-Image formation and representation

2-42
 Perspective Projection
These are not linear

Using homogenous coordinate, we have a linear

relation:

u=x’, v=y’
Computer Vision 2-Image formation and representation
2-43
 Image Coordinates to Pixel Coordinates
Image coordinates have their origin C’ at the image
center where the z axis of the camera reference system
intersects the image plane.
On the other hands, digital images have their origin at
the lower-left corner of the image.
◆ Thus, 2D points in the image plane and 2D points in a
digital image are equal up to an offset translation vector
[cx, cy], where C’’= [cx, cy] is the location of C’ in the digital
image.
◆ In order to accommodate this change of coordinate systems,
the mapping is now described by translating the image
plane by cx and cy in the x and y direction.

Computer Vision 2-Image formation and representation

2-44
 Image Coordinates to Pixel Coordinates

Computer Vision 2-Image formation and representation

2-45
 Image Coordinates to Pixel Coordinates
In addition to the translation b/n the image and pixel
coordinates, we need to consider the fact that points in a
digital image are expressed in pixels whereas points in the
image plane are expressed in metric units.
In order to accommodate this further change of
coordinate systems, the mapping is now described by:

Since we have different scale factors in x and y, we

don’t necessarily have square pixels (k might be
different from l).
Aspect ratio is k/l.
Computer Vision 2-Image formation and representation
2-46
 Image Coordinates to Pixel Coordinates
If 𝑓𝑘 = 𝛼 𝑎𝑛𝑑 𝑓𝑙 = 𝛽

Using homogenous coordinate system

Computer Vision 2-Image formation and representation

2-47
 Camera Coordinates to Pixel Coordinates
Pc : P in the camera coordinate
I is a 3x3 identity matrix and
K is known as the camera matrix.

Computer Vision 2-Image formation and representation

2-48
 World Coordinates to Pixel Coordinates
Remember the relation between the world and camera
coordinate systems:

Hence,

Computer Vision 2-Image formation and representation

2-49
 Projection Matrix
The 3x4 matrix 𝑴 = 𝑲 𝑹 𝑻 is called Projection Matrix.
The 3x3 matrix K are called internal or intrinsic
parameter.
◆ Related to the intrinsic physical properties of the camera
focal length (f)
the scale factor in the x-coordinate direction (k)
the scale factor in the y-coordinate direction (l)
the coordinates of the principal point (cx, cy)
The 3x4 matrix [R T] are called the external or
extrinsic parameters.
◆ Related to pose and location of the camera.
rotation angles, translation

Computer Vision 2-Image formation and representation

2-50
 Image representation
Images are actually spatial data (data which is
indexed by two spatial co-ordinates).
◆ The camera senses the brightness at a point.
◆ The brightness sensed by the camera is transformed to a
signal which is then fed to the A/D converter and stored as
a value within the computer, referenced to the co-ordinates
x, y in the image.
◆ Accordingly, a computer image is a matrix of points. These
points are the picture elements, or pixels.
◆ There are different types of images: Binary images,
Greyscale images and Color images.

Computer Vision 2-Image formation and representation

2-51
 Image Types
Binary Images:
◆ contain pixels that are either black (0) or white (1).
◆ Usually applicable where the only information required is
general shape and outline.
Grayscale Images:
◆ have a wider range of intensity than black and white.
◆ Each pixel is a shade of gray with pixel values ranging
between 0 (black) and 255 (white).
Grayscale image contains 8bits/pixel.
Color Images:
◆ have multiple color channels;
◆ each color image can be represented in different color models
(color spaces).
Computer Vision 2-Image formation and representation
2-52
 Types of images

Computer Vision 2-Image formation and representation

2-53
 Color space
A color space describes the range of colors that a
camera can see.
Each pixel in an image has a color that is described in
the color space.
There are different ways to describe all colors, so there
are also different color spaces.
◆ RGB-space

Computer Vision 2-Image formation and representation

2-54
 RGB color space
In the RGB color space
◆ each color is described as a combination of three main
colors, namely Red, Green and Blue.
◆ color space can be visualized as a 3D matrix with the main
colors set out on the axis.
◆ image on a computer is thus transformed into 3 matrices
with values per pixel for the representing main color.

Computer Vision 2-Image formation and representation

2-55
 RGB color space

Computer Vision 2-Image formation and representation

2-56
 Images as functions

We think of an image as a function, 𝑓, from 𝑅2 to 𝑅:

◆ 𝑓(x,y) gives the intensity or value at position (x,y)
Practically define the image over a rectangle, with a
finite range:
𝑓: [a,b] x [c,d] →[min, max]
In the case of color images , an image can be
considered as a multi-valued function.
◆ The multiple values may correspond to different color
intensities.
◆ An RGB image is just three functions “stacked” together.

Computer Vision 2-Image formation and representation

2-57
 Digital images
In computer vision we typically operate on digital
(discrete) images.
Digitalization of an image involves two operations:
◆ Sampling
◆ Quantization
Both operations correspond to a discretization of a
quantity, but in different domains.
Sampling corresponds to a discretization of the space
(That is, of the domain of the function) into a regular
grid.
f : [1 ,…., N ] x[ ,…., M ] →R m

Computer Vision 2-Image formation and representation

2-58
 Sampling
After sampling, the image can be seen as matrix.

The smallest element resulting from the discretization

of the space is called a pixel (picture element).

Computer Vision 2-Image formation and representation

2-59
 Quantization
Quantization corresponds to a discretization of the
intensity values.

After sampling and quantization, we get

f : [1, …. , N ]x[ 1 , …. , M ] →[0 ,….,L ]

Computer Vision 2-Image formation and representation

2-60
 Digital images
Image thus represented as a matrix of integer values.

Computer Vision 2-Image formation and representation

2-61
 Binary image representation

Computer Vision 2-Image formation and representation

2-62
 Grayscale image representation

Computer Vision 2-Image formation and representation

2-63
 Color image representation

Computer Vision 2-Image formation and representation

2-64
 Color image representation

Original RGB colour image (upper left), and its decomposition into the
three contributing channels: Red (upper right), Green (lower left), and Blue
(lower right). For example, red is shown with high intensity in the red
channel,but in low intensity in the green and blue channel

Computer Vision 2-Image formation and representation

2-65
 Matlab
Use imread to read an image into Matlab:
» img = imread (’peppers.jpg’);
Use size to know the size of image.
» size (img)
Use imshow to display the image:
» imshow(img);
» imshow(img(:,:,1)); % Shows only the red component of the image
Use imwrite to save an image from Matlab:
» imwrite (img,’peppers2.jpg’,’jpg’);
» imwrite (img(:,:,1),’peppersR.jpg’,’jpg’); % Saves only the red
component of the image

Im2bw- to convert gray image into binary image

Computer Vision 2-Image formation and representation
2-66