Chapter two

Machine Vision and Robot Process Capabilities

Lesson outline:
 Introduction
 Image Processing versus Image Analysis
 Two- and Three-Dimensional Image Types
 Acquisition of Images
 Digital Images
 Frequency Domain vs Spatial Domain
 Image-Processing Techniques
 Binary Morphology Operations
 Gray Morphology Operations
 Object Recognition by Features
 Applications of vision system

1
Introduction
 Machine vision involves the acquisition, processing, and
interpretation of image data by computer for some useful
application.
 Vision systems can be classified as 2-dimensional or 3-

dimensional.
 Two-dimensional systems view the scene as a 2-D image,

which is quite adequate for applications involving a planar

object.
 Examples include dimensional measuring and gaging,

verifying the presence of components, and checking for

features on a flat (or almost fl at) surface.
 Three-dimensional vision systems are required for

applications requiring a 3-D analysis of the scene, where

contours or shapes are involved.
2
 Introduction contd…
 "Vision technology “is still a relatively young discipline,
which had its breakthrough in the early1980s.
 It deals with images or sequences of images with the

objective of manipulating and analyzing them in order

to
a) improve image quality (contrast, colour, etc.),
b) restore images(e.g. noise reduction),
c) code pictures (data compression) or
d) understand and interpret images (image analysis,
pattern recognition).
 Machine vision refers the industrial application of vision

technology.
 It describes the understanding and interpretation of

technically obtained images for controlling production

processes.
Operation of Machine Vision Systems
Operation of a machine vision system consists of three
steps, depicted in the Fig.below: (1) Image acquisition and
digitization,
(2) Image processing and analysis, and
(3) Interpretation.

4
1. Image acquisition and digitizing
 Itis accomplished by a video camera connected to a
digitizing system to store the image data for subsequent
processing.
 With the camera focused on the subject, an image is obtained

by dividing the viewing area into a matrix of discrete picture

elements (called pixels), in which each element assumes a
value proportional to the light intensity of that portion of the
scene. The intensity value for each pixel is converted to its
equivalent digital value by analog-to-digital conversion.
 A binary vision system, is shown in the Fig. below in which
Fig.(a) the scene
consists of a dar
the light intensity is reduced to either of two values (black
colored partor
white 0 or 1 in the table), against
a light
background;
(b) a12x12 matr
of pixels imposed
5
 The resolution of a vision system is its ability to sense fine
details and features in the image. This depends on the
number of pixels used. Common pixel arrays include640
(horizontal) 480 (vertical), 1024 768, or 1040 1392 picture
elements.
 The more pixels in the vision system, the higher its

resolution. However, system cost increases as pixel count

increases. Also, time required to read the picture elements
and process the data increases with number of pixels.

6
ILLUMINATION
 Illumination is another important aspect of machine vision.
 The scene viewed by the vision camera must be well illuminated, and

the illumination must be constant over time.

 This almost always requires that special lighting be installed for a

machine
vision application rather than rely on ambient lighting in the facility.

7
 Image acquisition and digitizing contd…
There are two types of vision cameras:
◦ Analog
◦ Digital
 Analog cameras are not very common any more, but are still around.
 Digital cameras are much more common and mostly similar to each other.
 Video camera is a digital camera with added videotape recording section.
 The captured image is analog or digital, in vision system the image is

eventually digitized.
 In a digital form, all data are binary and are stored in computer file.
 The first stage of any vision system is the image acquisition stage.
 After the image has been obtained, various methods of processing can be

applied to the image to perform the many different vision tasks required
today.
 However, if the image has not been acquired satisfactorily then the

intended tasks may not be achievable, even with the aid of some form of

image enhancement
Vidicon camera
 Is an analog camera that transforms an image into analog

electrical signal.
 The signal, variable voltage (current) vs time , can be

stored, digitized, reconstructed into an image( The

fig.below).
 With the use of lens, the scene is projected onto a screen

made up of two layers: a transparent metallic film and a

photoconductive mosaic that is sensitive to light.
 The mosaic reacts to the varying intensity of light by

varying its resistance.

 As the image is projected onto it, the magnitude of the

resistance at each location varies with intensity of light.

 An electron gun generates and sends a continuous cathode

beam through two pairs of capacitors (deflectors).Depending

on the charge on each pair of capacitors, the electron beam is
 Image acquisition and digitizing contd…
Schematic of a vidicon
camera

We routinely change the charges in the two capacitors and thus

deflect the beam both sideways and up and down, so as to
cause it to scan the mosaic (a process called a raster scan).As
the beam scans the image at each instant the output is
proportional to the resistance of the mosaic or proportional to
the intensity of the light on the mosaic. By reading the output
voltage continuously, an analog representation of the can be
 Image acquisition and digitizing contd…

A raster scan depiction of a vidicon:

Scan 1

Scan 2 1

2
3
4

5 Return
6
beam
 Image acquisition and digitizing contd…
Rasterization
 Rasterization is the task of taking an image described in a
vector graphics format (shapes) and converting it into a raster
image (pixels or dots) for output on a video display or printer,
or for storage in a bitmap file format.
 For
Image acquisition and digitizing contd…
color images, the projected image is decomposed into the
three colors of red, green, and blue (RGB).
Each pixel in a color image has a “color value” which is a
combination of amounts of red, green and blue.
Each pixel value is an ordered triple, such as (23,14,51), that
prescribes the densities of the red, green and blue.
Each value in the triple (RGB) has a certain number of bits, and
color depth is the sum of these values.
Many images have a color depth of eight (one byte): 3 bits for the
red and the green and 2 bits for the blue. Each pixel has one of
256 colors.
In highest quality (true-color) images, have a color depth of 24,
one byte for each component.
This achieves the best color production the eye can perceive: More bits do
not improve an image.
However, such images require a lot of memory. A true-color image of
1,080  1,024 pixels requires over 3 million bytes.
 The Color Depth

pixel value

y
Color depth = 6 bits  2 (= 64) colors
6

 Color:Typically one byte per pixel and color. Three bytes are
needed to obtain full color information. One pixel thus
contains three components(R, G, B).
Image acquisition and digitizing contd…
Common Colors

Color depth of three

One bit for each component
Color Value Value
(triple)
0,0,0 Black
0,0,1 Blue
0,1,0 Green
0,1,1 Cyan
1,0,0 Red
1,0,1 Magenta
1,1,0 Yellow
1,1,1 White
 Image acquisition and digitizing contd…

Digital camera
 As with other cameras, a set of lenses is used to project the

area of interest onto the image area of the camera.

 The main part of the camera is a solid-state silicon wafer

image area that has hundreds of thousands of extremely

small photosensitive area called photosites printed on it.
 Each small area of the wafer is a pixel.
 As the image is projected onto the image area, at each pixel

location of the wafer a charge id developed that is

proportional to the intensity of the light.
 Image acquisition and digitizing contd…

Digital images

 A digital image is an electronic file that forms into square picture

elements (pixels) when displayed on a viewing device (e.g., a
computer monitor).
 First digitized through an analog to digital convertor (ADC) and then

either stored in the computer storage unit in an image format such as:
◦ Tagged Image File Format (TIFF) files,
◦ Graphics Interchange Format (GIF) files,
◦ Joint Photographic Group (JPG) files,
◦ Bitmap (BMP) files, etc.
 Image acquisition and digitizing contd…
 The displayed image is a two-dimensional matrix of thousands or
millions of pixels each of which has its own address, size, and
color representation.
 Digitizing a photograph means converting or capturing its image

electronically through a scanner or digital camera.

 Digital image processing software allows you to magnify an image

to see the pixels, and to sometimes measure the numeric color

values for each pixel.
 Each pixel is assigned value (black, white, or color), which is

represented in binary code (0 and 1).

 The binary digits ("bits") for each pixel are stored in a sequence by

a computer and often reduced to a mathematical representation

(compressed).
 The bits are then interpreted and read by the computer to produce

an analog version for display or printing.

Digital images
 Digital images
 There are two important types of digital images—color and
black and white.
 Color images are made up of colored pixels while black and

white images are made of pixels in different shades of gray.

 A black and white image is made up of pixels each of which

holds a single number corresponding to the gray level of the

image at a particular location.
 These gray levels span the full range from black to white in

a series of very fine steps, normally 256 different grays.

 Image acquisition and digitizing contd…
Digital images
 Binary images, in most cases a gray image is converted by
using the histogram of the image, and cut-off value called a
threshold.
 The threshold operation is an important member of the level

mapping class.
 It converts a grayscale image into a binary image.
 It has advantages in terms of both speed and in allowing you

to use some bits of each byte for other purposes (e.g., bits can
be turned on or off for binary masking).
Histogram
 A histogram determines for each gray level, count the number of
pixels having that level for each level, a stick represent the number
of pixels or determines the distribution of the different gray
level(can group nearby levels to form a bin and count number of
pixels in it)
Histogram

 Image « A » has 3 different gray levels : 0, 1 and 2.

 Count the number of pixels for each gray level
There are respectively 24, 12 and 28 pixels for the leve
0, 1and 2
⇒ Histogram of image “A”
Frequency Domain versus Spatial Domain
Many processes used in image processing and analysis are either
based on frequency domain or on spatial domain.
 In frequency domain processing, the frequency spectrum of the
image is used to alter, analyze, or process the image. In this case,
the individual pixels and their contents are not used. Instead, a
frequency representation of the whole image is used for the process.
 In spatial domain processing, the process is applied to the
individual pixels of the image. As a result, each pixel is affected
directly by the process.
 Both techniques are equally important and powerful and are

used for different purposes. It should be noted here that

although spatial and frequency domain techniques are used
differently, they are both related. For example, suppose a spatial
filter is used to reduce noise in an image. As a result of this filter,
noise level in the image will be reduced, but at the same time, the
frequency spectrum of the image will also be affected due to this
reduction in noise. 24
 Frequency domain vs. spatial domain contd…

Frequency Domain
Definition
 Techniques are based on modifying the spectral transform of an

image
 Transform the image to its frequency representation
 Perform image processing
 Compute inverse transform back to the spatial domain

◦High frequencies correspond to pixel values that

change rapidly across the image (e.g. text, texture, leaves, etc.)
◦Strong low frequency components correspond to large
scale features in the image (e.g. a single, homogenous object
that dominates the image) 
Technique
 Filtering example : Smooth an image with a Gaussian
Frequency Content of an Image; Noise, Edges
 The graph is a discrete representation of varying amplitudes

showing the intensity of light at each pixel (or versus time).

 Let’s say we are on the 9th row and are looking at pixel numbers

129–144. The intensity of pixel number 136 is very different from

the ones around it and may be considered noise, which is generally
information that does not belong to the surrounding environment.
 The intensities of pixels 134 and 141 are also different from the

neighboring pixels and may indicate a transition between the

object and the background, and therefore, can be construed as an
edge of the object.

26
 Frequency domain vs. spatial domain

Spatial Domain (Image Enhancement)

Definition
 Spatial Domain is manipulating or changing an image

representing an object in space to enhance the image for a

given application.
 Techniques are based on direct manipulation of pixels in an

image.
 Used for filtering basics, smoothing filters, sharpening

filters, etc.
 Techniques
 Smoothing
Example: A video camera has a 512 x 512 pixel matrix. Each pixel
must be converted from an analog signal to the corresponding
digital signal by an AOC. The analog-to-digital conversion
process takes 0.1 microseconds (O.l x lO-6sec) to complete,
including the time to move between pixels.
(a)How long will it take to collect the image data for one frame ?
(b)Is this time compatible with processing at the rate of 30 frames
per second?
Solution:
There are 512 X 512 = 262.144 pixels to be scanned and converted.
The total time to complete the analog-to-digital conversion process
is:
(262,144 pixels)(O.1 x 10-6 sec) = 0.0262 sec
At a processing rate of 30 frames per second, the processing time
for each frame is 0.0333 sec, which is significantly longer than the
0.0262 sec required to perform the 262,144 analog-to-digital
conversions. 28
2. Image processing and analysis
 As indicated by the above Example, the amount of data that must be
processed is significant. The data for each frame must be analyzed within
the time required to complete one scan (1/30 sec).
 A number of techniques have been developed for analyzing the image dat

in a machine vision system.

Segmentation techniques: are intended to define & separate regions of
interest within the image and the common segmentation techniques are
thresholding,edge detection
◦ Thresholding involves the conversion of each pixel intensity level int
a
binary value, representing either white or black.
◦ Edge detection involves determining the locations of boundaries
between an
object and its surroundings.
Feature extraction is concerned with determining feature values of an
image
Features of an object include area, length, width, or diameter of the29
 Image processing and analysis contd…
 Image processing techniques are used to improve, simplify,
enhance or alter an image to prepare it for image analysis.
 Image processing is dividing into many sub-processes ,

including histogram analysis, thresholding, mask , edge

detection, segmentation, region grow and modeling, among
others.
 The histogram of an image is a plot of the gray levels values

versus the number of pixels at that value.

 A histogram appears as a graph with "brightness" on the

horizontal axis from 0 to 255 (for an 8-bit) intensity scale)

and "number of pixels "on the vertical axis.
 Image processing and analysis contd…

 So the shape of histogram provide us with information about

nature of the image or sub image if we considering an object
within the image.
 For example:

1.Very narrow histogram implies a low-contrast image

2.Histogram skewed to word the high end implies a bright
image
3.Histogram with two major peaks , called bimodal, implies
an object that is in contrast with the background
 Image processing and analysis contd…
Effect of histogram equalization in improving
an image

Number of pixels
Number of pixels

8K 8K
6K 6K
4K 4K
2K 2K

0 50 100 150 200250 0 50 100 150 200250

(a) (b)
Gray Gray
level level
Actual grayness values and number of pixels for the
Leve 1 2 3 4 5 images
6 7 in8 fig.
9 above
10 11 12 13 14 15 16
l
No
of 0 750 5223 8147 8584 7769 6419 5839 5392 5179 5185 3451 2078 16925 341 0
pixel
s

For 0 17 34 51 68 85 102 119 136 153 170 187 204 221 238 256
(b)
Image processing and analysis contd…
Some examples on histogram
 Image processing and analysis contd…
Thresholding is the process of dividing an image into
different portions or levels by picking a certain grayness level
as a threshold, comparing each pixel value with the
threshold&then assigning the pixel to different portions or
level .
 Depending on whether the pixel’s grayness level is below the

threshold (0 or off) or above the threshold (1 or on).

 Thresholding provides an easy and convenient way to

perform image segmentation based on different intensities or

colors in the foreground and background regions of an image.
 Image processing and analysis contd…
 Detecting edges is a basic operation in image processing.
 The edges of items in an image hold much of the information in

the image.
 The edges tell you where:

-items are. -Shape and

-their size. -something about their texture.
 Edge detection methods are used as a first step in the line

detection processes. They are used to find object boundaries by

marking potential edge points corresponding to place in an image
where rapid changes in brightness occur.
 After these edge points have been marked, they can be merged to

form lines and objects outlines.

 Edge detection operations are based on the idea that edge

information in an mage is found by looking at the relationship a

pixel has with its neighbors.
 In practice, edges are caused by: Change in color or texture or
Edge detecting examples
 Image processing vs image analysis
 Image processing is the collection of routines and
techniques that improve, simplify, enhance or alter an image.

 Image analysis is the collection of processes in which a

captured image that is prepared by image processing is
analyzed in order to extract information about the image and
to identify objects or facts about the object.
 Two and three Dimensional images
Images can either be two or three
dimensional.
Two dimensional images are used when the
depth of the scene or its feature need not be
determined.
 A 2D signal or image is a function of two

independent variables, for example, f (x, y).

Three –dimensional image processing deals with operations

that require motion detection, depth measurement, remote
sensing, relative positioning, and navigation.
Photographs of a still scene are the images that are
functions of the (x, y)-plane.
By adding a time variable, the 3D signals represent image
sequences of a dynamic scene that are called video signals
or images.
 Image processing and analysis contd…

Binary Morphology Operations

 Morphology operations refer to a family of operations
performed on the shape (therefore, morphology) of subjects in
an image. They include many different operations, both for
binary and gray images such as thickening, dilation, erosion,
skeletonization, opening, closing, and filling. These operations
are performed on an image in order to aid in image analysis as
well as for reducing the ‘‘extra’’ information that may be
present in the image.
 For example, in the Figure below, the union between the two

lines creates the parallelogram (apply the first line to the

second line), while the union between the two smaller circles is
the larger circle. In this case, the radius of the first circle is
added to the second one, enlarging it. This is called dilation( )
Fig.The union between two geometries creates
39
Examples:
 Dilation :Figure below shows the effect of a union

operation between two shapes.This union reduces the

appearance of the peaks and valleys in the original shape.

 Errosion :The figure below, subtracting the second set

from the first results in an eroded ( ) result

40
 Opening: The figure below shows erosion followed by
dilation (opening) in order to remove protrusion

 Closing: The figure below shows dilation followed by

erosion (closing) This causes a limited smoothing of
concave parts of the object and, like opening, can be used as
an intermediate operation

Figure . As a result of a fill operation, the hole in the nut is filled with
foreground pixels, thus eliminating the hole. 41
 Structuring Elements
A structuring element is a shape mask used in
the basic morphological operations.

They can be any shape and size that is

digitally representable, and each has an origin.

box
hexagon disk something

box(length,width) disk(diameter)
42
 Dilation with Structuring Elements

The arguments to dilation and erosion are

1. a binary image B
2. a structuring element S
dilate(B,S) takes binary image B, places the origin
of structuring element S over each 1-pixel, and ORs
the structuring element S into the output image at
the corresponding position.

0000 dilate 0110

1
0110 0111
11
0000 S 0000
B origin BS
43
Erosion with Structuring Elements

erode(B,S) takes a binary image B, places the origin

of structuring element S over every pixel position, and
ORs a binary 1 into that position of the output image only if
every position of S (with a 1) covers a 1 in B.

origin
0 01 1 0 0 0 0 0 0
1 erode
0 01 1 0 0 0 1 1 0
1
0 01 1 0 0 0 1 1 0
1
1 11 1 1 0 0 0 0 0
B S B S

44
 Opening and Closing

• Closing is the compound operation of dilation followed

by erosion (with the same structuring element)

• Opening is the compound operation of erosion followed

by dilation (with the same structuring element)

45
46
Gray Morphology Operations
 Gray morphology operations are similar to binary morphology

operations, except that they operate on a gray image.

 Usually, a 3 x 3 mask is used to apply the operations, where each

cell in the mask may be either 0 or 1.

 The light areas in gray image are peaks and the dark areas are

valleys. The mask will be applied to the image by moving it from

pixel to pixel. Where the mask matches the gray values in the
image, there are no changes made. If the gray values of the pixels
do not match the mask, they will be changed according to the
selected operation

47
Object Recognition by Features
 Objects in an image may be recognized by their features. These
features may include: gray level histograms; morphological
features such as area, perimeter,number of holes, and others;
eccentricity; cord length; and moments.
Moments
 The moment of an object within an image is the moment of the

object with a&b

indices, x and y are the coordinates of each pixel raised to the
power of a and b, and I is the intensity of the pixel, as in Figure
below. If the image is binary, the intensities are either 1 (or on)
for the object and 0(off) for the background, and thus, only the
pixels that are turned on are considered.
Equation of moment:

where Ma,b is the moment of the object 48

Example : For the simple object in a low-resolution image of
Figure below, calculate the area, center of the area,
and second moments of area of the
object relative to the x1,y1 axes.

Solution: Measuring the distances of each on-pixel from the x1,y1

axes and
substituting the measurements into the moment
equations

49
Aspect Ratio
 Aspect ratio is the width to length ratio of an enclosing
rectangle about the object, as shown in Figure below.
 All aspect ratios are sensitive to orientation, except the

minimum aspect ratio. Therefore, the minimum aspect

ratio is usually used to identify objects

Fig. (a) Aspect ratio of an object, (b) minimum aspect ratio.

50
3. Interpretation of an image
 For any given application, the image must be interpreted
based on the extracted features.
 The interpretation function is usually concerned with

recognizing the object, a task termed object recognition or

pattern recognition.
 The objective in these tasks is to identify the object in the

image by comparing it with predefined models or standard

values.
Two commonly used interpretation techniques are :
 Template matching and
 Feature weighting.
Template matching
 Template matching is the name given to various methods

that attempt to compare one or more features of an image

with the corresponding features of a model or template 51
 Within certain statistical tolerances, the computer determines
whether the image matches the template.
 One of the technical difficulties with this method is the

problem of aligning the part in the same position and

orientation in front of the camera to allow the comparison to
be made without complications in image processing
Feature weighting
 Feature weighting is a technique in which several features

(e.g., area, length, and perimeter) are combined into a single

measure by assigning a weight to each feature according to
its relative importance in identifying the object.
 The score of the object in the image is compared with the

score of an ideal object residing in computer memory to

achieve proper identification
52
Applications of Vision Systems
 Vision systems may be used for many different applications, including in
conjunction with robotic operations and robots.
 Visual guidance and control involves applications in which a vision system is

teamed
with a robot or similar machine to control the movement of the machine.
 Examples of these applications include seam tracking in continuous arc

welding, part positioning and/or reorientation. bin picking, collision

avoidance, machining operations, and assembly tasks.
 Vision systems are commonly used for operations that require information

from the work environment and include inspection, navigation, part

identification, assembly operations, surveillance, control, and communication
 One important part in this operation is the inspection of the board at different

states before and after certain operations. A common method is to set up a cell
where an image of the part is taken and subsequently modified, improved, and
altered.
 The processed image is compared to a look-up image. If there is a match, the

part is accepted. Otherwise, either the part is rejected or is repaired.

53
 Applications of Vision Systems contd…
Industrial machine vision application include the following:
 Dimensional measurement.

These applications involve determining the size of certain

dimensional features of parts or products usually moving at
relatively high speeds on a moving conveyor.
 The machine vision system must compare the features

(dimansions)with the corresponding features of a computer

stored model and determine the size value
• Dimensional gaging.
 This is similar to the preceding except that a gaging function

rather than a measurement is performed.

 Machine vision has proved to he an important element in

automated assembly systems

• Verification of hole location and number of holes in a part.
54
Machine Vision system block diagram for on-line product
inspection and/or process control.
 Applications of Vision Systems contd…
Few examples for (machine) vision technology applications:
 Inspecting
 Applying adhesives evenly and correctly.
 Verifying that welds are strong enough.
 Finding irregularities on flat glass.
 Guiding robots so that they can adapt to changes in their

environment.
 Reading license plates of cars.
 Reading addresses on parcels and checking their dimensions.
Summary
 Image processing can modify, alter, improve, or enhance an
image as well as data can be extracted from an image for
subsequent applications.
 This information may be used for a variety of applications,

including manufacturing, surveillance, navigation, and

robotics.
 Vision systems are very powerful tools that can be used with

ease. They are flexible and inexpensive.

 There are countless routines that can be used for a variety of

different purposes. Most of these types of routines are

created for specific operations and applications.

57
Thank you !!!

58