Computer Vision

Introduction
In the previous chapter, you studied the concepts of Artificial Intelligence for Data Sciences. It is a
concept to unify statistics, data analysis, machine learning and their related methods in order to
understand and analyse actual phenomena with data.

As we all know, artificial intelligence is a technique that enables computers to mimic human
intelligence. As humans we can see things, analyse it and then do the required action on the basis of
what we see.

But can machines do the same? Can machines have the eyes that humans have? If you answered Yes,
then you are absolutely right. The Computer Vision domain of Artificial Intelligence, enables machines
to see through images or visual data, process and analyse them on the basis of algorithms and
methods in order to analyse actual phenomena with images.

Now before we get into the concepts of Computer Vision, let us experience this domain with the help
of the following game:

* Emoji Scavenger Hunt :

https://emojiscavengerhunt.withgoogle.com/

Go to the link and try to play the game of Emoji Scavenger Hunt. The challenge here is to find 8 items
within the time limit to pass.

Did you manage to win?

__________________________________________________________________________________
__________________________________________________________________________________

What was the strategy that you applied to win this game?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

Was the computer able to identify all the items you brought in front of it?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Did the lighting of the room affect the identifying of items by the machine?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

Applications of Computer Vision

The concept of computer vision was first introduced in the 1970s. All these new applications of
computer vision excited everyone. Having said that, the computer vision technology advanced enough
to make these applications available to everyone at ease today. However, in recent years the world
witnessed a significant leap in technology that has put computer vision on the priority list of many
industries. Let us look at some of them:

Facial Recognition*: With the advent of smart cities and smart homes,
Computer Vision plays a vital role in making the home smarter. Security
being the most important application involves use of Computer Vision
for facial recognition. It can be either guest recognition or log
maintenance of the visitors.

It also finds its application in schools for an attendance system based on

facial recognition of students.

Face Filters*: The modern-day apps like Instagram and snapchat have
a lot of features based on the usage of computer vision. The
application of face filters is one among them. Through the camera the
machine or the algorithm is able to identify the facial dynamics of the
person and applies the facial filter selected.

Google’s Search by Image*: The maximum amount

of searching for data on Google’s search engine comes
from textual data, but at the same time it has an
interesting feature of getting search results through an
image. This uses Computer Vision as it compares
different features of the input image to the database
of images and give us the search result while at the
same time analysing various features of the image.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Computer Vision in Retail*: The retail field has been one of the
fastest growing field and at the same time is using Computer
Vision for making the user experience more fruitful. Retailers can
use Computer Vision techniques to track customers’ movements
through stores, analyse navigational routes and detect walking
patterns.
Inventory Management is another such application. Through
security camera image analysis, a Computer Vision algorithm can
generate a very accurate estimate of the items available in the
store. Also, it can analyse the use of shelf space to identify
suboptimal configurations and suggest better item placement.

Self-Driving Cars: Computer Vision is the fundamental

technology behind developing autonomous vehicles.
Most leading car manufacturers in the world are
reaping the benefits of investing in artificial intelligence
for developing on-road versions of hands-free
technology.

This involves the process of identifying the objects,

getting navigational routes and also at the same time
environment monitoring.

Medical Imaging*: For the last decades, computer-

supported medical imaging application has been a
trustworthy help for physicians. It doesn’t only
create and analyse images, but also becomes an
assistant and helps doctors with their interpretation.
The application is used to read and convert 2D scan
images into interactive 3D models that enable
medical professionals to gain a detailed
understanding of a patient’s health condition.

Google Translate App*: All you need to do to read signs in a

foreign language is to point your phone’s camera at the words and
let the Google Translate app tell you what it means in your preferred
language almost instantly. By using optical character recognition to
see the image and augmented reality to overlay an accurate
translation, this is a convenient tool that uses Computer Vision.

OCR stands for "Optical Character Recognition."

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Computer Vision: Getting Started
Computer Vision is a domain of Artificial Intelligence, that deals with the images. It involves the
concepts of image processing and machine learning models to build a Computer Vision based
application.

Computer Vision Tasks

The various applications of Computer Vision are based on a certain number of tasks which are
performed to get certain information from the input image which can be directly used for prediction
or forms the base for further analysis. The tasks used in a computer vision application are :

For Single For Multiple

Objects Objects

Object
Classification
Detection

Classification + Instance
Localisation Segementation

Classification
Image Classification problem is the task of assigning an input image one label from a fixed set of
categories. This is one of the core problems in CV that, despite its simplicity, has a large variety of
practical applications.

Classification + Localisation
This is the task which involves both processes of identifying what object is present in the image and
at the same time identifying at what location that object is present in that image. It is used only for
single objects.

Object Detection
Object detection is the process of finding instances of real-world objects such as faces, bicycles, and
buildings in images or videos. Object detection algorithms typically use extracted features and
learning algorithms to recognize instances of an object category. It is commonly used in applications
such as image retrieval and automated vehicle parking systems.

Instance Segmentation
Instance Segmentation is the process of detecting instances of the objects, giving them a category and
then giving each pixel a label on the basis of that. A segmentation algorithm takes an image as input
and outputs a collection of regions (or segments).
Basics of Images
We all see a lot of images around us and use them daily either through our mobile phones or computer
system. But do we ask some basic questions to ourselves while we use them on such a regular basis.

Don’t know the answer yet? Don’t worry, in this section we will study about the basics of an image:

Basics of Pixels
The word “pixel” means a picture element. Every photograph, in digital form, is made up of pixels.
They are the smallest unit of information that make up a picture. Usually round or square, they are
typically arranged in a 2-dimensional grid.

In the image below, one portion has been magnified many times over so that you can see its individual
composition in pixels. As you can see, the pixels approximate the actual image. The more pixels you
have, the more closely the image resembles the original.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Resolution
The number of pixels in an image is sometimes called the resolution. When the term is used to describe
pixel count, one convention is to express resolution as the width by the height, for example a monitor
resolution of 1280×1024. This means there are 1280 pixels from one side to the other, and 1024 from
top to bottom.

Another convention is to express the number of pixels as a single number, like a 5 mega pixel camera
(a megapixel is a million pixels). This means the pixels along the width multiplied by the pixels along
the height of the image taken by the camera equals 5 million pixels. In the case of our 1280×1024
monitors, it could also be expressed as 1280 x 1024 = 1,310,720, or 1.31 megapixels.

Pixel value
Each of the pixels that represents an image stored inside a computer has a pixel value which describes
how bright that pixel is, and/or what colour it should be. The most common pixel format is the byte
image, where this number is stored as an 8-bit integer giving a range of possible values from 0 to 255.
Typically, zero is to be taken as no colour or black and 255 is taken to be full colour or white.

Why do we have a value of 255 ? In the computer systems, computer data is in the form of ones and
zeros, which we call the binary system. Each bit in a computer system can have either a zero or a one.

Since each pixel uses 1 byte of an image, which is equivalent to 8 bits of data. Since each bit can have
two possible values which tells us that the 8 bit can have 255 possibilities of values which starts from
0 and ends at 255.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Grayscale Images
Grayscale images are images which have a range of shades of gray without apparent colour. The
darkest possible shade is black, which is the total absence of colour or zero value of pixel. The lightest
possible shade is white, which is the total presence of colour or 255 value of a pixel . Intermediate
shades of gray are represented by equal brightness levels of the three primary colours.

A grayscale has each pixel of size 1 byte having a single plane of 2d array of pixels. The size of a
grayscale image is defined as the Height x Width of that image.

Let us look at an image to understand about grayscale images.

Here is an example of a grayscale image. as you check, the value of pixels are within the range of 0-
255.The computers store the images we see in the form of these numbers.

RGB Images
All the images that we see around are coloured images. These images are made up of three primary
colours Red, Green and Blue. All the colours that are present can be made by combining different
intensities of red, green and blue.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Let us experience!

Go to this online link https://www.w3schools.com/colors/colors_rgb.asp. On the basis of this online

tool, try and answer all the below mentioned questions.

1) What is the output colour when you put R=G=B=255 ?

___________________________________________________________________________

2) What is the output colour when you put R=G=B=0 ?

___________________________________________________________________________

3) How does the colour vary when you put either of the three as 0 and then keep on varying
the other two?

___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________

4) How does the output colour change when all the three colours are varied in same
proportion ?

___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________

5) What is the RGB value of your favourite colour from the colour palette?

___________________________________________________________________________

Were you able to answer all the questions? If yes, then you would have understood how every colour
we see around is made.

Now the question arises, how do computers store RGB images? Every RGB image is stored in the form
of three different channels called the R channel, G channel and the B channel.

Each plane separately has a number of pixels with each pixel value varying from 0 to 255. All the three
planes when combined together form a colour image. This means that in a RGB image, each pixel has
a set of three different values which together give colour to that particular pixel.
For Example,

As you can see, each colour image is stored in the form of three different channels, each having
different intensity. All three channels combine together to form a colour we see.

In the above given image, if we split the image into three different channels, namely Red (R), Green
(G) and Blue (B), the individual layers will have the following intensity of colours of the individual
pixels. These individual layers when stored in the memory looks like the image on the extreme right.
The images look in the grayscale image because each pixel has a value intensity of 0 to 255 and as
studied earlier, 0 is considered as black or no presence of colour and 255 means white or full presence
of colour. These three individual RGB values when combined together form the colour of each pixel.

Therefore, each pixel in the RGB image has three values to form the complete colour.

Task :
Go to the following link www.piskelapp.com and create your own pixel art. Try and make a GIF using
the online app for your own pixel art.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Image Features
In computer vision and image processing, a feature is a piece of information which is relevant for
solving the computational task related to a certain application. Features may be specific structures in
the image such as points, edges or objects.
For example:
Imagine that your security camera is capturing an image. At the top of the image we are given six small
patches of images. Our task is to find the exact location of those image patches in the image.
Take a pencil and mark the exact location of those patches in the image.

Were you able to find the exact location of all the patches?
__________________________________________________________________________________

Which one was the most difficult to find?

__________________________________________________________________________________
__________________________________________________________________________________

Which one was the easiest to find?

__________________________________________________________________________________
__________________________________________________________________________________

Let’s Reflect:
Let us take individual patches into account at once and then check the exact location of those patches.
For Patch A and B: The patch A and B are flat surfaces in the image and are spread over a lot of area.
They can be present at any location in a given area in the image.
For Patch C and D: The patches C and D are simpler as compared to A and B. They are edges of a
building and we can find an approximate location of these patches but finding the exact location is
still difficult. This is because the pattern is the same everywhere along the edge.
For Patch E and F: The patches E and F are the easiest to find in the image. The reason being that E
and F are some corners of the building. This is because at the corners, wherever we move this patch
it will look different.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Conclusion
In image processing, we can get a lot of features from the image. It can be either a blob, an edge or a
corner. These features help us to perform various tasks and then get the analysis done on the basis of
the application. Now the question that arises is which of the following are good features to be used?
As you saw in the previous activity, the features having the corners are easy to find as they can be
found only at a particular location in the image, whereas the edges which are spread over a line or an
edge look the same all along. This tells us that the corners are always good features to extract from
an image followed by the edges.
Let’s look at another example to understand this. Consider the images given below and apply the
concept of good features for the following.

In the above image how would we determine the exact location of each patch?
The blue patch is a flat area and difficult to find and track. Wherever you move the blue patch it looks
the same. The black patch has an edge. Moved along the edge (parallel to edge), it looks the same.
The red patch is a corner. Wherever you move the patch, it looks different, therefore it is unique.
Hence, corners are considered to be good features in an image.

Introduction to OpenCV
Now that we have learnt about image features and its importance in image processing, we will learn
about a tool we can use to extract these features from our image for further processing.
OpenCV or Open Source Computer Vision Library is that tool which helps a computer extract these
features from the images. It is used for all kinds of images and video processing and analysis. It is
capable of processing images and videos to identify objects, faces, or even handwriting.

In this chapter we will use OpenCV for basic image processing operations on
images such as resizing, cropping and many more.
To install OpenCV library, open anaconda prompt and then write the following
command:

pip install opencv-python Preferred Installer Program

Now let us take a deep dive on the various functions of OpenCV to understand the various image
processing techniques. Head to Jupyter Notebook for introduction to OpenCV given on this link:
http://bit.ly/cv_notebook

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Convolution
We have learnt that computers store images in numbers, and that pixels are arranged in a particular
manner to create the picture we can recognize. These pixels have value varying from 0 to 255 and the
value of the pixel determines the color of that pixel.

But what if we edit these numbers, will it bring a change to the image? The answer is yes. As we change
the values of these pixels, the image changes. This process of changing pixel values is the base of image
editing.

We all use a lot of image editing software like photoshop and at the same time use apps like Instagram
and snapchat, which apply filters to the image to enhance the quality of that image.

As you can see, different filters applied to an image change the pixel values evenly throughout the
image. How does this happen? This is done with the help of the process of convolution and the
convolution operator which is commonly used to create these effects.

Before we understand how the convolution operation works, let us try and create a theory for the
convolution operator by experiencing it using an online application.

Task
Go to the link http://matlabtricks.com/post-5/3x3-convolution-kernels-with-online-demo and at the
bottom of the page click on load “Click to Load Application”

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Once the application is loaded try different filters and apply it on the image. Observe how the value
of the kernel is changing for different filters. Try these steps

1) Change all to positive values

2) Change all to negative values
3) Have a mixture of negative and positive values

Let us follow the following steps to understand how a convolution operator works. The steps to be
followed are:

Try experimenting with the following values to come up with a theory:

1) Make 4 numbers negative. Keep the rest as 0.

2) Now make one of them as positive.
3) Observe what happens.
4) Now make the second positive.

What theory do you propose for convolution on the basis of the observation?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

It is time to test the theory. Change the location of the four numbers and follow the above mentioned
steps. Does your theory hold true?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
If yes, change the picture and try whether the theory holds true or not. If it does not hold true, modify
your theory and keep trying until it satisfies all the conditions.

Let’s Discuss
What effect did you apply?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

How did different kernels affect the image?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

Why do you think we apply these effects?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

How do you think the convolution operator works?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

Convolution : Explained
Convolution is a simple Mathematical operation which is fundamental to many
common image processing operators. Convolution provides a way of `multiplying together' two arrays
of numbers, generally of different sizes, but of the same dimensionality, to produce a third array of
numbers of the same dimensionality.

An (image) convolution is simply an element-wise multiplication of image arrays and another array
called the kernel followed by sum.
As you can see here,

I = Image Array

K = Kernel Array

I * K = Resulting array after performing the convolution operator

Note: The Kernel is passed over the whole image to get the resulting array after convolution.
In image processing, a kernel, convolution matrix,
What is a Kernel? or mask is a small matrix used for blurring, sharpening, embossing, edge detection, and more.

A Kernel is a matrix, which is slid across the image and multiplied with the input such that the output
is enhanced in a certain desirable manner. Each kernel has a different value for different kind of effects
that we want to apply to an image.

In Image processing, we use the convolution operation to extract the features from the images which
can le later used for further processing especially in Convolution Neural Network (CNN), about which
we will study later in the chapter.

In this process, we overlap the centre of the image with the centre of the kernel to obtain the
convolution output. In the process of doing it, the output image becomes smaller as the overlapping
is done at the edge row and column of the image. What if we want the output image to be of exact
size of the input image, how can we achieve this?

To achieve this, we need to extend the edge values out by one in the original image while overlapping
the centres and performing the convolution. This will help us keep the input and output image of the
same size. While extending the edges, the pixel values are considered as zero.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Let’s try
In this section we will try performing the convolution operator on paper to understand how it works.
Fill the blank places of the output images by performing the convolution operation.

150 0 255 240 190 25 89 255

100 179 25 0 200 255 67 100

155 146 13 20 0 12 45 0

100 175 0 25 25 15 0 0
-1 0 -1
120 156 255 0 78 56 23 0 0 -1 0
-1 0 -1
115 113 25 90 0 80 56 155

135 190 115 116 178 0 145 165

123 255 255 0 255 255 255 0

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Write Your Output Here :

Summary
1. Convolution is a common tool used for image editing.
2. It is an element wise multiplication of an image and a kernel to get the desired output.
3. In computer vision application, it is used in Convolutional Neural Network (CNN) to extract
image features.

Convolution Neural Networks (CNN)

Introduction
In class 9, you studied about the concepts of Neural Network. You played a neural network game to
understand how a neural network works.

Let’s recall

What is a Neural Network?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Fill in the names of different layers of Neural Network.

Did you get the answers right? In this section, we are going to study about one such neural network
which is Convolutional Neural Network (CNN). Many of the current computer vision applications use
a powerful neural network called the convolutional neural network.

What is a Convolutional Neural Network?

A Convolutional Neural Network (CNN) is a Deep Learning algorithm which can take in an input image,
assign importance (learnable weights and biases) to various aspects/objects in the image and be able
to differentiate one from the other.

The process of deploying a CNN is as follows:

* Images shown here are the property of individual organisations and are used here for reference purpose only.
In the above diagram, we give an input image, which is then processed through a CNN and then gives
prediction on the basis of the label given in the particular dataset.

The different layers of a Convolutional Neural Network (CNN) is as follows:

A convolutional neural network consists of the following layers:

1) Convolution Layer
2) Rectified linear Unit (ReLU)
3) Pooling Layer
4) Fully Connected Layer

Convolution Layer

It is the first layer of a CNN. The objective of the Convolution Operation is to extract the high-level
features such as edges, from the input image. CNN need not be limited to only one Convolutional
Layer. Conventionally, the first Convolution Layer is responsible for capturing the Low-Level features
such as edges, colour, gradient orientation, etc. With added layers, the architecture adapts to the
High-Level features as well, giving us a network which has the wholesome understanding of images in
the dataset.

It uses convolution operation on the images. In the convolution layer, there are several kernels that
are used to produce several features. The output of this layer is called the feature map. A feature map
is also called the activation map. We can use these terms interchangeably.
There’s several uses we derive from the feature map:
• We reduce the image size so that it can be processed more efficiently.
• We only focus on the features of the image that can help us in processing the image further.
For example, you might only need to recognize someone’s eyes, nose and mouth to recognize the
person. You might not need to see the whole face.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Rectified Linear Unit Function

The next layer in the Convolution Neural Network is the Rectified Linear Unit function or the ReLU
layer. After we get the feature map, it is then passed onto the ReLU layer. This layer simply gets rid of
all the negative numbers in the feature map and lets the positive number stay as it is.

The process of passing it to the ReLU layer introduces non – linearity in the feature map. Let us see it
through a graph.

If we see the two graphs side by side, the one on the left is a linear graph. This graph when passed
through the ReLU layer, gives the one on the right. The ReLU graph starts with a horizontal straight
line and then increases linearly as it reaches a positive number.

Now the question arises, why do we pass the feature map to the ReLU layer? it is to make the colour
change more obvious and more abrupt?

* Images shown here are the property of individual organisations and are used here for reference purpose only.
As shown in the above convolved image, there is a smooth grey gradient change from black to white.
After applying the ReLu function, we can see a more abrupt change in color which makes the edges
more obvious which acts as a better feature for the further layers in a CNN as it enhances the
activation layer.

Pooling Layer

Similar to the Convolutional Layer, the Pooling layer is responsible for reducing the spatial size of the
Convolved Feature while still retaining the important features.

There are two types of pooling which can be performed on an image.

1) Max Pooling : Max Pooling returns the maximum value from the portion of the image covered
by the Kernel.
2) Average Pooling: Max Pooling returns the maximum value from the portion of the image
covered by the Kernel.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
The pooling layer is an important layer in the CNN as it performs a series of tasks which are as
follows :

1) Makes the image smaller and more manageable

2) Makes the image more resistant to small transformations, distortions and translations in the
input image.

A small difference in input image will create very similar pooled image.

Fully Connected Layer

The final layer in the CNN is the Fully Connected Layer (FCP). The objective of a fully connected layer
is to take the results of the convolution/pooling process and use them to classify the image into a label
(in a simple classification example).

The output of convolution/pooling is flattened into a single vector of values, each representing a
probability that a certain feature belongs to a label. For example, if the image is of a cat, features
representing things like whiskers or fur should have high probabilities for the label “cat”.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Let’s Summarize:

Write the whole process of how a CNN works on the basis of the above diagram.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

* Images shown here are the property of individual organisations and are used here for reference purpose only.