0% found this document useful (0 votes)

129 views68 pages

Ai Class 10 - Merged

This document provides an overview of key concepts in artificial intelligence (AI) including what intelligence is, how machines can be artificially intelligent, applications of AI, and what AI is not. It then discusses the basics of AI including relationships between AI, machine learning, and deep learning. It introduces common AI domains like data science, computer vision, and natural language processing. Finally, it discusses the AI project cycle and some Python concepts needed for AI like packages, basic statistics, data visualization, and classification models.

Uploaded by

Ananta Sri

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

129 views68 pages

Ai Class 10 - Merged

Uploaded by

Ananta Sri

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 68

Introduction to AI: Foundational Concepts ........................................................................................ 9

What is Intelligence? ....................................................................................................................... 9
Decision Making ............................................................................................................................ 12
How do you make decisions?.................................................................................................... 12
Make Your Choices! .................................................................................................................. 12
What is Artificial Intelligence? ...................................................................................................... 14
How do machines become Artificially Intelligent? ....................................................................... 14
Applications of Artificial Intelligence around us ........................................................................... 15
What is not AI? .............................................................................................................................. 16
Introduction to AI: Basics of AI ......................................................................................................... 18
AI, ML & DL.................................................................................................................................... 20
Introduction to AI Domains........................................................................................................... 21
Data Sciences ............................................................................................................................ 21
Computer Vision ....................................................................................................................... 21
Natural Language Processing .................................................................................................... 22
AI Ethics ......................................................................................................................................... 23
Moral Issues: Self-Driving Cars.................................................................................................. 23
Data Privacy .............................................................................................................................. 24
AI Bias ........................................................................................................................................ 26
AI Access ................................................................................................................................... 27
AI Project Cycle ................................................................................................................................. 29
Introduction .................................................................................................................................. 29
Problem Scoping ....................................................................................................................... 30
Data Acquisition ........................................................................................................................ 34
Data Exploration ....................................................................................................................... 35
Modelling .................................................................................................................................. 36
Learning Based Approach ......................................................................................................... 37
Evaluation ................................................................................................................................. 39
Neural Networks ........................................................................................................................... 40
Advance Python ................................................................................................................................ 42
Recap............................................................................................................................................. 42
Recap 1: Jupyter Notebook ....................................................................................................... 42
Introduction to Virtual Environments ....................................................................................... 43
Recap 2: Introduction to Python ............................................................................................... 47
Applications of Python .............................................................................................................. 48
Recap 3: Python Basics.............................................................................................................. 48
Python Packages ........................................................................................................................... 52
Data Sciences .................................................................................................................................... 54
Introduction .................................................................................................................................. 54
Applications of Data Sciences ....................................................................................................... 55
Getting Started.............................................................................................................................. 57
Revisiting AI Project Cycle ......................................................................................................... 57
Data Collection .......................................................................................................................... 62
Data Access ............................................................................................................................... 63
Basic Statistics with Python .......................................................................................................... 66
Data Visualisation ......................................................................................................................... 67
Data Sciences: Classification Model .............................................................................................. 71
Personality Prediction ............................................................................................................... 71
K-Nearest Neighbour: Explained ............................................................................................... 72
Computer Vision ............................................................................................................................... 75
Introduction .................................................................................................................................. 75
Applications of Computer Vision .................................................................................................. 76
Computer Vision: Getting Started................................................................................................. 78
Computer Vision Tasks .................................................................................................................. 78
Classification ............................................................................................................................. 78
Classification + Localisation ...................................................................................................... 78
Object Detection ....................................................................................................................... 78
Instance Segmentation ............................................................................................................. 78
Basics of Images ........................................................................................................................ 79
Basics of Pixels .......................................................................................................................... 79
Image Features.............................................................................................................................. 84
Introduction to OpenCV ................................................................................................................ 85
Convolution ................................................................................................................................... 86
Convolution : Explained ............................................................................................................ 88
Convolution Neural Networks (CNN) ............................................................................................ 91
Introduction .............................................................................................................................. 91
What is a Convolutional Neural Network ?............................................................................... 92
Convolution Layer ..................................................................................................................... 93
Rectified Linear Unit Function .................................................................................................. 94
Pooling Layer ............................................................................................................................. 95
Fully Connected Layer ............................................................................................................... 96
Natural Language Processing ............................................................................................................ 99
Introduction .................................................................................................................................. 99
Applications of Natural Language Processing ............................................................................. 100
Natural Language Processing: Getting Started ........................................................................... 101
Revisiting the AI Project Cycle................................................................................................. 101
Chatbots ...................................................................................................................................... 104
Human Language VS Computer Language .................................................................................. 105
Arrangement of the words and meaning................................................................................ 106
Multiple Meanings of a word .................................................................................................. 107
Perfect Syntax, no Meaning .................................................................................................... 107
Data Processing ........................................................................................................................... 108
Text Normalisation .................................................................................................................. 108
Bag of Words ........................................................................................................................... 112
TFIDF: Term Frequency & Inverse Document Frequency ....................................................... 114
Applications of TFIDF .................................................................................................................. 118
DIY – Do It Yourself! .................................................................................................................... 118
Evaluation ....................................................................................................................................... 119
Introduction ................................................................................................................................ 119
What is evaluation? .................................................................................................................... 119
Model Evaluation Terminologies ................................................................................................ 119
The Scenario............................................................................................................................ 119
Confusion matrix ......................................................................................................................... 122
Evaluation Methods .................................................................................................................... 123
Accuracy .................................................................................................................................. 123
Precision .................................................................................................................................. 124
Recall ....................................................................................................................................... 125
Which Metric is Important? .................................................................................................... 126
F1 Score ................................................................................................................................... 127
Introduction to AI: Foundational Concepts
What is Intelligence?
Humans have been developing machines which can make their lives easier. Machines are made with
an intent of accomplishing tasks which are either too tedious for humans or are time consuming.
Hence, machines help us by working for us, thereby sharing our load and making it easier for us to
fulfil such goals.

Life without machines today is unimaginable, and because of this, humans have been putting efforts
into making them even more sophisticated and smart. As a result, we are surrounded by smart devices
and gadgets like smartphones, smartwatches, smart TV, etc. But what makes them smart?

For example, how is a smartphone today different from the telephones we had in the last century?

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Today’s phones can do much more than just call-up
people. They can help us in navigating, recommend
which songs we should listen to or which movies we
should watch according to our own likes and
dislikes. Our phones can help us connect with like-
minded people, make our selfies fun with face
filters, help us maintain a record of our health and
fitness and a lot more. These drastic technological
advancements lead us to recognize one key
concept: the concept of Artificial Intelligence.

What is Artificial Intelligence anyway? Well, the answer lies in the term itself. If we break up this term
up, we get the words “Artificial” and “Intelligence”. Artificial is something which is man-made, which
does not occur naturally. But what about Intelligence, how do we define that?

Humans are said to be an intelligent species, so what is it that makes us intelligent?

According to researchers, intelligence is the ‘ability to perceive or infer information, and to retain it as
knowledge to be applied towards adaptive behaviours within an environment or context.’

If we try to define intelligence with the help of its traits, these are the abilities that are involved in
intelligence:

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Let us define each term mentioned above to get a proper understanding:

Mathematical •A person's ability to regulate, measure, and understand numerical

Logical Reasoning symbols, abstraction and logic.

Linguistic •Language processing skills both in terms of understanding or

Intelligence implementation in writing or verbally.

Spatial Visual •It is defined as the ability to perceive the visual world and the
Intelligence relationship of one object to another.

Kineasthetic •Ability that is related to how a person uses his limbs in a skilled
Intelligence manilr.

Musical •As the name suggests, this intelligence is about a person's ability to
Intelligence recognize and create sounds, rhythms, and sound patterns.

Intrapersonal •Describes how high the level of self-awareness someone has is.
Intelligence Starting from realizing weakness, strength, to his own feelings.

Existential •An additional category of intelligence relating to religious and

Intelligence spiritual awareness.

Naturalist •An additional category of intelligence relating to the

Intelligence ability to process information on the environment around us.

Interpersonal •Interpersonal intelligence is the ability to communicate with others

intelligence by understanding other people's feelings & influence of the person.

But even though one is more skilled in intelligence than the other, it should be noted that in fact all humans have all 9 of
these intelligences only at different levels. One might be an expert at painting, while the other might be an expert in
mathematical calculations. One is a musician, the other is an expert dancer.
In other words, we may define intelligence as:

• Ability to interact with the real world

o To perceive, understand and act
▪ Example: Speech Recognition – Understanding and synthesis
▪ Example: Image Recognition
▪ Example: Ability to take action: to have an effect
• Reasoning and planning
o Modelling the external world, given input
▪ Solving new problems, planning and making decisions
▪ Ability to deal with unexpected problems, uncertainties
• Learning and adaptation
o Continuous learning and adapting graph
▪ Our internal models are always being updated
▪ Example: Baby learning to categorize and recognise animals

For example, if someone starts talking to us, we know how to keep the conversation going. We can
understand what people mean and can reply in the same way. When we are hungry, we can come up
with various options on what to eat depending upon the food we have at our homes. When we read
something, we are able to understand its meaning and answer anything regarding it.

While understanding the term intelligence, it must be noticed that decision making comprises of a
crucial part of intelligence. Let us delve deeper into it.

Decision Making

You’re trapped. All the doors seem to have started shrinking and only one of them leads you out.
Which door would you pick?

How do you make decisions?

The basis of decision making depends upon the availability of information and how we experience and
understand it. For the purposes of this article, ‘information’ includes our past experience, intuition,
knowledge, and self-awareness.

We can’t make “good” decisions without information because then we have to deal with unknown
factors and face uncertainty, which leads us to make wild guesses, flipping coins, or rolling a dice.
Having knowledge, experience, or insights given a certain situation, helps us visualize what the
outcomes could be. and how we can achieve/avoid those outcomes.

Make Your Choices!

Scenario 1
You are locked inside a room with 3 doors to move out of the locked room and you need to find a safe
door to get your way out. Behind the 1st door is a lake with a deadly shark. The 2nd door has a mad
psychopath ready to kill with a weapon and the third one has a lion that has not eaten since the last 2
months.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Which door would you choose? and Why?

The answer is gate number 3. The reason being that since the lion has not eaten for 2 months, he
wouldn't have survived till now and would already be dead . This makes going out from gate 3 the
correct option.

Scenario 2
Aarti invited four of her friends to her House.. They hadn't seen each other in a long time, so they
chatted all night long and had a good time. In the morning, two of the friends Aarti had invited, died.
The police arrived at the house and found that both the friends were poisoned and that the poison
was in the strawberry pie. The three surviving friends told the police that they hadn't eaten the pie.
The police asked," Why didn’t you eat the pie ?". Shiv said, " I am allergic to strawberries.". Seema
said, " I am on a diet." And Aarti said, "I ate too many strawberries while cooking the pie, I just didn't
want anymore."

The policemen looked at the pictures of the party and immediately identified the murderer.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Look at the picture and identify who is the murderer? Also state why do you think this is the murderer?

The answer is Seema, can you guess how the police could tell? It’s because she said she is on a diet
and in the picture, she is eating a burger and fries which means she lied.

The above scenarios show that it’s the information which helps humans take good decisions.

What is Artificial Intelligence?

When a machine possesses the ability to mimic human
traits, i.e., make decisions, predict the future, learn and
improve on its own, it is said to have artificial
intelligence.
In other words, you can say that a machine is artificially
intelligent when it can accomplish tasks by itself -
collect data, understand it, analyse it, learn from it, and
improve it. You will get to know more about it in the
next unit.
But, what makes a machine intelligent?

How do machines become Artificially Intelligent?

Humans become more and more intelligent with time as they gain experiences during their lives.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
For example, in elementary school, we learn about alphabets and eventually we move ahead to
making words with them. As we grow, we become more and more fluent in the language as we keep
learning new words and use them in our conversations.

Another example is how we learn walking.

Initially a baby struggles to walk. He takes
help from others while learning how to
walk and once he knows it, he keeps on
upgrading it by learning how to run, jump,
etc.

Similarly, machines also become

intelligent once they are trained with
some information which helps them
achieve their tasks. AI machines also keep
updating their knowledge to optimise
their output.

Applications of Artificial Intelligence around us

Whether we notice it or not, we are surrounded by machines that work on AI. They are becoming a
crucial part of our everyday life and provide us with an ease of having even some of the most
complicated and time-consuming tasks being done at the touch of a button or by the simple use of a
sensor.

Every now and then, we surf the internet for things on Google
without realizing how efficiently Google always responds to us
with accurate answers. Not only does it come up with results
to our search in a matter of seconds, it also suggests and auto-
corrects our typed sentences.

We nowadays have pocket assistants that can

do a lot of tasks at just one command. Alexa,
Google Assistant, Cortana, Siri are some very
common examples of the voice assistants which
are a major part of our digital devices.

To help us navigate to places, apps like UBER and Google Maps come in haman.
Thus, one no longer needs to stop repeatedly to ask for directions.

AI has completely enhanced the gaming experience for its users.

A lot of games nowadays are backed up with AI which helps in
enhancing the graphics, come up with new difficulty levels,
encourage gamers, etc.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
AI has not only made our lives easier but has also been
taking care of our habits, likes, and dislikes. This is why
platforms like Netflix, Amazon, Spotify, YouTube etc.
show us recommendations on the basis of what we
like.

Well, the recommendations are not just limited to

our preferences, they even cater to our needs of
connecting with friends on social media platforms
with apps like Facebook and Instagram. They also
send us customized notifications about our online
shopping details, auto-create playlists according
to our requests and so on. Taking selfies was never
this fun as Snapchat filters make them look so
cool.

This isn’t all. AI is also being used to monitor

our health. A lot of chatbots and other health
apps are available, which continuously
monitor the physical and mental health of its
users.

These applications are not limited to smart devices but

also vary to humanoids like Sophia, the very first
humanoid robot sophisticated enough to get
citizenship, biometric security systems like the face
locks we have in our phones, real-time language
translators, weather forecasts, and whatnot! This list is
huge, and this module will go on forever if we keep
tabulating them. So, take some time, discuss with a
friend and identify more and more AI applications
around you!

What is not AI?

Since we have a lot of different technologies which exist around us in today’s time, it is very
common for us to misunderstand any other technology as AI. That is why, we need to have a
clear distinction between what is AI and what is not.
As we discussed earlier, any machine that has been trained with data and can make
decisions/predictions on its own can be termed as AI. Here, the term ‘training’ is important.

A fully automatic washing machine can work on its own, but it requires human
intervention to select the parameters of washing and to do the necessary preparation for
it to function correctly before each wash, which makes it an example of automation, not
AI.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
An air conditioner can be turned on and off remotely with the help of internet but still
needs a human touch. This is an example of Internet of Things (IoT). Also, every now and
then we get to know about robots which might follow a path or maybe can avoid
obstacles but need to be primed accordingly each time.

We also get to see a lot of projects which can automate our surroundings with the
help of sensors. Here too, since the bot or the automation machine is not trained with
any data, it does not count as AI.

Also, it would be valid to say that not all the devices which are termed as "smart" are AI-enabled. For
example, a TV does not become AI-enabled if it is a smart one, it gets the power of AI when it is able
to think and process on its own.

Just as humans learn how to walk and then improve this skill with the help of their experiences, an AI
machine too gets trained first on the training data and then optimises itself according to its own
experiences which makes AI different from any other technological device/machine.

But well, surely these other technologies too can be integrated with AI to provide the users with a
much better and immersive experience!

Robotics and AI can definitely open the doors to humanoids and self-driving cars, AI when merged
with Internet of things can give rise to cloud computing of data and remote access of AI tools,
automation along with AI can help in achieving voice automated homes and so on. Such integrations
can help us get the best of both worlds!

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Introduction to AI: Basics of AI
As discussed in the last chapter, Artificial Intelligence has always been a term which intrigues people
all over the world. Various organisations have coined their own versions of defining Artificial
Intelligence. Some of them are mentioned below:

NITI Aayog: National Strategy for Artificial Intelligence

AI refers to the ability of machines to perform cognitive tasks like thinking, perceiving, learning,
problem solving and decision making. Initially conceived as a technology that could mimic human
intelligence, AI has evolved in ways that far exceed its original conception. With incredible advances
made in data collection, processing and computation power, intelligent systems can now be deployed
to take over a variety of tasks, enable connectivity and enhance productivity.

World Economic Forum

Artificial intelligence (AI) is the software engine that drives the Fourth Industrial Revolution. Its impact
can already be seen in homes, businesses and political processes. In its embodied form of robots, it
will soon be driving cars, stocking warehouses and caring for the young and elderly. It holds the
promise of solving some of the most pressing issues facing society, but also presents challenges such
as inscrutable “black box” algorithms, unethical use of data and potential job displacement. As rapid
advances in machine learning (ML) increase the scope and scale of AI’s deployment across all aspects
of daily life, and as the technology itself can learn and change on its own, multi-stakeholder
collaboration is required to optimize accountability, transparency, privacy and impartiality to create
trust.

European Artificial Intelligence (AI) leadership, the path for an integrated vision
AI is not a well-defined technology and no universally agreed definition exists. It is rather a cover term
for techniques associated with data analysis and pattern recognition. AI is not a new technology,
having existed since the 1950s. While some markets, sectors and individual businesses are more
advanced than others, AI is still at a relatively early stage of development, so that the range of
potential applications, and the quality of most existing applications, have ample margins left for
further development and improvement.

Encyclopaedia Britannica
Artificial intelligence (AI), is the ability of a digital computer or computer-controlled robot to
perform tasks commonly associated with intelligent beings. The term is frequently applied to the
project of developing systems endowed with the intellectual processes characteristic of humans, such
as the ability to reason, discover meaning, generalize, or learn from past experience.
As you can see, Artificial Intelligence is a vast domain. Everyone looks at AI in a different way according
to their mindset. Now, according to your knowledge of AI, start filling the KWLH chart:

K • What I Know?

W • What I Want to know?

L • What have I learned?

H • How I learnt this?

What do you know about Artificial Intelligence (AI)?

What do you want to know about AI?

What have you learnt about AI?

How have you learnt this about AI?

AI is a form of Intelligence; a type of technology and a field of study.

AI theory and development of computer systems (both machines and software) enables machines to
perform tasks that normally require human intelligence.

Artificial Intelligence covers a broad range of domains and applications and is expected to impact every
field in the future. Overall, its core idea is building machines and algorithms which are capable of
performing computational tasks that would otherwise require human like brain functions.

AI, ML & DL
As you have been progressing towards building AI readiness, you must have come across a very
common dilemma between Artificial Intelligence (AI) and Machine Learning (ML). Many times, these
terms are used interchangeably but are they the same? Is there no difference in Machine Learning
and Artificial Intelligence? Is Deep Learning (DL) Also Artificial Intelligence? What exactly is Deep
Learning? Let us see.

Artificial Intelligence (AI)

Refers to any technique that enables computers to mimic human intelligence. It gives the ability to
machines to recognize a human’s face; to move and manipulate objects; to understand the voice
commands by humans, and also do other tasks. The AI-enabled machines think algorithmically and
execute what they have been asked for intelligently.

Machine Learning (ML)

It is a subset of Artificial Intelligence which enables machines to improve at tasks with experience
(data). The intention of Machine Learning is to enable machines to learn by themselves using the
provided data and make accurate Predictions/ Decisions.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Deep Learning (DL)
It enables software to train itself to perform tasks with vast amounts of data. In Deep Learning, the
machine is trained with huge amounts of data which helps it in training itself around the data. Such
machines are intelligent enough to develop algorithms for themselves. Deep Learning is the most
advanced form of Artificial Intelligence out of these three. Then comes Machine Learning which is
intermediately intelligent and Artificial Intelligence covers all the concepts and algorithms which, in
some way or the other mimic human intelligence.

There are a lot of applications of AI out of which few are those which come under ML out of which
very few can be labelled as DL. Therefore, Machine Learning (ML) and Deep Learning (DL) are part of
Artificial Intelligence (AI), but not everything that is Machine learning will be Deep learning.

Introduction to AI Domains
Artificial Intelligence becomes intelligent according to the training which it gets. For training, the
machine is fed with datasets. According to the applications for which the AI algorithm is being
developed, the data which is fed into it changes. With respect to the type of data fed in the AI
model, AI models can be broadly categorised into three domains:

Data Sciences Computer Vision Natural Language Processing

Data Sciences
Data sciences is a domain of AI related to data systems and processes, in which the system collects
numerous data, maintains data sets and derives meaning/sense out of them.

The information extracted through data science can be used to make a decision about it.

Example of Data Science

Price Comparison Websites

These websites are being driven by lots and lots of data. If you have
ever used these websites, you would know, the convenience of
comparing the price of a product from multiple vendors at one
place. PriceGrabber, PriceRunner, Junglee, Shopzilla, DealTime are
some examples of price comparison websites. Now a days, price
comparison website can be found in almost every domain such as
technology, hospitality, automobiles, durables, apparels etc.

Computer Vision
Computer Vision, abbreviated as CV, is a domain of AI that depicts the capability of a machine to get
and analyse visual information and afterwards predict some decisions about it. The entire process
involves image acquiring, screening, analysing, identifying and extracting information. This extensive
processing helps computers to understand any visual content and act on it accordingly. In computer
vision, Input to machines can be photographs, videos and pictures from thermal or infrared sensors,
indicators and different sources.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Computer vision related projects translate digital visual data into descriptions. This data is then turned
into computer-readable language to aid the decision-making process. The main objective of this
domain of AI is to teach machines to collect information from pixels.

Examples of Computer Vision

Self-Driving cars/ Automatic Cars
CV systems scan live objects and analyse them, based on whether
the car decides to keep running or to stop.

Face Lock in Smartphones

Smartphones nowadays come with the feature of face locks in

which the smartphone’s owner can set up his/her face as an
unlocking mechanism for it. The front camera detects and captures
the face and saves its features during initiation. Next time onwards,
whenever the features match, the phone is unlocked.

Natural Language Processing

Natural Language Processing, abbreviated as NLP, is a branch of artificial intelligence that deals with
the interaction between computers and humans using the natural language. Natural language refers
to language that is spoken and written by people, and natural language processing (NLP) attempts to
extract information from the spoken and written word using algorithms.

The ultimate objective of NLP is to read, decipher, understand, and make sense of the human languages
in a manilr that is valuable.

Examples of Natural Language Processing

Email filters
Email filters are one of the most basic and
initial applications of NLP online. It started
out with spam filters, uncovering certain
words or phrases that signal a spam
message.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Smart assistants
Smart assistants like Apple’s Siri and Amazon’s Alexa recognize
patterns in speech, then infer meaning and provide a useful
response.

AI Ethics
Nowadays, we are moving from the Information era to Artificial Intelligence era. Now we do not use
data or information, but the intelligence collected from the data to build solutions. These solutions
can even recommend the next TV show or movies you should watch on Netflix.

We can proudly say that India is leading in the AI usage trends, so we need to keep aspects relating to
ethical practices in mind while developing solutions using AI. Let us understand some of the ethical
concerns in detail.

Moral Issues: Self-Driving Cars

Scenario 1:
Let us imagine that we are in year 2030. Self-Driving cars which are just a concept in today’s time are
now on roads. People like us are buying them for ease and are using it for our daily transits. Of-course
because of all the features which this car has, it is expensive. Now, let us assume, one day your father
is going to office in his self-driving car. He is sitting in the back seat as the car is driving itself. Suddenly,
a small boy comes in front of this car. The incident was so sudden that the car is only able to make
either of the two choices:

1. Go straight and hit the boy who has come in front of the car and injure him severely.

2. Take a sharp right turn to save the boy and smash the car into a metal pole thus damaging the car
as well as injuring the person sitting in it.

With the help of this scenario, we need to understand that the developer of the car goes through all
such dilemmas while developing the car’s algorithm. Thus, here the morality of the developer gets
transferred into the machine as what according to him/her is right would have a higher priority and
hence would be the selection made by the machine.

If you were in the place of this developer and if there was no other alternative to the situation, which
one of the two would you prioritise and why?

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Scenario 2:
Let us now assume that the car has hit the boy who came in front of it. Considering this as an accident,
who should be held responsible for it? Why?

1. The person who bought this car

2. The Manufacturing Company
3. The developer who developed the car’s algorithm
4. The boy who came in front of the car and got severely injured

Here, the choices might differ from person to person and one must understand that nobody is wrong
in this case. Every person has a different perspective and hence he/she takes decisions according to
their moralities.

Data Privacy

The world of Artificial Intelligence revolves around Data. Every company whether small or big is mining
data from as many sources as possible. More than 70% of the data collected till now has been collected
in the last 3 years which shows how important data has become in recent times. It is not wrongly said
that Data is the new gold. This makes us think:

Where do we collect data from?

Why do we need to collect data?

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
One of the major sources of data for many major companies is the device which all of us have in our
hands all the time: Smartphones. Smartphones have nowadays become an integral part of our lives.
Most of us use smartphones more than we interact with people around us. Smartphones in today’s
era provide us with a lot of facilities and features which have made our lives easier. Feeling hungry?
Order food online. Want to shop but don’t have time to go out? Go shopping online. From booking
tickets to watching our favourite shows, everything is available in this one small box loaded with
technology.

Another feature of smartphones nowadays is that they provide us with customised recommendations
and notifications according to our choices. Let us understand this with the help of some examples:

1. When you are talking to your friend on a mobile network or on an app like WhatsApp. You tell
your friend that you wish to buy new shoes and are looking for suggestions from him/her. You
discuss about shoes and that is it. After some time, the online shopping websites start giving
you notifications to buy shoes! They start recommending some of their products and urge you
to you buy some.
2. If you search on Google for a trip to Kerala or any other destination, just after the search, all
the apps on your phone which support advertisements, will start sending messages about
packages that you can buy for the trip.
3. Even when you are not using your phone and talking to a person face-to-face about a book
you’ve read recently while the phone is kept in a locked mode nearby, the phone will end up
giving notifications about similar books or messages about the same book once you operate
it.

In all such examples, how does the smartphone get to know about the discussions and thoughts that
you have? Remember whenever you download an app and install it, it asks you for several permissions
to access your phone’s data in different ways. If you do not allow the app these permissions, you
normally cannot access it. And to access the app and make use of it, we sometimes don’t even give it
a thought and allow the app to get all the permissions that it wants. Hence every now and then, the
app has the permission to access various sensors which are there in your smartphone and gather data
about you and your surroundings. We forget that the smartphone which we use is a box full of sensors
which are powered all the time while the phone is switched on.

This leads us to a crucial question: Are we okay with sharing our data with the external world?

Why do these apps collect data?

We need to understand that the data which is collected by various applications is ethical as the
smartphone users agree to it (by clicking on allow when it asks for permission and by agreeing to all
the terms and conditions). But at the same time if one does not want to share his/her data with
anyone, he/she can opt for alternative applications which are of similar usage and keep your data
private. For example, an alternative to WhatsApp is the Telegram app which does not collect any data
from us. But since WhatsApp is more popular and used by the crowd, people go for it without thinking
twice.

AI Bias

Another aspect to AI Ethics is bias. Everyone has a bias of their own no matter how much one tries to
be unbiased, we in some way or the other have our own biases even towards smaller things. Biases
are not negative all the time. Sometimes, it is required to have a bias to control a situation and keep
things working.

When we talk about a machine, we know that it is artificial and cannot think on its own. It can have
intelligence, but we cannot expect a machine to have any biases of its own. Any bias can transfer from
the developer to the machine while the algorithm is being developed. Let us look at some of the
examples:

1. Majorly, all the virtual assistants have a female voice. It is only now that some companies have
understood this bias and have started giving options for male voices but since the virtual assistants
came into practice, female voices are always preferred for them over any other voice. Can you think
of some reasons for this?

2. If you search on Google for salons, the first few searches are mostly for female salons. This is based
on the assumption that if a person is searching fora salon, in all probability it would be a female. Do
you think this is a bias? If yes, then is it a Negative bias or Positive one?

Various other biases are also found in various systems which are not thought up by the machine but
have got transferred from the developer intentionally or unintentionally.
AI Access

Since Artificial Intelligence is still a budding technology, not everyone has the opportunity to access it.
The people who can afford AI enabled devices make the most of it while others who cannot are left
behind. Because of this, a gap has emerged between these two classes of people and it gets widened
with the rapid advancement of technology. Let us understand this with the help of some examples:

AI creates unemployment
AI is making people’s lives easier. Most of the things nowadays are done in just a few clicks. In no time
AI will manage to be able to do all the laborious tasks which we humans have been doing since long.
Maybe in the coming years, AI enabled machines will replace all the people who work as labourers.
This may start an era of mass unemployment where people having little or no skills may be left without
jobs and others who keep up with their skills according to what is required, will flourish.

This brings us to a crossroads. On one hand where AI is advancing and improving the lives of people
by working for them and doing some of their tasks, the other hand points towards the lives of people
who are dependent on laborious jobs and are not skilled to do anything else.

Should AI replace laborious jobs? Is there an alternative for major unemployment?

Should AI not replace laborious jobs? Will the lives of people improve if they keep on being unskilled?

Here, we need to understand that to overcome such an issue, one needs to be open to changes. As
technology is advancing with time, humans need to make sure that they are a step ahead and
understand this technology with its pros and cons.

AI for kids
As we all can see, kids nowadays are smart enough to understand technology from a very early age.
As their thinking capabilities increase, they start becoming techno-savvy and eventually they learn
everything more easily than an adult. But should technology be given to children so young?

Consider this: A young boy in class 3 has got some Maths homework to finish. He is sitting at a table
which has the Google chat bot - Alexa on it, and he is struggling with his homework. Soon, he starts
asking Alexa to answer all his questions. Alexa replies with answers and the boy simply writes them
down in his notebook.

While this scenario seems funny, it still has some concerns related to it. On one hand where it is good
that the boy knows how to use technology effectively, on the other hand he uses it to complete his
homework without really learning anything since he is not applying his brain to solve the Math
problems. So, while he is smart, he might not be getting educated properly.

Is it ethical to let the boy use technology to help in this manilr?

Conclusion
Despite AI’s promises to bring forth new opportunities, there are certain associated risks that need to
be mitigated appropriately and effectively. To give a better perspective, the ecosystem and the socio-
technical environment in which the AI systems are embedded needs to be more trustworthy.
AI Project Cycle
In this chapter, we will revisit the concept of AI Project Cycle.

Introduction
Let us assume that you have to make a greeting card for your mother as it is her birthday. You are very
excited about it and have thought of many ideas to execute the same. Let us look at some of the steps
which you might take to accomplish this task:

1. Look for some cool greeting card ideas from different sources. You might go online and
checkout some videos or you may ask someone who has knowledge about it.
2. After finalising the design, you would make a list of things that are required to make this card.
3. You will check if you have the material with you or not. If not, you could go and get all the
items required, ready for use.
4. Once you have everything with you, you would start making the card.
5. If you make a mistake in the card somewhere which cannot be rectified, you will discard it and
start remaking it.
6. Once the greeting card is made, you would gift it to your mother.
Are these steps relatable?

__________________________________________________________________________________
__________________________________________________________________________________

Do you think your steps might differ? If so, write them down!

These steps show how we plan to execute the tasks around us. Consciously or Subconsciously our
mind makes up plans for every task which we have to accomplish which is why things become clearer
in our mind. Similarly, if we have to develop an AI project, the AI Project Cycle provides us with an
appropriate framework which can lead us towards the goal. The AI Project Cycle mainly has 5 stages:

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Starting with Problem Scoping, you set the goal for your AI project by stating the problem which you
wish to solve with it. Under problem scoping, we look at various parameters which affect the problem
we wish to solve so that the picture becomes clearer.

To proceed,

● You need to acquire data which will become the base of your project as it will help you in
understanding what the parameters that are related to problem scoping are.
● You go for data acquisition by collecting data from various reliable and authentic sources.
Since the data you collect would be in large quantities, you can try to give it a visual image of
different types of representations like graphs, databases, flow charts, maps, etc. This makes
it easier for you to interpret the patterns which your acquired data follows.
● After exploring the patterns, you can decide upon the type of model you would build to
achieve the goal. For this, you can research online and select various models which give a
suitable output.
● You can test the selected models and figure out which is the most efficient one.
● The most efficient model is now the base of your AI project and you can develop your
algorithm around it.
● Once the modelling is complete, you now need to test your model on some newly fetched
data. The results will help you in evaluating your model and improving it.
● Finally, after evaluation, the project cycle is now complete and what you get is your AI project.
Let us understand each stage of the AI Project Cycle in detail.

Problem Scoping
It is a fact that we are surrounded by problems. They could be small or big, sometimes ignored or
sometimes even critical. Many times, we become so used to a problem that it becomes a part of our
life. Identifying such a problem and having a vision to solve it, is what Problem Scoping is about. A lot
of times we are unable to observe any problem in our surroundings. In that case, we can take a look
at the Sustainable Development Goals. 17 goals have been announced by the United nations which
are termed as the Sustainable Development Goals. The aim is to achieve these goals by the end of
2030. A pledge to do so has been taken by all the member nations of the UN.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Here are the 17 SDGs. Let’s take a look:

As you can see, many goals correspond to the problems which we might observe around us too. One
should look for such problems and try to solve them as this would make many lives better and help
our country achieve these goals.

Scoping a problem is not that easy as we need to have a deeper understanding around it so that the
picture becomes clearer while we are working to solve it. Hence, we use the 4Ws Problem Canvas to
help us out.

4Ws Problem Canvas

The 4Ws Problem canvas helps in identifying the key elements related to the problem.

Who? What? Where? Why?

Let us go through each of the blocks one by one.

Who?
The “Who” block helps in analysing the people getting affected directly or indirectly due to it. Under
this, we find out who the ‘Stakeholders’ to this problem are and what we know about them.
Stakeholders are the people who face this problem and would be benefitted with the solution. Here is
the Who Canvas:

* Images shown here are the property of individual organisations and are used here for reference purpose only.
What?
Under the “What” block, you need to look into what you have on hand. At this stage, you need to
determine the nature of the problem. What is the problem and how do you know that it is a problem?
Under this block, you also gather evidence to prove that the problem you have selected actually exists.
Newspaper articles, Media, announcements, etc are some examples. Here is the What Canvas:

Where?
Now that you know who is associated with the problem and what the problem actually is; you need
to focus on the context/situation/location of the problem. This block will help you look into the
situation in which the problem arises, the context of it, and the locations where it is prominent. Here
is the Where Canvas:
Why?
You have finally listed down all the major elements that affect the problem directly. Now it is
convenient to understand who the people that would be benefitted by the solution are; what is to be
solved; and where will the solution be deployed. These three canvases now become the base of why
you want to solve this problem. Thus, in the “Why” canvas, think about the benefits which the
stakeholders would get from the solution and how it will benefit them as well as the society.

After filling the 4Ws Problem canvas, you now need to summarise all the cards into one template. The
Problem Statement Template helps us to summarise all the key points into one single Template so
that in future, whenever there is need to look back at the basis of the problem, we can take a look at
the Problem Statement Template and understand the key elements of it.
[stakeholder(s)] Who
Our

has /have a [issue, problem, need] What

problem that

when / while [context, situation] Where

An ideal [benefit of solution for them] Why

solution would

Data Acquisition
As we move ahead in the AI Project Cycle, we come across the second element which is : Data
Acquisition. As the term clearly mentions, this stage is about acquiring data for the project. Let us first
understand what is Data. Data can be a piece of information or facts and statistics collected together
for reference or analysis. Whenever we want an AI project to be able to predict an output, we need
to train it first using data.

For example, If you want to make an Artificially Intelligent system which can predict the salary of any
employee based on his previous salaries, you would feed the data of his previous salaries into the
machine. This is the data with which the machine can be trained. Now, once it is ready, it will predict
his next salary efficiently. The previous salary data here is known as Training Data while the next salary
prediction data set is known as the Testing Data.

For better efficiency of an AI project, the Training data needs to be relevant and authentic. In the
previous example, if the training data was not of the previous salaries but of his expenses, the machine
would not have predicted his next salary correctly since the whole training went wrong. Similarly, if
the previous salary data was not authentic, that is, it was not correct, then too the prediction could
have gone wrong. Hence….

For any AI project to be efficient, the training data should be authentic and relevant to the problem
statement scoped.
Data Features
Look at your problem statement once again and try to find the data features required to address this
issue. Data features refer to the type of data you want to collect. In our previous example, data
features would be salary amount, increment percentage, increment period, bonus, etc.

After mentioning the Data features, you get to know what sort of data is to be collected. Now, the
question arises- From where can we get this data? There can be various ways in which you can collect
data. Some of them are:

Surveys Web Scraping Sensors

API
Cameras Observations (Application Program
Interface)

Sometimes, you use the internet and try to acquire data for your project from some random websites.
Such data might not be authentic as its accuracy cannot be proved. Due to this, it becomes necessary
to find a reliable source of data from where some authentic information can be taken. At the same
time, we should keep in mind that the data which we collect is open-sourced and not someone’s
property. Extracting private data can be an offence. One of the most reliable and authentic sources of
information, are the open-sourced websites hosted by the government. These government portals
have general information collected in suitable format which can be downloaded and used wisely.

Some of the open-sourced Govt. portals are: data.gov.in, india.gov.in

Data Exploration
In the previous modules, you have set the goal of your project and have also found ways to acquire
data. While acquiring data, you must have noticed that the data is a complex entity – it is full of
numbers and if anyone wants to make some sense out of it, they have to work some patterns out of
it. For example, if you go to the library and pick up a random book, you first try to go through its
content quickly by turning pages and by reading the description before borrowing it for yourself,
because it helps you in understanding if the book is appropriate to your needs and interests or not.

Thus, to analyse the data, you need to visualise it in some user-friendly format so that you can:

● Quickly get a sense of the trends, relationships and patterns contained within the data.
● Define strategy for which model to use at a later stage.
● Communicate the same to others effectively. To visualise data, we can use various types of
visual representations.

Are you aware of visual representations of data? Fill them below:

Bar
Graphs

Visual
Representations

Modelling
In the previous module of Data exploration, we have seen various types of graphical representations
which can be used for representing different parameters of data. The graphical representation makes
the data understandable for humans as we can discover trends and patterns out of it. But when it
comes to machines accessing and analysing data, it needs the data in the most basic form of numbers
(which is binary – 0s and 1s) and when it comes to discovering patterns and trends in data, the machine
goes in for mathematical representations of the same. The ability to mathematically describe the
relationship between parameters is the heart of every AI model. Thus, whenever we talk about
developing AI models, it is the mathematical approach towards analysing data which we refer to.

Generally, AI models can be classified as follows:

Machine
Learning
Learning
Based Deep
AI Models
Learning
Rule Based

Rule Based Approach

Refers to the AI modelling where the rules are defined by the developer. The machine follows the
rules or instructions mentioned by the developer and performs its task accordingly. For example, we
have a dataset which tells us about the conditions on the basis of which we can decide if an elephant
may be spotted or not while on safari. The parameters are: Outlook, Temperature, Humidity and Wind.
Now, let’s take various possibilities of these parameters and see in which case the elephant may be
spotted and in which case it may not. After looking through all the cases, we feed this data in to the
machine along with the rules which tell the machine all the possibilities. The machine trains on this
data and now is ready to be tested. While testing the machine, we tell the machine that Outlook =
Overcast; Temperature = Normal; Humidity = Normal and Wind = Weak. On the basis of this testing
dataset, now the machine will be able to tell if the elephant has been spotted before or not and will
display the prediction to us. This is known as a rule-based approach because we fed the data along
with rules to the machine and the machine after getting trained on them is now able to predict
answers for the same. A drawback/feature for this approach is that the learning is static. The machine
once trained, does not take into consideration any changes made in the original training dataset. That
is, if you try testing the machine on a dataset which is different from the rules and data you fed it at
the training stage, the machine will fail and will not learn from its mistake. Once trained, the model
cannot improvise itself on the basis of feedbacks. Thus, machine learning gets introduced as an
extension to this as in that case, the machine adapts to change in data and rules and follows the
updated path only, while a rule-based model does what it has been taught once.

Learning Based Approach

Refers to the AI modelling where the machine learns by itself. Under the Learning Based approach,
the AI model gets trained on the data fed to it and then is able to design a model which is adaptive to
the change in data. That is, if the model is trained with X type of data and the machine designs the
algorithm around it, the model would modify itself according to the changes which occur in the data
so that all the exceptions are handled in this case. For example, suppose you have a dataset comprising
of 100 images of apples and bananas each. These images depict apples and bananas in various shapes
and sizes. These images are then labelled as either apple or banana so that all apple images are
labelled ‘apple’ and all the banana images have ‘banana’ as their label. Now, the AI model is trained
with this dataset and the model is programmed in such a way that it can distinguish between an apple
image and a banana image according to their features and can predict the label of any image which is
fed to it as an apple or a banana. After training, the machine is now fed with testing data. Now, the
testing data might not have similar images as the ones on which the model has been trained. So, the
model adapts to the features on which it has been trained and accordingly predicts if the image is of
an apple or banana. In this way, the machine learns by itself by adapting to the new data which is
flowing in. This is the machine learning approach which introduces the dynamicity in the model.

The learning-based approach can further be

Learning Based Approach

divided into three parts:

Supervised Learning
Supervised
In a supervised learning model, the dataset
Learning which is fed to the machine is labelled. In
other words, we can say that the dataset is
Unsupervised known to the person who is training the
Learning machine only then he/she is able to label the
data. A label is some information which can
Reinforcement be used as a tag for data. For example,
students get grades according to the marks
Learning they secure in examinations. These grades
are labels which categorise the students
according to their marks.
There are two types of Supervised Learning models:

Classification: Where the data is classified

according to the labels. For example, in the grading
system, students are classified on the basis of the
grades they obtain with respect to their marks in
the examination. This model works on discrete
dataset which means the data need not be
continuous.

Regression: Such models work on continuous data.

For example, if you wish to predict your next salary,
then you would put in the data of your previous
salary, any increments, etc., and would train the
model. Here, the data which has been fed to the
machine is continuous.

Unsupervised Learning
An unsupervised learning model works on unlabelled dataset. This means that the data which is fed
to the machine is random and there is a possibility that the person who is training the model does not
have any information regarding it. The unsupervised learning models are used to identify
relationships, patterns and trends out of the data which is fed into it. It helps the user in understanding
what the data is about and what are the major features identified by the machine in it.

For example, you have a random data of 1000 dog images and you wish to understand some pattern
out of it, you would feed this data into the unsupervised learning model and would train the machine
on it. After training, the machine would come up with patterns which it was able to identify out of it.
The Machine might come up with patterns which are already known to the user like colour or it might
even come up with something very unusual like the size of the dogs.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Unsupervised learning models can be further divided into two categories:

Clustering: Refers to the unsupervised learning

algorithm which can cluster the unknown data
according to the patterns or trends identified out of
it. The patterns observed might be the ones which
are known to the developer or it might even come
up with some unique patterns out of it.

Dimensionality Reduction: We humans are able to visualise upto 3-Dimensions only but according to
a lot of theories and algorithms, there are various entities which exist beyond 3-Dimensions. For
example, in Natural language Processing, the words are considered to be N-Dimensional entities.
Which means that we cannot visualise them as they exist beyond our visualisation ability. Hence, to
make sense out of it, we need to reduce their dimensions. Here, dimensionality reduction algorithm
is used.

As we reduce the dimension of an entity, the information which it contains starts getting distorted.
For example, if we have a ball in our hand, it is 3-Dimensions right now. But if we click its picture, the
data transforms to 2-D as an image is a 2-Dimensional entity. Now, as soon as we reduce one
dimension, at least 50% of the information is lost as now we will not know about the back of the ball.
Whether the ball was of same colour at the back or not? Or was it just a hemisphere? If we reduce the
dimensions further, more and more information will get lost.

Hence, to reduce the dimensions and still be able to make sense out of the data, we use Dimensionality
Reduction.

Evaluation
Once a model has been made and trained, it needs to go through proper testing so that one can
calculate the efficiency and performance of the model. Hence, the model is tested with the help of
Testing Data (which was separated out of the acquired dataset at Data Acquisition stage) and the
efficiency of the model is calculated on the basis of the parameters mentioned below:

Accuracy Precision Recall F1 Score

You will read more about this stage in Chapter 7.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Neural Networks
Neural networks are loosely modelled after how neurons in the human brain behave. The key
advantage of neural networks are that they are able to extract data features automatically without
needing the input of the programmer. A neural network is essentially a system of organizing machine
learning algorithms to perform certain tasks. It is a fast and efficient way to solve problems for which
the dataset is very large, such as in images.

As seen in the figure given, the larger Neural Networks tend to perform better with larger amounts of
data whereas the traditional machine learning algorithms stop improving after a certain saturation
point.

This is a representation of how neural networks work. A Neural Network is divided into multiple layers
and each layer is further divided into several blocks called nodes. Each node has its own task to
accomplish which is then passed to the next layer. The first layer of a Neural Network is known as the
input layer. The job of an input layer is to acquire data and feed it to the Neural Network. No
processing occurs at the input layer. Next to it, are the hidden layers. Hidden layers are the layers in
which the whole processing occurs. Their name essentially means that these layers are hidden and are
not visible to the user.

Each node of these hidden layers has its own machine learning algorithm which it executes on the
data received from the input layer. The processed output is then fed to the subsequent hidden layer

* Images shown here are the property of individual organisations and are used here for reference purpose only.
of the network. There can be multiple hidden layers in a neural network system and their number
depends upon the complexity of the function for which the network has been configured. Also, the
number of nodes in each layer can vary accordingly. The last hidden layer passes the final processed
data to the output layer which then gives it to the user as the final output. Similar to the input layer,
output layer too does not process the data which it acquires. It is meant for user-interface.

Some of the features of a Neural Network are listed below:

Natural Language Processing
Introduction
Till now, we have explored two domains of AI: Data Science and Computer Vision. Both these domains
differ from each other in terms of the data on which they work. Data Science works around numbers
and tabular data while Computer Vision is all about visual data like images and videos. The third
domain, Natural Language Processing (commonly called NLP) takes in the data of Natural Languages
which humans use in their daily lives and operates on this.

Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to
understand and process human languages. AI is a subfield of Linguistics, Computer Science,
Information Engineering, and Artificial Intelligence concerned with the interactions between
computers and human (natural) languages, in particular how to program computers to process and
analyse large amounts of natural language data.

But how do computers do that? How do they understand what we say in our language? This chapter
is all about demystifying the Natural Language Processing domain and understanding how it works.

Before we get deeper into NLP, let us experience it with the help of this AI Game:

Identify the mystery animal: http://bit.ly/iai4yma

Go to this link on Google Chrome, launch the experiment and try to identify the Mystery Animal by
asking the machine 20 Yes or No questions.

Were you able to guess the animal?

__________________________________________________________________________________
__________________________________________________________________________________

If yes, in how many questions were you able to guess it?

__________________________________________________________________________________
__________________________________________________________________________________

If no, how many times did you try playing this game?

__________________________________________________________________________________
__________________________________________________________________________________

What according to you was the task of the machine?

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Were there any challenges that you faced while playing this game? If yes, list them down.

What approach must one follow to win this game?

Applications of Natural Language Processing

Since Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very
commonly used by the majority of people in their daily lives. Here are some of the applications of
Natural Language Processing which are used in the real-life scenario:

Automatic Summarization: Information overload is a real

problem when we need to access a specific, important piece
of information from a huge knowledge base. Automatic
summarization is relevant not only for summarizing the
meaning of documents and information, but also to
understand the emotional meanings within the information,
such as in collecting data from social media. Automatic
summarization is especially relevant when used to provide an
overview of a news item or blog post, while avoiding
redundancy from multiple sources and maximizing the
diversity of content obtained.

Sentiment Analysis: The goal of sentiment

analysis is to identify sentiment among several
posts or even in the same post where emotion is
not always explicitly expressed. Companies use
Natural Language Processing applications, such as
sentiment analysis, to identify opinions and
sentiment online to help them understand what
customers think about their products and services
(i.e., “I love the new iPhone” and, a few lines later
“But sometimes it doesn’t work well” where the
person is still talking about the iPhone) and overall

* Images shown here are the property of individual organisations and are used here for reference purpose only.
indicators of their reputation. Beyond determining simple polarity, sentiment analysis understands
sentiment in context to help better understand what’s behind an expressed opinion, which can be
extremely relevant in understanding and driving purchasing decisions.

Text classification: Text classification makes it possible to assign

predefined categories to a document and organize it to help you
find the information you need or simplify some activities. For
example, an application of text categorization is spam filtering in
email.

Virtual Assistants: Nowadays Google Assistant, Cortana,

Siri, Alexa, etc have become an integral part of our lives. Not
only can we talk to them but they also have the abilities to
make our lives easier. By accessing our data, they can help
us in keeping notes of our tasks, make calls for us, send
messages and a lot more. With the help of speech
recognition, these assistants can not only detect our speech
but can also make sense out of it. According to recent
researches, a lot more advancements are expected in this
field in the near future.

Natural Language Processing: Getting Started

Natural Language Processing is all about how machines try to understand and interpret human
language and operate accordingly. But how can Natural Language Processing be used to solve the
problems around us? Let us take a look.

Revisiting the AI Project Cycle

Let us try to understand how we can develop a project in Natural Language processing with the help
of an example.

The Scenario
The world is competitive nowadays. People face
competition in even the tiniest tasks and are expected to
give their best at every point in time. When people are
unable to meet these expectations, they get stressed and
could even go into depression. We get to hear a lot of cases
where people are depressed due to reasons like peer
pressure, studies, family issues, relationships, etc. and they
eventually get into something that is bad for them as well
as for others. So, to overcome this, cognitive behavioural
therapy (CBT) is considered to be one of the best methods
to address stress as it is easy to implement on people and
also gives good results. This therapy includes

* Images shown here are the property of individual organisations and are used here for reference purpose only.
understanding the behaviour and mindset of a person in their normal life. With the help of CBT,
therapists help people overcome their stress and live a happy life.
To understand more about the concept of this therapy, visit this link:
https://en.wikipedia.org/wiki/Cognitive_behavioral_therapy

Problem Scoping
CBT is a technique used by most therapists to cure patients out of stress and depression. But it has
been observed that people do not wish to seek the help of a psychiatrist willingly. They try to avoid
such interactions as much as possible. Thus, there is a need to bridge the gap between a person who
needs help and the psychiatrist. Let us look at various factors around this problem through the 4Ws
problem canvas.

Who Canvas – Who has the problem?

Who are the

o People who suffer from stress and are at the onset of depression.
stakeholders?

What do we know
o People who are going through stress are reluctant to consult a psychiatrist.
about them?

What Canvas – What is the nature of the problem?

What is the o People who need help are reluctant to consult a psychiatrist and hence live
problem? miserably.

How do you know o Studies around mental stress and depression available on various authentic
it is a problem? sources.

Where Canvas – Where does the problem arise?

What is the context/situation

o When they are going through a stressful period of time
in which the stakeholders
o Due to some unpleasant experiences
experience this problem?

Why Canvas – Why do you think it is a problem worth solving?

o People get a platform where they can talk and vent out their
What would be of key feelings anonymously
value to the stakeholders? o People get a medium that can interact with them and applies
primitive CBT on them and can suggest help whenever needed

How would it improve their o People would be able to vent out their stress
situation? o They would consider going to a psychiatrist whenever required
Now that we have gone through all the factors around the problem, the problem statement templates
go as follows:

Our People undergoing stress Who?

Have a problem of Not being able to share their feelings What?
While They need help in venting out their emotions Where?
Provide them a platform to share their thoughts
An ideal solution would Why
anonymously and suggest help whenever required

This leads us to the goal of our project which is:

“To create a chatbot which can interact with people, help them
to vent out their feelings and take them through primitive CBT.”

Data Acquisition
To understand the sentiments of people, we need to collect their conversational data so the machine
can interpret the words that they use and understand their meaning. Such data can be collected from
various means:

1. Surveys 2. Observing the therapist’s sessions

3. Databases available on the internet 4. Interviews, etc.
Data Exploration
Once the textual data has been collected, it needs to be processed and cleaned so that an easier
version can be sent to the machine. Thus, the text is normalised through various steps and is lowered
to minimum vocabulary since the machine does not require grammatically correct statements but the
essence of it.

Modelling
Once the text has been normalised, it is then fed to an NLP based AI model. Note that in NLP, modelling
requires data pre-processing only after which the data is fed to the machine. Depending upon the type
of chatbot we try to make, there are a lot of AI models available which help us build the foundation of
our project.

Evaluation
The model trained is then evaluated and the accuracy for the same is generated on the basis of the
relevance of the answers which the machine gives to the user’s responses. To understand the
efficiency of the model, the suggested answers by the chatbot are compared to the actual answers.
As you can see in the above diagram, the blue line talks about the model’s output while the green one
is the actual output along with the data samples.

The model’s output does not match the true function at all. Hence the model is said
Figure 1 to be underfitting and its accuracy is lower.

In the second one, the model’s performance matches well with the true function
Figure 2 which states that the model has optimum accuracy and the model is called a
perfect fit.

In the third case, model performance is trying to cover all the data samples even if
Figure 3 they are out of alignment to the true function. This model is said to be overfitting
and this too has a lower accuracy.

Once the model is evaluated thoroughly, it is then deployed in the form of an app which people can
use easily.

Chatbots
As we have seen earlier, one of the most common applications of Natural Language Processing is a
chatbot. There are a lot of chatbots available and many of them use the same approach as we used in
the scenario above.. Let us try some of the chatbots and see how they work.

• Mitsuku Bot*
https://www.pandorabots.com/mitsuku/

• CleverBot*
https://www.cleverbot.com/

• Jabberwacky*
http://www.jabberwacky.com/

• Haptik*
https://haptik.ai/contact-us

* Images shown here are the property of individual organisations and are used here for reference purpose only.
• Rose*
http://ec2-54-215-197-164.us-west-1.compute.amazonaws.com/speech.php

• Ochatbot*
https://www.ometrics.com/blog/list-of-fun-chatbots/

Let us discuss!
• Which chatbot did you try? Name any one.
• What is the purpose of this chatbot?
• How was the interaction with the chatbot?
• Did the chat feel like talking to a human or a robot? Why do you think so?
• Do you feel that the chatbot has a certain personality?
As you interact with more and more chatbots, you would realise that some of them are scripted or in
other words are traditional chatbots while others were AI-powered and had more knowledge. With
the help of this experience, we can understand that there are 2 types of chatbots around us: Script-
bot and Smart-bot. Let us understand what each of them mean in detail:

Script-bot Smart-bot
Script bots are easy to make Smart-bots are flexible and powerful
Script bots work around a script which is Smart bots work on bigger databases and other
programmed in them resources directly
Mostly they are free and are easy to integrate Smart bots learn with more data
to a messaging platform
No or little language processing skills Coding is required to take this up on board
Limited functionality Wide functionality

The story speaker activity which was done in class 9 can be considered as a script-bot as in that activity
we used to create a script around which the interactive story revolved. As soon as the machine got
triggered by the person, it used to follow the script and answer accordingly. Other examples of script
bot may include the bots which are deployed in the customer care section of various companies. Their
job is to answer some basic queries that they are coded for and connect them to human executives
once they are unable to handle the conversation.

On the other hand, all the assistants like Google Assistant, Alexa, Cortana, Siri, etc. can be taken as
smart bots as not only can they handle the conversations but can also manage to do other tasks which
makes them smarter.

Human Language VS Computer Language

Humans communicate through language which we process all the time. Our brain keeps on processing
the sounds that it hears around itself and tries to make sense out of them all the time. Even in the
classroom, as the teacher delivers the session, our brain is continuously processing everything and
storing it in some place. Also, while this is happening, when your friend whispers something, the focus
of your brain automatically shifts from the teacher’s speech to your friend’s conversation. So now, the
brain is processing both the sounds but is prioritising the one on which our interest lies.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
The sound reaches the brain through a long channel. As a person speaks, the sound travels from his
mouth and goes to the listener’s eardrum. The sound striking the eardrum is converted into neuron
impulse, gets transported to the brain and then gets processed. After processing the signal, the brain
gains understanding around the meaning of it. If it is clear, the signal gets stored. Otherwise, the
listener asks for clarity to the speaker. This is how human languages are processed by humans.

On the other hand, the computer understands the language of numbers. Everything that is sent to the
machine has to be converted to numbers. And while typing, if a single mistake is made, the computer
throws an error and does not process that part. The communications made by the machines are very
basic and simple.

Now, if we want the machine to understand our language, how should this happen? What are the
possible difficulties a machine would face in processing natural language? Let us take a look at some
of them here:

Arrangement of the words and meaning

There are rules in human language. There are nouns, verbs, adverbs, adjectives. A word can be a noun
at one time and an adjective some other time. There are rules to provide structure to a language.

This is the issue related to the syntax of the language. Syntax refers to the grammatical structure of a
sentence. When the structure is present, we can start interpreting the message. Now we also want to
have the computer do this. One way to do this is to use the part-of-speech tagging. This allows the
computer to identify the different parts of a speech.

Besides the matter of arrangement, there’s also meaning behind the language we use. Human
communication is complex. There are multiple characteristics of the human language that might be
easy for a human to understand but extremely difficult for a computer to understand.

Analogy with programming language:

Different syntax, same semantics: 2+3 = 3+2

Here the way these statements are written is different, but their meanings are the same that is 5.

Different semantics, same syntax: 2/3 (Python 2.7) ≠ 2/3 (Python 3)

Here the statements written have the same syntax but their meanings are different. In Python 2.7,
this statement would result in 1 while in Python 3, it would give an output of 1.5.

Think of some other examples of different syntax and same semantics and vice-versa.

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Multiple Meanings of a word
Let’s consider these three sentences:

His face turned red after he found out that he took the wrong bag
What does this mean? Is he feeling ashamed because he took another person’s bag instead of his? Is
he feeling angry because he did not manage to steal the bag that he has been targeting?

The red car zoomed past his nose

Probably talking about the color of the car

His face turns red after consuming the medicine

Is he having an allergic reaction? Or is he not able to bear the taste of that medicine?

Here we can see that context is important. We understand a sentence almost intuitively, depending
on our history of using the language, and the memories that have been built within. In all three
sentences, the word red has been used in three different ways which according to the context of the
statement changes its meaning completely. Thus, in natural language, it is important to understand
that a word can have multiple meanings and the meanings fit into the statement according to the
context of it.

Think of some other words which can have multiple meanings and use them in sentences.

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

Perfect Syntax, no Meaning

Sometimes, a statement can have a perfectly correct syntax but it does not mean anything. For
example, take a look at this statement:

Chickens feed extravagantly while the moon drinks tea.

This statement is correct grammatically but does this make any sense? In Human language, a perfect
balance of syntax and semantics is important for better understanding.
Think of some other sentences having correct syntax and incorrect semantics.

__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________

These are some of the challenges we might have to face if we try to teach computers how to
understand and interact in human language. So how does Natural Language Processing do this magic?

Data Processing
Humans interact with each other very easily. For us, the natural languages that we use are so
convenient that we speak them easily and understand them well too. But for computers, our
languages are very complex. As you have already gone through some of the complications in human
languages above, now it is time to see how Natural Language Processing makes it possible for the
machines to understand and speak in the Natural Languages just like humans.

Since we all know that the language of computers is Numerical, the very first step that comes to our
mind is to convert our language to numbers. This conversion takes a few steps to happen. The first
step to it is Text Normalisation. Since human languages are complex, we need to first of all simplify
them in order to make sure that the understanding becomes possible. Text Normalisation helps in
cleaning up the textual data in such a way that it comes down to a level where its complexity is lower
than the actual data. Let us go through Text Normalisation in detail.

Text Normalisation
In Text Normalisation, we undergo several steps to normalise the text to a lower level. Before we
begin, we need to understand that in this section, we will be working on a collection of written text.
That is, we will be working on text from multiple documents and the term used for the whole textual
data from all the documents altogether is known as corpus. Not only would we go through all the
steps of Text Normalisation, we would also work them out on a corpus. Let us take a look at the steps:

Sentence Segmentation
Under sentence segmentation, the whole corpus is divided into sentences. Each sentence is taken as
a different data so now the whole corpus gets reduced to sentences.
Tokenisation
After segmenting the sentences, each sentence is then further divided into tokens. Tokens is a term
used for any word or number or special character occurring in a sentence. Under tokenisation, every
word, number and special character is considered separately and each of them is now a separate
token.

Removing Stopwords, Special Characters and Numbers

In this step, the tokens which are not necessary are removed from the token list. What can be the
possible words which we might not require?

Stopwords are the words which occur very frequently in the corpus but do not add any value to it.
Humans use grammar to make their sentences meaningful for the other person to understand. But
grammatical words do not add any essence to the information which is to be transmitted through the
statement hence they come under stopwords. Some examples of stopwords are:

* Images shown here are the property of individual organisations and are used here for reference purpose only.
These words occur the most in any given corpus but talk very little or nothing about the context or the
meaning of it. Hence, to make it easier for the computer to focus on meaningful terms, these words
are removed.

Along with these words, a lot of times our corpus might have special characters and/or numbers. Now
it depends on the type of corpus that we are working on whether we should keep them in it or not.
For example, if you are working on a document containing email IDs, then you might not want to
remove the special characters and numbers whereas in some other textual data if these characters do
not make sense, then you can remove them along with the stopwords.

Converting text to a common case

After the stopwords removal, we convert the whole text into a similar case, preferably lower case.
This ensures that the case-sensitivity of the machine does not consider same words as different just
because of different cases.

Here in this example, the all the 6 forms of hello would be converted to lower case and hence would
be treated as the same word by the machine.

Stemming
In this step, the remaining words are reduced to their root words. In other words, stemming is the
process in which the affixes of words are removed and the words are converted to their base form.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Note that in stemming, the stemmed words (words which are we get after removing the affixes) might
not be meaningful. Here in this example as you can see: healed, healing and healer all were reduced
to heal but studies was reduced to studi after the affix removal which is not a meaningful word.
Stemming does not take into account if the stemmed word is meaningful or not. It just removes the
affixes hence it is faster.

Lemmatization
Stemming and lemmatization both are alternative processes to each other as the role of both the
processes is same – removal of affixes. But the difference between both of them is that in
lemmatization, the word we get after affix removal (also known as lemma) is a meaningful one.
Lemmatization makes sure that lemma is a word with meaning and hence it takes a longer time to
execute than stemming.

As you can see in the same example, the output for studies after affix removal has become study
instead of studi.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Difference between stemming and lemmatization can be summarized by this example:

With this we have normalised our text to tokens which are the simplest form of words present in the
corpus. Now it is time to convert the tokens into numbers. For this, we would use the Bag of Words
algorithm

Bag of Words
Bag of Words is a Natural Language Processing model which helps in extracting features out of the
text which can be helpful in machine learning algorithms. In bag of words, we get the occurrences of
each word and construct the vocabulary for the corpus.

This image gives us a brief overview about how bag of words works. Let us assume that the text on
the left in this image is the normalised corpus which we have got after going through all the steps of
text processing. Now, as we put this text into the bag of words algorithm, the algorithm returns to us
the unique words out of the corpus and their occurrences in it. As you can see at the right, it shows us
a list of words appearing in the corpus and the numbers corresponding to it shows how many times
the word has occurred in the text body. Thus, we can say that the bag of words gives us two things:

1. A vocabulary of words for the corpus

2. The frequency of these words (number of times it has occurred in the whole corpus).

Here calling this algorithm “bag” of words symbolises that the sequence of sentences or tokens does
not matter in this case as all we need are the unique words and their frequency in it.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Here is the step-by-step approach to implement bag of words algorithm:

1. Text Normalisation: Collect data and pre-process it

2. Create Dictionary: Make a list of all the unique words occurring in the corpus. (Vocabulary)
3. Create document vectors: For each document in the corpus, find out how many times the
word from the unique list of words has occurred.
4. Create document vectors for all the documents.
Let us go through all the steps with an example:

Step 1: Collecting data and pre-processing it.

Document 1: Aman and Anil are stressed

Document 2: Aman went to a therapist

Document 3: Anil went to download a health chatbot

Here are three documents having one sentence each. After text normalisation, the text becomes:

Document 1: [aman, and, anil, are, stressed]

Document 2: [aman, went, to, a, therapist]

Document 3: [anil, went, to, download, a, health, chatbot]

Note that no tokens have been removed in the stopwords removal step. It is because we have very
little data and since the frequency of all the words is almost the same, no word can be said to have
lesser value than the other.

Step 2: Create Dictionary

Go through all the steps and create a dictionary i.e., list down all the words which occur in all three
documents:

Dictionary:

aman and anil are stressed went

download health chatbot therapist a to

Note that even though some words are repeated in different documents, they are all written just once
as while creating the dictionary, we create the list of unique words.

Step 3: Create document vector

In this step, the vocabulary is written in the top row. Now, for each word in the document, if it matches
with the vocabulary, put a 1 under it. If the same word appears again, increment the previous value
by 1. And if the word does not occur in that document, put a 0 under it.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Since in the first document, we have words: aman, and, anil, are, stressed. So, all these words get a
value of 1 and rest of the words get a 0 value.

Step 4: Repeat for all documents

Same exercise has to be done for all the documents. Hence, the table becomes:

In this table, the header row contains the vocabulary of the corpus and three rows correspond to three
different documents. Take a look at this table and analyse the positioning of 0s and 1s in it.

Finally, this gives us the document vector table for our corpus. But the tokens have still not converted
to numbers. This leads us to the final steps of our algorithm: TFIDF.

TFIDF: Term Frequency & Inverse Document Frequency

Suppose you have a book. Which characters or words do you think would occur the most in it?

__________________________________________________________________________________
__________________________________________________________________________________

Bag of words algorithm gives us the frequency of words in each document we have in our corpus. It
gives us an idea that if the word is occurring more in a document, its value is more for that document.
For example, if I have a document on air pollution, air and pollution would be the words which occur
many times in it. And these words are valuable too as they give us some context around the document.
But let us suppose we have 10 documents and all of them talk about different issues. One is on women
empowerment, the other is on unemployment and so on. Do you think air and pollution would still be
one of the most occurring words in the whole corpus? If not, then which words do you think would
have the highest frequency in all of them?

And, this, is, the, etc. are the words which occur the most in almost all the documents. But these words
do not talk about the corpus at all. Though they are important for humans as they make the
statements understandable to us, for the machine they are a complete waste as they do not provide
us with any information regarding the corpus. Hence, these are termed as stopwords and are mostly
removed at the pre-processing stage only.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Take a look at this graph. It is a plot of occurrence of words versus their value. As you can see, if the
words have highest occurrence in all the documents of the corpus, they are said to have negligible
value hence they are termed as stop words. These words are mostly removed at the pre-processing
stage only. Now as we move ahead from the stopwords, the occurrence level drops drastically and the
words which have adequate occurrence in the corpus are said to have some amount of value and are
termed as frequent words. These words mostly talk about the document’s subject and their
occurrence is adequate in the corpus. Then as the occurrence of words drops further, the value of
such words rises. These words are termed as rare or valuable words. These words occur the least but
add the most value to the corpus. Hence, when we look at the text, we take frequent and rare words
into consideration.

Let us now demystify TFIDF. TFIDF stands for Term Frequency and Inverse Document Frequency. TFIDF
helps un in identifying the value for each word. Let us understand each term one by one.

Term Frequency
Term frequency is the frequency of a word in one document. Term frequency can easily be found from
the document vector table as in that table we mention the frequency of each word of the vocabulary
in each document.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Here, you can see that the frequency of each word for each document has been recorded in the table.
These numbers are nothing but the Term Frequencies!

Inverse Document Frequency

Now, let us look at the other half of TFIDF which is Inverse Document Frequency. For this, let us first
understand what does document frequency mean. Document Frequency is the number of documents
in which the word occurs irrespective of how many times it has occurred in those documents. The
document frequency for the exemplar vocabulary would be:

Here, you can see that the document frequency of ‘aman’, ‘anil’, ‘went’, ‘to’ and ‘a’ is 2 as they have
occurred in two documents. Rest of them occurred in just one document hence the document
frequency for them is one.

Talking about inverse document frequency, we need to put the document frequency in the
denominator while the total number of documents is the numerator. Here, the total number of
documents are 3, hence inverse document frequency becomes:

Finally, the formula of TFIDF for any word W becomes:

TFIDF(W) = TF(W) * log( IDF(W) )

Here, log is to the base of 10. Don’t worry! You don’t need to calculate the log values by yourself.
Simply use the log function in the calculator and find out!

Now, let’s multiply the IDF values to the TF values. Note that the TF values are for each document
while the IDF values are for the whole corpus. Hence, we need to multiply the IDF values to each row
of the document vector table.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Here, you can see that the IDF values for Aman in each row is the same and similar pattern is followed
for all the words of the vocabulary. After calculating all the values, we get:

Finally, the words have been converted to numbers. These numbers are the values of each for each
document. Here, you can see that since we have less amount of data, words like ‘are’ and ‘and’ also
have a high value. But as the IDF value increases, the value of that word decreases. That is, for
example:

Total Number of documents: 10

Number of documents in which ‘and’ occurs: 10

Therefore, IDF(and) = 10/10 = 1

Which means: log(1) = 0. Hence, the value of ‘and’ becomes 0.

On the other hand, number of documents in which ‘pollution’ occurs: 3

IDF(pollution) = 10/3 = 3.3333…

Which means: log(3.3333) = 0.522; which shows that the word ‘pollution’ has considerable value in
the corpus.

Summarising the concept, we can say that:

1. Words that occur in all the documents with high term frequencies have the least values and
are considered to be the stopwords.
2. For a word to have high TFIDF value, the word needs to have a high term frequency but less
document frequency which shows that the word is important for one document but is not a
common word for all documents.
3. These values help the computer understand which words are to be considered while
processing the natural language. The higher the value, the more important the word is for a
given corpus.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Applications of TFIDF
TFIDF is commonly used in the Natural Language Processing domain. Some of its applications are:

Document Information
Topic Modelling Stop word filtering
Classification Retrieval System

Helps in classifying the To extract the Helps in removing the

It helps in predicting
type and genre of a important information unnecessary words
the topic for a corpus.
document. out of a corpus. out of a text body.

DIY – Do It Yourself!
Here is a corpus for you to challenge yourself with the given tasks. Use the knowledge you have
gained in the above sections and try completing the whole exercise by yourself.

The Corpus
Document 1: We can use health chatbots for treating stress.

Document 2: We can use NLP to create chatbots and we will be making health chatbots now!

Document 3: Health Chatbots cannot replace human counsellors now. Yay >< !! @1nteLA!4Y

Accomplish the following challenges on the basis of the corpus given above. You can use the tools
available online for these challenges. Link for each tool is given below:

1. Sentence Segmentation: https://tinyurl.com/y36hd92n

2. Tokenisation: https://text-processing.com/demo/tokenize/

3. Stopwords removal: https://demos.datasciencedojo.com/demo/stopwords/

4. Lowercase conversion: https://caseconverter.com/

5. Stemming: http://textanalysisonline.com/nltk-porter-stemmer

6. Lemmatisation: http://textanalysisonline.com/spacy-word-lemmatize

7. Bag of Words: Create a document vector table for all documents.

8. Generate TFIDF values for all the words.

9. Find the words having highest value.

10. Find the words having the least value.

Evaluation
Introduction
Till now we have learnt about the 4 stages of AI project cycle, viz. Problem scoping, Data acquisition,
Data exploration and modelling. While in modelling we can make different types of models, how do
we check if one’s better than the other? That’s where Evaluation comes into play. In the Evaluation
stage, we will explore different methods of evaluating an AI model. Model Evaluation is an integral
part of the model development process. It helps to find the best model that represents our data and
how well the chosen model will work in the future

What is evaluation?
Evaluation is the process of understanding the reliability of any AI model, based on outputs by feeding
test dataset into the model and comparing with actual answers. There can be different Evaluation
techniques, depending of the type and purpose of the model. Remember that It’s not recommended
to use the data we used to build the model to evaluate it. This is because our model will simply
remember the whole training set, and will therefore always predict the correct label for any point in
the training set. This is known as overfitting.

Firstly, let us go through various terms which are very important to the evaluation process.

Model Evaluation Terminologies

There are various new terminologies which come into the picture when we work on evaluating our
model. Let’s explore them with an example of the Forest fire scenario.

The Scenario
Imagine that you have come up with an AI based prediction model which has been deployed in a forest
which is prone to forest fires. Now, the objective of the model is to predict whether a forest fire has
broken out in the forest or not. Now, to understand the efficiency of this model, we need to check if
the predictions which it makes are correct or not. Thus, there exist two conditions which we need to
ponder upon: Prediction and Reality. The prediction is the output which is given by the machine and
the reality is the real scenario in the forest when the prediction has been made. Now let us look at
various combinations that we can have with these two conditions.
Case 1: Is there a forest fire?

Here, we can see in the picture that a forest fire has broken out in the forest. The model predicts a Yes
which means there is a forest fire. The Prediction matches with the Reality. Hence, this condition is
termed as True Positive.

Case 2: Is there a forest fire?

Here there is no fire in the forest hence the reality is No. In this case, the machine too has predicted
it correctly as a No. Therefore, this condition is termed as True Negative.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Case 3: Is there a forest fire?

Here the reality is that there is no forest fire. But the machine has incorrectly predicted that there is
a forest fire. This case is termed as False Positive.

Case 4: Is there a forest fire?

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Here, a forest fire has broken out in the forest because of which the Reality is Yes but the machine has
incorrectly predicted it as a No which means the machine predicts that there is no Forest Fire.
Therefore, this case becomes False Negative.

Confusion matrix
The result of comparison between the prediction and reality can be recorded in what we call the
confusion matrix. The confusion matrix allows us to understand the prediction results. Note that it is
not an evaluation metric but a record which can help in evaluation. Let us once again take a look at
the four conditions that we went through in the Forest Fire example:

Let us now take a look at the confusion matrix:

Prediction and Reality can be easily mapped together with the help of this confusion matrix.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Evaluation Methods
Now as we have gone through all the possible combinations of Prediction and Reality, let us see how
we can use these conditions to evaluate the model.

Accuracy
Accuracy is defined as the percentage of correct predictions out of all the observations. A prediction
can be said to be correct if it matches the reality. Here, we have two conditions in which the Prediction
matches with the Reality: True Positive and True Negative. Hence, the formula for Accuracy becomes:

Here, total observations cover all the possible cases of prediction that can be True Positive (TP), True
Negative (TN), False Positive (FP) and False Negative (FN).

As we can see, Accuracy talks about how true the predictions are by any model. Let us ponder:

Is high accuracy equivalent to good performance?

__________________________________________________________________________________
__________________________________________________________________________________

How much percentage of accuracy is reasonable to show good performance?

__________________________________________________________________________________
__________________________________________________________________________________

Let us go back to the Forest Fire example. Assume that the model always predicts that there is no fire.
But in reality, there is a 2% chance of forest fire breaking out. In this case, for 98 cases, the model will
be right but for those 2 cases in which there was a forest fire, then too the model predicted no fire.

Here,

True Positives = 0

True Negatives = 98

Total cases = 100

Therefore, accuracy becomes: (98 + 0) / 100 = 98%

* Images shown here are the property of individual organisations and are used here for reference purpose only.
This is a fairly high accuracy for an AI model. But this parameter is useless for us as the actual cases
where the fire broke out are not taken into account. Hence, there is a need to look at another
parameter which takes account of such cases as well.

Precision
Precision is defined as the percentage of true positive cases versus all the cases where the prediction
is true. That is, it takes into account the True Positives and False Positives.

Going back to the Forest Fire example, in this case, assume that the model always predicts that there
is a forest fire irrespective of the reality. In this case, all the Positive conditions would be taken into
account that is, True Positive (Prediction = Yes and Reality = Yes) and False Positive (Prediction = Yes
and Reality = No). In this case, the firefighters will check for the fire all the time to see if the alarm was
True or False.

You might recall the story of the boy who falsely cries out that there are wolves every time and so
when they actually arrive, no one comes to his rescue. Similarly, here if the Precision is low (which
means there are more False alarms than the actual ones) then the firefighters would get complacent
and might not go and check every time considering it could be a false alarm.

* Images shown here are the property of individual organisations and are used here for reference purpose only.
This makes Precision an important evaluation criteria. If Precision is high, this means the True Positive
cases are more, giving lesser False alarms.

But again, is good Precision equivalent to a good model performance? Why?

Let us consider that a model has 100% precision. Which means that whenever the machine says
there’s a fire, there is actually a fire (True Positive). In the same model, there can be a rare exceptional
case where there was actual fire but the system could not detect it. This is the case of a False Negative
condition. But the precision value would not be affected by it because it does not take FN into account.
Is precision then a good parameter for model performance?

Recall
Another parameter for evaluating the model’s performance is Recall. It can be defined as the fraction
of positive cases that are correctly identified. It majorly takes into account the true reality cases where
in Reality there was a fire but the machine either detected it correctly or it didn’t. That is, it considers
True Positives (There was a forest fire in reality and the model predicted a forest fire) and False
Negatives (There was a forest fire and the model didn’t predict it).

* Images shown here are the property of individual organisations and are used here for reference purpose only.
Now as we notice, we can see that the Numerator in both Precision and Recall is the same: True
Positives. But in the denominator, Precision counts the False Positives while Recall takes False
Negatives into consideration.

Let us ponder… Which one do you think is better? Precision or Recall? Why?

Which Metric is Important?

Choosing between Precision and Recall depends on the condition in which the model has been
deployed. In a case like Forest Fire, a False Negative can cost us a lot and is risky too. Imagine no alert
being given even when there is a Forest Fire. The whole forest might burn down.

Another case where a False Negative can be dangerous is Viral Outbreak. Imagine a deadly virus has
started spreading and the model which is supposed to predict a viral outbreak does not detect it. The
virus might spread widely and infect a lot of people.

On the other hand, there can be cases in which the False Positive condition costs us more than False
Negatives. One such case is Mining. Imagine a model telling you that there exists treasure at a point
and you keep on digging there but it turns out that it is a false alarm. Here, False Positive case
(predicting there is treasure but there is no treasure) can be very costly.

Similarly, let’s consider a model that predicts that a mail is spam or not. If the model always predicts
that the mail is spam, people would not look at it and eventually might lose important information.
Here also False Positive condition (Predicting the mail as spam while the mail is not spam) would have
a high cost.
Think of some more examples having:

• High False Negative cost

• High False Positive cost

To conclude the argument, we must say that if we want to know if our model’s performance is good,
we need these two measures: Recall and Precision. For some cases, you might have a High Precision
but Low Recall or Low Precision but High Recall. But since both the measures are important, there is
a need of a parameter which takes both Precision and Recall into account.

F1 Score
F1 score can be defined as the measure of balance between precision and recall.

Take a look at the formula and think of when can we get a perfect F1 score?

An ideal situation would be when we have a value of 1 (that is 100%) for both Precision and Recall. In
that case, the F1 score would also be an ideal 1 (100%). It is known as the perfect value for F1 Score.
As the values of both Precision and Recall ranges from 0 to 1, the F1 score also ranges from 0 to 1.
Let us explore the variations we can have in the F1 Score:

Precision Recall F1 Score

Low Low Low

Low High Low

High Low Low

High High High

In conclusion, we can say that a model has good performance if the F1 Score for that model is high.

Let’s practice!
Let us understand the evaluation parameters with the help of examples.

Challenge
Find out Accuracy, Precision, Recall and F1 Score for the given problems.

Scenario 1:
In schools, a lot of times it happens that there is no water to drink. At a few places, cases of water
shortage in schools are very common and prominent. Hence, an AI model is designed to predict if
there is going to be a water shortage in the school in the near future or not. The confusion matrix for
the same is:
Scenario 2:
Nowadays, the problem of floods has worsened in some parts of the country. Not only does it damage
the whole place but it also forces people to move out of their homes and relocate. To address this
issue, an AI model has been created which can predict if there is a chance of floods or not. The
confusion matrix for the same is:

Scenario 3:
A lot of times people face the problem of sudden downpour. People wash clothes and put them out
to dry but due to unexpected rain, their work gets wasted. Thus, an AI model has been created which
predicts if there will be rain or not. The confusion matrix for the same is:
Scenario 4:
Traffic Jams have become a common part of our lives nowadays. Living in an urban area means you
have to face traffic each and every time you get out on the road. Mostly, school students opt for buses
to go to school. Many times the bus gets late due to such jams and students are not able to reach their
school on time. Thus, an AI model is created to predict explicitly if there would be a traffic jam on their
way to school or not. The confusion matrix for the same is:

AI Handbook For Class 10
100% (2)
AI Handbook For Class 10
194 pages
417 Class 10 AI Curriculum (2025-26)
No ratings yet
417 Class 10 AI Curriculum (2025-26)
35 pages
Ai Notes
No ratings yet
Ai Notes
17 pages
Revisiting AI Project Cycle & Ethical Frameworks For All
No ratings yet
Revisiting AI Project Cycle & Ethical Frameworks For All
61 pages
Unit 1 Ai Project Cycle
No ratings yet
Unit 1 Ai Project Cycle
16 pages
CH 1 Notes
No ratings yet
CH 1 Notes
11 pages
Ai 10
No ratings yet
Ai 10
244 pages
Revisiting AI Project Cycle Class 10 Questions and Answers CHP 1 CLASS 10
100% (1)
Revisiting AI Project Cycle Class 10 Questions and Answers CHP 1 CLASS 10
5 pages
Classx-Ai - Unit 1
0% (1)
Classx-Ai - Unit 1
19 pages
Class 9 AI Supplement 1
100% (1)
Class 9 AI Supplement 1
141 pages
Revisiting AI Project Cycle Class 10 Notes
No ratings yet
Revisiting AI Project Cycle Class 10 Notes
6 pages
Class 10 Unit 1
No ratings yet
Class 10 Unit 1
6 pages
Unit 1 Notebook Assignment - 1
No ratings yet
Unit 1 Notebook Assignment - 1
5 pages
Part B - Unit 1 Ai Revisiting Notes - Dpgos
No ratings yet
Part B - Unit 1 Ai Revisiting Notes - Dpgos
2 pages
Cbse - Department of Skill Education: Units
No ratings yet
Cbse - Department of Skill Education: Units
7 pages
Ch-1 Revisiting AI Project Cycle and Ethical Frameworks
No ratings yet
Ch-1 Revisiting AI Project Cycle and Ethical Frameworks
3 pages
Ai Unit 1
No ratings yet
Ai Unit 1
16 pages
PRE BOARD EXAM (2024-25) Class-X Subject - Artificial Intelligence
100% (1)
PRE BOARD EXAM (2024-25) Class-X Subject - Artificial Intelligence
6 pages
Worksheet 1 AI Project Cycle
No ratings yet
Worksheet 1 AI Project Cycle
3 pages
Data Science Class X 417 (Amit Gaur)
No ratings yet
Data Science Class X 417 (Amit Gaur)
14 pages
Unit-1 - Class 10
No ratings yet
Unit-1 - Class 10
3 pages
Kips Brochure
No ratings yet
Kips Brochure
12 pages
417 AI Handbook Class9!74!79
No ratings yet
417 AI Handbook Class9!74!79
6 pages
Python For AI Developers
No ratings yet
Python For AI Developers
45 pages
Aritficial Intelligence Curriculum Sub Code 417 Class 9 and 10
No ratings yet
Aritficial Intelligence Curriculum Sub Code 417 Class 9 and 10
19 pages
Class Xii Ai Worksheet Booklet Part2 2023-2024
No ratings yet
Class Xii Ai Worksheet Booklet Part2 2023-2024
26 pages
Class 9 AI Unit 1 (Smart Home & Smart City)
100% (1)
Class 9 AI Unit 1 (Smart Home & Smart City)
10 pages
X - AI - PA1 - February 2024
No ratings yet
X - AI - PA1 - February 2024
2 pages
Introduction To AI: Foundational Concepts
No ratings yet
Introduction To AI: Foundational Concepts
10 pages
Python Full Stack Project On E-Commerce Website Design
No ratings yet
Python Full Stack Project On E-Commerce Website Design
26 pages
Assignment Session 3 Part B Unit 1
No ratings yet
Assignment Session 3 Part B Unit 1
4 pages
Artificial Intelligence (417) : Class X
No ratings yet
Artificial Intelligence (417) : Class X
204 pages
Class 10 AI 417 Computer Vision
No ratings yet
Class 10 AI 417 Computer Vision
22 pages
X-AI - 417-Guide 2024
No ratings yet
X-AI - 417-Guide 2024
105 pages
10 AI 2024-25 Preboard I Blueprint
100% (1)
10 AI 2024-25 Preboard I Blueprint
1 page
Ai QB 2025
No ratings yet
Ai QB 2025
28 pages
Kendriya Vidyalaya Sangathan, Lucknow Region October Monthly Test (2024-25)
No ratings yet
Kendriya Vidyalaya Sangathan, Lucknow Region October Monthly Test (2024-25)
4 pages
Python Full Stack
No ratings yet
Python Full Stack
18 pages
Ai 417 Sqp-Iii
No ratings yet
Ai 417 Sqp-Iii
8 pages
AI Handbook Notes
No ratings yet
AI Handbook Notes
14 pages
843 Ai Xi
No ratings yet
843 Ai Xi
11 pages
Artificial Intelligence (Cbse Code 417) Sample Questions Paper TERM 1 2021-22
No ratings yet
Artificial Intelligence (Cbse Code 417) Sample Questions Paper TERM 1 2021-22
3 pages
AI Reference File
No ratings yet
AI Reference File
37 pages
Sample Paper AI 2not
No ratings yet
Sample Paper AI 2not
16 pages
Color Theory Latest
100% (5)
Color Theory Latest
58 pages
Sample Paper 1 AI Class 10 For 2024
No ratings yet
Sample Paper 1 AI Class 10 For 2024
7 pages
Artificial Intelligence (417) Class IX
No ratings yet
Artificial Intelligence (417) Class IX
6 pages
Artificial Intelligence (AI) by CHATGPT
No ratings yet
Artificial Intelligence (AI) by CHATGPT
39 pages
AI Course Content
No ratings yet
AI Course Content
3 pages
Cbse - Department of Skill Education Curriculum For Session 2023-2024
No ratings yet
Cbse - Department of Skill Education Curriculum For Session 2023-2024
20 pages
AI For Everyone
No ratings yet
AI For Everyone
23 pages
School Management System Project Proposal
No ratings yet
School Management System Project Proposal
10 pages
417 - AI - SQP Simpal Paper
No ratings yet
417 - AI - SQP Simpal Paper
8 pages
XIInfo Pract Term2352
No ratings yet
XIInfo Pract Term2352
5 pages
Cbse - Department of Skill Education Curriculum For Session 2020-2021
No ratings yet
Cbse - Department of Skill Education Curriculum For Session 2020-2021
4 pages
Cbse - Department of Skill Education: Artificial Intelligence (Subject Code 417)
No ratings yet
Cbse - Department of Skill Education: Artificial Intelligence (Subject Code 417)
4 pages
Cbse - Department of Skill Education Curriculum For Session 2020-2021
No ratings yet
Cbse - Department of Skill Education Curriculum For Session 2020-2021
15 pages
Programmes After 10 + 2 at Amity University Mumbai - 2021: Annexure - 1
No ratings yet
Programmes After 10 + 2 at Amity University Mumbai - 2021: Annexure - 1
4 pages
(Computer Science and Engineering) : University of Engineering & Management (UEM), Jaipur
No ratings yet
(Computer Science and Engineering) : University of Engineering & Management (UEM), Jaipur
35 pages
Ai Project Life Cycle
No ratings yet
Ai Project Life Cycle
5 pages
Acronimos RM
No ratings yet
Acronimos RM
3 pages
Detailed Drawing Exercises: Solidworks Education
No ratings yet
Detailed Drawing Exercises: Solidworks Education
51 pages
Lecture 28 - Digital Image Processing
No ratings yet
Lecture 28 - Digital Image Processing
61 pages
Gradient Echo Pulse Sequences in MRI
No ratings yet
Gradient Echo Pulse Sequences in MRI
9 pages
Yolo1 11
No ratings yet
Yolo1 11
38 pages
ImSeg 10 11 18
No ratings yet
ImSeg 10 11 18
41 pages
Exploring Features in A Bayesian Framework For Material Recognition
No ratings yet
Exploring Features in A Bayesian Framework For Material Recognition
8 pages
IP Final Exam Solutions
No ratings yet
IP Final Exam Solutions
4 pages
Optimising Manufacturing Systems With Ai
No ratings yet
Optimising Manufacturing Systems With Ai
10 pages
Radiology Medical Image Data Ver3
No ratings yet
Radiology Medical Image Data Ver3
14 pages
Road Network Extraction From Satellite Images in Egypt
No ratings yet
Road Network Extraction From Satellite Images in Egypt
195 pages
Unit 4
No ratings yet
Unit 4
57 pages
Towards Real-Time Photorealistic 3D Holography With Deep Neural Networks
No ratings yet
Towards Real-Time Photorealistic 3D Holography With Deep Neural Networks
20 pages
FAVPNG - PNG Image Editor
No ratings yet
FAVPNG - PNG Image Editor
2 pages
Color Sorting Robot
No ratings yet
Color Sorting Robot
38 pages
Twitter Following Exported 1
No ratings yet
Twitter Following Exported 1
31 pages
Cim Part 2
No ratings yet
Cim Part 2
23 pages
Module #1-Making Auxiliary Views: 1. 1. Introduction/Overview
No ratings yet
Module #1-Making Auxiliary Views: 1. 1. Introduction/Overview
5 pages
A Review On Image Retrieval Techniques
No ratings yet
A Review On Image Retrieval Techniques
4 pages
Eye Directive Wheelchair@Fnl-1
No ratings yet
Eye Directive Wheelchair@Fnl-1
21 pages
A QR-code Localization System For Mobile Robots - Application To Smart Wheelchairs
No ratings yet
A QR-code Localization System For Mobile Robots - Application To Smart Wheelchairs
6 pages
Man Overboard and PPE Detection Using Cameras and Thermal Cameras
No ratings yet
Man Overboard and PPE Detection Using Cameras and Thermal Cameras
10 pages
Group 2 Report - CCTV
No ratings yet
Group 2 Report - CCTV
10 pages
Vision Assisted Pick and Place Robotic Arm Guided by Image Processing Concepts For Object Sorting System
No ratings yet
Vision Assisted Pick and Place Robotic Arm Guided by Image Processing Concepts For Object Sorting System
28 pages
Sixth Sense Technology Seminar Report
No ratings yet
Sixth Sense Technology Seminar Report
26 pages
Attendance Management System Using Image Andvoice Recognition
No ratings yet
Attendance Management System Using Image Andvoice Recognition
5 pages
S1TBX SAR Basics Tutorial
No ratings yet
S1TBX SAR Basics Tutorial
19 pages
4405 13625 1 Ed
No ratings yet
4405 13625 1 Ed
5 pages
Edge Detection Techniques For Image Segmentation - A Survey of Soft Computing Approaches PDF
No ratings yet
Edge Detection Techniques For Image Segmentation - A Survey of Soft Computing Approaches PDF
5 pages
Touchpad Plus Ver. 4.0 Class 7
From Everand
Touchpad Plus Ver. 4.0 Class 7
Nidhi Gupta
No ratings yet