PYTHON

ARTIFICIAL LEARNING
Hove you ever thought the
different devices you ore using
with the prefix smort can be smarter
than you? Yes oil the smart devices like
smartphones, smort tv's, or even smort
homes ore oil machines that ore
created or more specifically
programmed by humons
to think like humans i.e.
Aritificial intelligence. So do
you wont to learn the intelligence
behind these and even leorn to use
these intelligences in your favour,
then you've got covered!

This book will help you to from the

storting point of whot artificial intelligence
or Al is; to oil the way to how to make
intelligences artificially!

Rahul Mula
 Artificial Intelligence with Python
by Rahul Mula
© 2021 Artificial Intelligence with Python
All rights reserved. No portion of this
book moy be reproduced in any form
without permission from the copyright
holder, except os permitted by U.S.
copyright low.

All the programs written in this book

ore tested and verified by the author.
Cover by Rohul Mulo
Template from freepik.com

ISBN : 979-8-58-957189-9
 The reader is assumed to have a knowledge of
advanced level programming and data science
skills. The book is for the beginners at Artficial
intelligence not for beginners at programming. You
can check the following books to catch up now:
http://bit.ly/PyBooks

Python for
Complete Beginners

Learn programming
from the basics
with Python and
become from a
beginner to an
advanced level
programmer

Basic
conncepts

All the basic

programming
concepts are
covered in this
book to help
complete
beginners
 Data Science
with python

Data science is
on of the most
in-demand skills
nowadays and python
has made it easy
to learn

Operating
data

Learn to perform
tasks like
data processing,
cleansing,
analysing and
visualizing

MACHINE
LEARNING
Machine Learning WITH PYTHON
MACHINE LEARNING
TH PYTHON

with Python FARN DIFFERENT MACHINE

ARNING ALGORITHMS REGRESSION,
OSSIFICATION, CLUSTERING, ETC
USE OF INFOGRAPHICS, VISUALS •
AND CODE SNIPPETS TO REMOVE THE
BOREDOM OF LEARNING

Beginners guide • EVEN IF THIS A MACHINE

LEARNING BOOK; MATHEMATICS, DATA
SCIENCE 8 OTHER KNOWLEDGE
REQUIRED FOR ML ARE INCLUDED TOO
to machine WITH DATASETS ft

learning MACHINE LEARNING

applications • ALL THE DATASETS USED AS EXAMPLFS
HAVE LINKS OF THEM PROVIDED
IN THE TEXT SO YOU CAN
EASnJ Ul> "ITH TUT0RIALS

SCIKIT
learn Lti>S TEXT
MORECODE

Models &
Algorithms

Learn about
different models
and algorithms
 We are living in an age where the machines or
devices like smartphones, computers, etc. are
becoming intelligent i.e. they can do their work
with much human interaction or guidance. THe best
example is your smartphones! So why are they
called ''smart-’phones? Well, earlier we had
telephones that only satisfied the function of
calling someone where you had to input the number
through keys and then the call is connected to the
person you want. They were considered
telephone but as techonligies grew
that telephone which only satisfied
the function of calling someone has
evolved into a smartphone that has
barely any tasks which it can't do.
Calling someone is the same function
which is satisfied by both but the input has
changed a lot, telephones required exclusive human
guidance i.e. dailing the numbers but now whoever
you want to call just think out loud and your
smartphone will call the person you want, look how
intelligent the device has became. But that's not
the end of it there is still more room for growth!
Decades ago no one thought that we will just tell
the phone to call this person and it
will do so. In the field of
artificial intelligence this is
called natural language processing
which we will learn about
This book will help you to
understand about the steps and
methods in the creation of
intelligence artificially. You may find these a
bit hard or too difficult if you're a complete
beginner but don't worry these are created by
human and you're likely to be a human too so don't
loose to the artificial intelligences and be the
smartest out there. I'm also a student so I know
what were the fears of a beginner, everything just
seems too difficult but that is the thought which
you have to go about. It is easy for the
knowledgeable to go about every thing but hard for
the beginners because even if they have good
programming skills but these things just seem out
of place, so keeping those in mind
I will explain every piece of code so nobody feels
left out! But I also want your cooperation to be
persistent. I will not guarantee you that youJll
become a pro at it immediately but I will make
sure you become a pro definitely!
r MACHINE LEARNING
03 I
CHAPTER | WV
• What is machine learning?
• Uses of machine learning
• How do machines learn?
1
PANDAS

1
ii
|>;l • Features of Pandas Library
CHAPTER pandas ’Series
• Data Frames
 MATPLOTLIB

k
CHAPTER
. «.ru ** Features of matplotlib
matplvthb .Datavisualizationp
• PyPlot in matplotlib 1
SCIKIT LEARN

k
08
dAPTER
CHAPTER
• Features of Scikit-learn library
• How to work with data?
• Why use Python?
1

09 A
TYPES OF MACHINE LEARNING

CHAPTER
HAPTER Q O
• Supervised learning
• Unsupervised learning
• Deep learning 1
■? IMPORTING DATA

CHAPTER
I 1
I I
I
I I
• Importing CSV data
• Importing JSON data
• Importing Excel data
1
£ DATA OPERATIONS

CHAPTER
’ NumPy operations
• Pandas operations
• Cleaning data 1
DATA ANALYSIS 8 PROCESSING

k
CHAPTER
• Data analytics
• Correlations between attributes
• Skewness of the data
1
 DATA VISUALIZATION
13
CHAPTER
• Plotting data
• Univaritae plots
• Multivariate plots
1

CLASSIFICATION

CHAPTER
• Decision tree
• Linear regression
• Naive Bayes
1
 NEURAL NETWORKS
• Creating neural networks artificially
• Perceptor classifier
CHAPTER
• Layers of networks

DEEP LEARNING

k
21
CHAPTER
• Deep learning v/s Machine learning
• Deep learning models
• Regression using deep learning
1
 ARTIFICIAL
U1 INTELLIGENCE

• What is Al?
• How Al works?
• Applications
of Al

o
01
_______ /
ARTIFICIAL INTELLIGENCE INTRODUCTION

□--------------------------------- J
What is Artificial Intelligence?
//

Al is going to be the best

or the worst thing for
humanity
-Stephen Hawking

As mentioned in the preface, the technologies are

becoming more and more improved due to the
hardwork of a lot of peoples which has given rise
to another concept of improving these technologies
to new extent without hardwork i.e. smartwork.
What if these improvements are automated i.e. what
if the machines think like humans and perform
functions? Yes, as of now it is a lot possible
which led to artificial intelligence. Well hold
on, it is all good that machines
will become intelligent not as much
as us but to an extent.
Self-driving cars, your virtual
assistant in your smartphone or all
the devices with the prefix fsmartJ
are all intelligences created
artificially. So as of now, what is the image of
Al in your mind? You may get the idea that Al
creating machines that thinks, learns and behaves
like humans; which is correct to an extent but if
you just revise the word artificial intelligence
i.e. artificial means man-made and intelligence is
something incalculabel which every human posses
but how do we create these intellegences
artificially?
 15 Al INTRODUCTION

We train the machines or algorithms or models or

etc. with sample data which is called machine
learning. Though machine learning and artificial
are dependent topics but it doesn't applies
everytime that's why they are seperate
So once again, what do you think of 'artificial
intelligence'? Is it simply an intelligence
created artificially? As mentioned
intelligence is incalculable, what we
might think is the limit of Al may be
not tomorrow. At some point a simple
calculator that can perform mathematics
was considered great but now it's just an
app that stays in the corner of your
homescreen. Al is evolving rapidly and it is not
wrong to say that at somepoint Al may surpass
human intelligence. So the definition of Al is not
distinct but human intelligence but the case is
it's artificial

How Al works?
So how does your brain works? For example how do
you talk? Since we are born we learn multiple
words from our surrounding people and speak the
same. It is a simple method of learning and
performing tasks. The same is for the Al systems.
We train them and they perform but in some cases
they even learn on their own. So how do even they
learn, we know that we have brains with complex
network of nerves but how do machines learn? Well
there are a lot of approches used by machines to
learn. Machine learning algorithms, deep learning
algorithms, neural networks, etc. are used to
create Al systems. But learning isn't a prominent
part of Al because Al focuses more on performing
tasks rather than learning
 r*---------------------------------------------------------------------

16 Al INTRODUCTION

Applications of AI
As mentioned earlier AI is equivalent to human
intelligence so anything that humans can perform,
AI systems can too or in development where not.
There are lot of AI applications around you like:
• The games like chess or cards where you play
against the computer or mobile. AI is used
to learn all the possibilities
in the game and perform
them like, what do if the
user does this or does that

• Everyonce in a while you

may have used the voice search on
• 11 In
your smartphone i.e. called speech
recognition where the smartphone
understands your voice and shows
results on the basis of that

• Nowadays productions in many factories are

automated i.e. performed by the machines
which are programmed to
a particular task
 ■x
02I/PES OF
• Al systems
• Al algorithms
• Robotics

J
/------- Q A A

02 TYPES OF ARTIFICIAL INTELLIGENCE

\______ 7 i___________________________________ J

We do a lot tasks which can be classified into

daily-basis tasks like talking, walking, etc. and
important tasks or jobs like production,
development, etc. All these tasks can be performed
by Al but with different algorithms, systems, etc.
We can perform all of the tasks but Al is on the
stage where it can only handle one particular
task. All these can be broadly classified into
learning, reasoning, approach and decision. So
let's study the different types of artificial
intelligence

Al Systems
There are many Al systems like location tracking
systems, clinical systems, that are called Expert
systems. These systems are used in complex areas
like healthcare which requires high intelligence.
They can conclude some results, predict some
results, explaining, etc.
These systems have three essential parts namely
Knowledge base, Inference engine and UI(User
Interface)
 -------------------------------------------------------------

19 TYPES OF Al

The knowledge base is what as it's name suggests.

It can be refered to a dataset used by the
Inference engine to performance tasks for the
input. The higher the qulaity of the knowledge
base the better the system. The knowledge bases
consists of a lot of if-else rules which are
defined by the data scientists who performs
researches on the subject of the system from
experts. They are also called Data Engineers
The next comes the Inference engine which uses
the following logic to understand a situation:
• What will happen?

Condition 1

Condition 2

Condition 3

If any of the above conditions happen

If condition 1 & 2 both happens then it results

Result 1 and if condition 3 happens then it
results Result 2. There can be another situation
where if all the conditions happen then it will
result Result 3
 Z—• -------------------------
20 TYPES OF Al

• Or it can use the inverse logic like, Why this

happened?

Result l

Result 2

Here we want know which conditions resulted in

Result 1 & 2

So it resulted in Result 1 because of condition 1

& 2 and Result 2 because of condition 3
Finally we have the User Interface from where the
user provides the input values and the inference
engine gets the input and performs conclusion,
explanation, decision, etc. based on the Al system
and pass the output to the user through the UI
So how are these expert systems are created?
Expert systems creation requires the followings:
• Developement environment whici consits of
workstations, minicomputers, mainframes, large
databases and high level symbolic programming
languages like LISP (List Programming), PROLOG,
etc.
• Tools like powerful editors and debugging tools,
in-built model definitions, knowledge
representations, etc.
• Shells i.e. an expert system without knowledge
base which can be said to be a template for
an expert system which provides inference
engine, knowledge inquistion. Like 3ESS (lava
Expert System Shell)
 r*----------------------------------------------------
TYPES OF Al

Al Algorithms
Algorithms are widely used to solve a lot of
problems like searching. In Al these algorithms to
solve a specefic problem. For example let's
consider an example of an algorithm that will play
tic toe with the user. The algorithm will have the
following attributes:
• Search Space, the set of possible *'
solutions i.e. to decide which
box to use out of 9
X o
• Start state, the state from o
where the search begins i.e. X
the empty boxes if you have the
first turn or one filled box if you
have the second turn
• Goal test, is a function which checks the
current state with the state which is needed
to be achieved i.e. whether three crosses or
circles are in row or not
• Problem instance, it is the intial state and the
Goal
• Problem Space graph, is the representation of
the problem state i.e. we need three crosses or
circles in a row
• Depth of a problem, shortest path to the goal
from the intial state i.e. how to align three
crosses or circles
IP

X
o X
IQ

O
 22
--
TYPES OF Al

Properties of a search algorithm:

• Completeness, a search algorithm is said to be
complete if it guarantees to return a solution
if at least any solution exists for any random
input
• Optimality, if a solution found for an algorithm
is guaranteed to be the best solution (lowest
path cost) among all other solutions, then such
a solution for is said to be an optimal
solution
• Time Complexity, time complexity is a measure of
time for an algorithm to complete its task
• Space Complexity, it is the maximum storage
space required at any point during the search,
as the complexity of the problem

There are two types of search algorithms i.e.

uniformed and informed. Uninformed search doesn't
have any prior knowledege of the search like the
location or state of the goal. The followings are
the uniformed search algorithms:
• Breadth-first search
• Uniform cost search
• Depth-first search
• Iterative deepening depth-first search
• Bidirectional Search
except for the Depth-first search every algorithm
has Completeness and optimality. These are also
called Brute-force strategies or blind search
But when solving problems with a lot of
possibilities we need to use the informed searches
or heuristic strategies i.e. we need to provide
the attributes like the state of the Goal, etc.
The followings are some examples:
• Greedy Best first search
• A* search
but sometimes they cannot become optimal
Robotics
23
--
TYPES OF Al

Robots are mechinacal devices

with Al applications that
operate in the physical world.
They have different inputs and
tasks than the Al programs.
These technologies can be used
to in places humans can reach
like a robot named Nomad, is moving
in Antartica in search of meteorites or the Mars
rover that is in place where human haven't reached
physically ever. Except for these robots can
perform tasks like produnction in factories with
accuracte and more precise results. As they
operate in real world they make uses of the Al
programs like speech recognition or computer
vision to get input. They also have sensors to get
input from their environment

Natural Language Processing

We use different programming languages to command
computers to perform different tasks. But what if
we do the same using human languages like English?
NLP or Natural Language
Processing refers to
communicating with devices in
native languages. These types
of Al understands the input
text by different methods like
lexing. We can use these
intelligences in feilds like
Text analysis, Text Generation,
Translation etc. You may have used translations
apps, they break(stemming) the input into
different nouns, verbs, etc. and change them into
desired language and reform them into sentences
 r*----------------------------------------------------
TYPES OF Al

Speech
Speech is a widely used Al for speech
recognition, speech generation, etc.
Voice search is one of the
applications of speech recognition.
NLP and Speech can be used
together to create applications 4
that can understand human
language and perform different
tasks like translation and then
convert it back into speech

Anomaly detection
Anomaly detection is a method which can be used
to monitor unusual behaviours or errors in a
dataset or system. It is used in many factories to
monitor the status of machines and detect whether
everything is running smoothly or not

Computer Vision
Computer vision is an Al that is used to help the
softwares or programs to use images as input and
perform different actions like image analysis,
object detection, Image classification, face
recognition, etc. There are a lot of application
used for special facial effects i.e. face
recognition. Computer vision
can be combined with Robotics
to create devices for
monitoring tasks at factories.
Computer vision has application
like OCR i.e. Optical Character
Recognition which is used to
recognise handwritten text or
printed texts
 25
----------
TYPES OF Al

Neural Networks
Inspired from the human neural
networks; a new research area of
Al came into existence i.e.
Artificial Neural Networks(ANN).
Like neurons they create
different nodes and take the
input and pass the input to
different nodes which analyze.,
classify, etc. and create the output.
But the downside to these networks is
that, it doesn't provide any resoning why it came
to that decision or the working of ANN are a
mystery

Deep learning
Deep learning models are based on Artificial
Neural Networks (ANN), or more specifically
Convolutional Neural Networks (CNN)s. There are
several architectures used in deep learning such
as deep neural networks, deep belief networks,
recurrent neural networks, and convolutional
neural networks.
These models are used to solve
problems like:
• computer vision, image
classification, etc.
• bioinformatics
• drug design
• games, etc.
Deep learning is entirely a
concept in itself that it is
completely different type of machine learning or
different from it, we will not discuss it in much
detail but let's see what is it and how do it
solves problems.
 26
----------
TYPES OF Al

Deep learning requires a lot of data and

computational power. But nowadays high performance
computing is available to us. Let's consider an
example where the deep learning model tells us
about whether an animal is horse or not. The net­
work consists of large amount of horse photos as
data and analyze them and try to extract patterns
from it like horns, color, saddle, eyes, etc.

the neural networks came to conclusion whether the

animal his horse or not, but how did it reach the
conclusion is unknown. The reasoning cannot be
obtained from deep learning models that's it is
also considered as black box
 ■x

MACHINE
UO LEARNING
• What is Machine
Learning?
• Use of Machine
Learning
• How Machines
Learn?
0
o
 A

03 MACHINE LEARNING
\________/ □------------------
What is Machine Learning?
//

Data is what you need to

do ANALYTICS,
Information is what you
need to do BUSSINESS.

Commonly referred to as the “OiL of the 21st

century" our digital data carries the most
importance in the field. It has incalculable
benefits in business, research and our everyday
lives.
Machine Learning is the field of computer science
where machines provide meaning to the data like we
humans do. Machine Learning is an type of
artifical intelligence which finds patterns in raw
data through various algorithms and perform
predictions like humans. Machine Learning also
means machines learn on their own. To better
understand it think of a new born child and refer
it to a machine learning model. The parents cannot
teach them everything that's why they leave them
to schools which can be you in this case with the
machine learning model. The school has text books,
tests, etc. to help you learn on your own which
are algorithms, data, etc. in this case.
 z
29 MACHINE LEARNING
---'•
Uses of Machine Learning
Organizations are investing heavily in
technologies like Artificial Intelligence, Machine
Learning and Deep Learning to get the key
information from data to perform several
real-world tasks and solve problems. We can call
it data-driven decisions taken by machines,
particularly to automate the process. These
data-driven decisions can be used, instead of
using programing logic, in the problems that
cannot be programmed inherently. The fact is that
we can't do without human intelligence, but other
aspect is that we all need to solve real-world
problems with efficiency at a huge scale. That is
why the need for machine learning arises.
Followings are some of it's applications in real
world:

Forecasting weather of
a day beforehand through finding
patters of weather in the data
of weather of previous days

Predicting the future

prices of stocks in
stock market

suggesting a product to a
customer in an oniine store
according to the users previous
search terms
 Z—• F*-----------------------------------------------------------

30 MACHINE LEARNING

How do machines learn?

So what magic happens that machines learns like
us and perform tasks? LetJs understand that by an
example. LetJs say you are a new computer dealer.
You have very basic experience in it. So you ask
another dealer and obtain information. You
summarized the following points to be important
like the processor cores, ram and gpu. The dealer
tells you these and you learn in return. Then you
are provided with following data about 8gb ram of
different brands:

3500

3000
o'
c
| 2500
<u
| 2000
§
1500

----------------------------- >
60 65 70 75 80 85
Price

By observing the data we can tell that the price

increases with the increase in frequency speeds.
You understand an simple logic behind the data.
Then if you get ram with the following
specifications you can tell itJs price, like a ram
with 1666 Mhz of frequency so its price is 60
 31 MACHINE LEARNING
-

But what if you get an frequency that you don't

have record of like 2600Mhz? Then you have to
learn how to decide the prices. We start to find a
way to calculate with the given data. We assume
that their is a linear relationship between the
two. We define the relationship as a straight line
as shown below:

Price
Now we can use the line as reference and predict
values. SOj for 2600Mhz the cost will be about 77
So how do we draw out the line. We follow the
formula cost - a + b * Mhz, but what are a & b?
a and b are parameters of the straight line which
you don't need to sweat about.

1500
 z—• /----------------------------
32 MACHINE LEARNING
-

Likewise in machine learning, the machine i.e.

computer learns the patters or relations in the
data through algorithms and predict values when
new value is asked.
So will there be no errors? Definitely it will
predict wrong than the actual answer. We also do
many mistakes but learn from our mistakes or
change our tutor if the result stays negative.
Machine learning models too learn from their
mistakes and change algorithms when results are
not improved
 ■x

<\/| PANADAS
MH LIBRARY
• Features of Pandas
library i'll
• Series I'1
• Dataframes pandas

o J
 A

04 PANDAS library
________________________________________ >

Pandas
Data science requires high-performance data ma­
nipulation and data analysis., which we can achieve
with Pandas Data Structures.Python with pandas is
in use in a variety of academic and commercial
domains, including Finance, Economics, Statistics,
Advertising, Web Analytics, and more. Using
Pandas, we can accomplish five typical steps in
the processing and analysis of data, regardless of
the origin of data - load, organize, manipulate,
model, and analyse the data.

Key features of Pandas library

We can achieve a lot with Pandas library using
its features like:
• Fast and efficient DataFrame object with
default and customized indexing.
• Tools for loading data into in-memory data
objects from different file formats.
• Data alignment and integrated handling of
missing data.
 35 PANDAS LIBRARY
---'•
• Label-based slicing, indexing and subsetting
of large data sets.
• Columns from a data structure can be deleted
or inserted.
• Group by data for aggregation and
transformations.

Series
Pandas deals with data with itJs data
structures known as series, data frames and panel.
Series is an one-dimensional array like structure
with homogeneous data. For example, the following
series is a collection of integers

10 17 23 55 67 71 92
As series are homogeneous data structure, it can
contain only one type of data [here integer]. So,
we conclude that Pandas Series is:
• It is a homogeneous data structure
• Its size cannot be mutated
• Values in series can be mutated

Data Frames
DataFrame is a two-dimensional array with
heterogeneous data.

Day Sales
Monday 33
Tuesday 37
Wednesday 14
Thursday 29
 Z—• F*-----------------------------------------------------------

36 PANDAS LIBRARY
—

The data shows the sales of certain product for 4

days. You can think of Data Frames a container for
2 or more series. So, we conclude that pandas data
frames is:
• It can contain heterogeneous data
• Its size is mutable
• ALso its data is mutable.

We will use Pandas series and data frames a lot

in the future lessons, make sure to go through the
lesson again and get the grasp of it.

r
Key Points
• Pandas library is a high performance data
manupilation and data analysing tool.
• Pandas data structures include series and
data frames
• Series is a 1-Dimensional array of
homogeneous data, whose size is immutable
but values in a series are mutable.
• Data Frames is a 2-Dimensional array of
heterogeneous data of 2 or more series,
whose size and data are mutable.
 ■x

NUMPY
UD PACKAGE
• Features of NumPy
• ndarray Objects
• List vs. ndarrays

o J
/• A

k
05 ______ /
NUMPY PACKAGE
________________________________________ J

NumPy
NumPy is a Python package which stands for
'Numerical Python'. It is a library consisting of
multidimensional array objects and a collection of
routines for processing of array.

3D array

2D array
ID array
5.2 3.0 4.5

7 2 9 10 9.1 0.1 0.3

axis 0 axis 1

shape: (4,) shape: (2, 3)

Key features of NumPy

NumPy is powerful that consists of many features
like :
• Mathematical and logical operations on arrays.
• Fourier transforms and routines for shape
manipulation.
• Operations related to linear algebra. NumPy
has in-built functions for linear algebra and
random number generation.
• NumPy ndarrays are much much faster than
Python Built-in lists and less memoray
consuming.
• Most of the part that requires fast
computation are written C and C++
 PACKAGE

ndarray objects
NumPy aims to provide an array object that is up to
50x faster that traditional Python lists. The array
object in NumPy is called ndarray, it provides a lot
of supporting functions that make working with
ndarray very easy. Arrays are very frequently used in
data science, where speed and resources are very
important.
In NumPy, we can create 0-D,l-D,2-D and 3-D
ndarrays.

0-D (33)

1- D ([11,27,18])

2- D ([ 3, 5,6],
[5, 7,11])

3- D ([ 5,8,19],
[ 6,9,10],
[4,1,11])

In breif ndarrays or n-dimensional arrays are:

• It describes the collection of items of the
same type.
• Items in the collection can be accessed using a
zero-based index.
• Every item in an ndarray takes the same size of
block in the memory.
• Each element in ndarray is an object of data-type
object (called dtype). Any item extracted from
ndarray object (by slicing) is represented by a
Python object of one of array scalar types.
 PACKAGE

Lists vs. ndarray

In Python we have lists that serve the purpose of
arrays, but they are slow to process. NumPy aims to
provide an array object that is up to 50x faster
that traditional Python lists.

Lists ndarrays
• List is an array of • ndarray is an array of
heterogeneous objects homogeneous objects
• List arrays are • ndarrays arrays are
stored in different stored in one
places in the memory continuous place in
which, makes it slow the memory which,
to process data. makes it faster to
• Lists are not process data.
optimized to work • ndarrays are
with latest CPU's optimized to work
• A 1-Dimensional with latest CPU's
List • A 1-Dimensional
ndarray

[ 'A',56,67.05] ([ 12, 17, 25])

Lists arrays
memory toe
-12044567

memory too
-12044568

PyObject_Head
"Length
items

List arrays memory

allocation
_________________ /
 41 NUMPY PACKAGE
-

ndarrays

PyObject_Head 1
data 2
3
“7

dimensions
4
strides
5
f------------------------------------ s
6 ndarrays memory
7 allocation
\______________________ z

You can clearly understand why the built-in list

arrays are slower than ndarrays.
To accelerate and process data much faster we will
use NumPy in future lessons., make sure to geta hold
of it.

Key Points
• NumPy stands for Numerical Python, which
is a Python Package used for working with
arrays.
• It also has functions for working in domain
of linear algebra, fourier transform, and
matrices.
• ndarrays or n-dimensional arrays are
homogeneous arrays, which are optimized
for fast processing.
• ndarrays also provide many functions that
makes it suitable to work with data
 ■x

AQ SCIPY
MO PACKAGE
• Features of SciPy
• Data Structures 1

• SciPy
Sub-Packages

o
—--------------------------------------------------- A
06 SCIPY PACKAGE
k
SciPy
The SciPy library of Python is built to work with
NumPy arrays and provides many user-friendly and
efficient numerical practices such as routines for
numerical integration and optimization. Together,
they run on all popular operating systems, are
quick to install.
A—
in [1]: #Import packages
from scipy import integrate
import numpy as np

def my_integrator(a,b,c):
my_fun = lambda x: a*np.exp(b*x)+c
NumPy
y,err = integrate.quad(my_fun,0,100)
print(’ans: %1.4e, error: %1.4e’ % (y,err
return(y,err) /

#CaLL function
my_integrator (5,-10,3)

ans: 3.0050e+02, error: 4.5750e-10

Out[l]: (300.5, 4.574965520082099e-10)

\_______________ -_________________________

Key features of SciPy

SciPy combined with NumPy results a powerful tool
for data processing with features like:
• The SciPy package contains various toolboxes
dedicated to common issues in scientific
computing. Its different submodules correspond
to different applications, such as
interpolation, integration, optimization, image
processing, statistics, special functions, etc.
• SciPy is the core package for scientific
routines in Python; it is meant to operate
efficiently on NumPy arrays, so that numpy and
scipy work hand in hand.
• SciPy is organized into sub-packages covering
different scientific computing domains, which
makes it more efficient
 Z—•
44 SCIPY PACKAGE

Data structures
The basic data structure used by SciPy is a mul­
tidimensional array provided by the NumPy module.
NumPy provides some functions for Linear Algebra,
Fourier Transforms and Random Number Generation,
but not with the generality of the equivalent
functions in SciPy. Except for these, SciPy offers
Physical and mathematical constants, fourier
transform, interpolation, data input and output,
sparse metrics, etc

Dense Matrix Sparse Matrix

1 2 31 2 9 7 34 22 11 5 1 3 9 3

11 92 4 3 2 2 3 3 2 1 11 4 2 1

3 9 13 8 21 17 4 2 1 4 1 4 1

8 32 1 2 34 18 7 78 10 7 8 3 1

9 22 3 9 8 71 12 22 17 3 9 1 17

13 21 21 9 2 47 1 81 21 9 13 21 9 2 47 1 81 21 9

21 12 53 12 91 24 81 8 91 2

61 8 33 82 19 87 16 3 1 55 19 8 16 55

54 4 78 24 18 11 4 2 99 5 54 4 11

13 22 32 42 9 15 9 22 1 21 2 22 21

^Use of Sparse matrix)

SciPy sub-packages
As we already know, SciPy is organized into
sub-packages covering different scientific comput­
ing domains, we can import them according to our
needs rather than importing the whole library.
The following table shows the list of all the
sub-packages of SciPy :
[next page]
 —• *
45 SCIPY PACKAGE

scipy.constants Mathematical constants

scipy.fftpack Fourier transform
scipy.integrate Integrate routines
scipy.interpolate Interpolation
scipy.io Data input and output
scipy.linalg Linear algebra routines
scipy.optimize Optimization
scipy.signal Signal processing
scipy.sparse Sparse matrices
scipy.spatial Spatial data structures
scipy.special Special mathematics
scipy.stats Statistics

Key Points
• SciPy Package is a toolbox which is used
for common scientific issues.
• SciPy together with NumPy creates a
dynamic tool for data processing.
• Along with NumPy functions, SciPy provides
a lot of functions to perform different
tasks with ndarrays.
• SciPy is divided into sub-packages
determined for different tasks.
 ■x

r\~f MALPLOTLIB
\Jf LIBRARY

’SSOf matplstlib
• Data Visualization
• PyPlot in Matplotlib

o
 A

07
X______ 7
MATPLOTLIB LIBRARY
________________________________________ J

MatPlotLib
Matplotlib is a python library used to create 2D
graphs and plots by using python scripts. It has a
module named pyplot which makes things easy for
plotting by providing feature to control line
styles, font properties, formatting axes etc.

50

40

30

■ ■ +
20

10

Sun Sat Thur Fri

X

Key features of MatPlotLib

Matplotlib is the best choice for data
visualization because of its features like:
• It supports a very wide variety of graphs and
plots namely - histogram, bar charts, power
spectra, error charts, and many more.
• It is used along with NumPy to provide an
environment that is an effective open source
alternative for MatLab.
• Using its PyPlot module, plotting simple
graphs or any other charts is very easy.
 43 MATPLOTLIB LIBRARY

Data Visualization
Data visualization is the graphical representa­
tion of information and data. By using visual
elements like charts, graphs, and maps, data visu­
alization tools provide an accessible way to see
and understand trends, outliers, and patterns in
data.

In the world of Big Data, data visualization

tools and technologies are essential to analyze
massive amounts of information and make
data-driven decisions. Data visulaization helps us
to view data in a graphical or more interesting
way rather than viewing a big chunk of numbers in
a uniform line.
We will process, analyze and then visualize our
data, if we don't visualize our data, it loose a
lot of impact as it will in the form bar graphs,
pie charts, etc.
 MATPLOTLIB LIBRARY
>

PyPlot in Matplotlib
matplotlib.pyplot is a collection of functions
that make matplotlib work like MATLAB. Each pyplot
function makes some change to a figure: e.g.,
creates a figure, creates a plotting area in a
figure, plots some lines in a plotting area,
decorates the plot with labels, etc.
To test it yourself, jump to lupyter Notebook and
start of by importing the matplotlib. pyplot
module.

In [ ]: import matplotlib.pyplot as mplt

To plot a simple graph, use the plot function

and pass a list, and then use the show function
to view the graph

In [1]: import matplotlib.pyplot as mplt

mplt.plot([1,3,6,9])
mplt.show()

—i--------------------- 1----------------------- 1---------------------- 1---------------------- 1---------------------- 1--------------------- 1—

0.0 0.5 L0 15 10 15 3.0
\____________________________ -__________________________ /

We have successfully plotted our graph with some

random values in a list.
If we want we can name x and y axis using the
xtabet and ylabet repectively
 f
50 MATPLOTLIB LIBRARY

In [2]: import matplotlib.pyplot as mplt

mplt.plot([1,3,6,9])
mplt.xlabel('X_Axis')
mplt.ylabel('Y_Axis*)
mplt.show()

The graph has solid blue line, we change it's

color and the line style by passing another
argument to the plot function like, 'ro' for 'r'
red and 'o' circles.

In [2]: import matplotlib.pyplot as mplt

mplt. plot( [1,3,6,9],Cro ’))
mplt.xlabel('X_Axis')
mplt.ylabel('Y_Axis')
mplt.show()

6
in

>
4

0.0 0.5 L0 L5 2.0 2.5 30

XAxis

The letters and symbols of the format string like

'ro' are from MATLAB, and you concatenate a color
string with a line style string. There are many
symbols for different shapes and colors like, 'b-'
for blue solid. You-’ll find all the symbols for
different color and shapes in the following list:
 MATPLOTLIB LIBRARY

line and shape styles

r
- Solid line
-- Dashed line
• Dotted line
Dash-Dot line
'o' Circle
Plus sign
•*• Asterisk
Point
'X' Cross
Horizontal line
■ i ■ Vertical line
' s' Square
' d' Diamond
1 A 1
Upward-pointing triangle
'V' Downward-pointing triangle
'>' Right-pointing triangle
'<' Left-pointing triangle
'P' Pentagram
' h' Hexagram
k_________
color styles
r
y yellow
m magenta
c cyan
r red
g green
b blue
w white
k black
k________________
 Z—• *
52 MATPLOTLIB LIBRARY

Except for the color and line & shape style we

have a lot of editibility on the plotted graphs,
you can learn those in the seperate book dedicated
for data science or data visualization.

r
Key Points
• MatPlotLib is a library used for
visualizing our data using itJs MATLAB
like functions
• MatPlotLib-’s PyPlot module makes it easier
to plot data, with full control over
color, line & shape, font, axis-labels,
etc.
• It supports wide range of graphs and plots
like, histogram, bar graphs, pie charts,
and even 3-D graphs.
• MatPlotLib is the best library for data
visualization
rQr ■x

r^QSCIKIT LEARN
UO LIBRARY
• Features of
Scikit learn
library team

o J
 A
08
______ /
SCIKIT LEARN LIBRARY
______________________ J
Scikit Learn or Sklearn
Scikit-learn or Sklearn is the most useful and
robust library for machine learning in Python. It
provides a selection of efficient tools for
machine learning and statistical modeling
including classification, regression, clustering
and dimensionality reduction via a consistence
interface in Python. This library, which is
largely written in Python, is built upon NumPy,
SciPy and Matplotlib.

Features of Sckikit Learn

Scikit-learn focuses on modelling data. The
followings are the most popular groups provided by
the library:
• Supervised learning algorithms, like Linear
Regression, Support Vector Machine (SVM), Decision
Tree etc. are the part of scikit-learn

$
• Unsupervised learning algorithms like
clustering, factor analysis, PCA (Principal
Component Analysis) to unsupervised neural
 Z—• F*------------------------------------------------------------------

50 SCILKIT LEARN LIBRARY

- —

• Cross Validation, Dimensionality Reduction,

Ensemble methods, Feature extraction, Feature
selection are also features of scikit learn that
are used to check the accuracy of supervised
models, reducing the number of attributes in a
data, combining the prediction of multiple
supervised models, extract features and identify
useful featurews in adata, respectively.

✓
 TYPES OF
kJ <3 MACHINE LEARNING
Supervised leorning
• Unsupervise leorning
• Deep learning
• Reinforcement
leorning
• Deep reinforcement
learning
o-------------
 A

09
\______ /
TYPES OF MACHINE LEARNING
_______________________J
In the previous lessons we learned about the
various libraries and packages used in the process
of machine learning. Now letJs look at the types
of machine learning

Types of machine learning

The followings are the different type of machine
learning:

reinforcement
Learnifig Supervised
Learning

Ur]supervised
Learning
deep
reinforcement
Learnirfg
D^ep Learning

Supervised learning
As it's name suggest in supervised
learning we train a machine learning
model with supervision. We feed the
data, train the model and tell the
model to make predictions. If the
predictions are correct we leave the
model to work on itJs own from there else
we help the model to predict correctly
until it learns so. It is the same as
teaching a child to solve questions at first
until he can solve them on his own.
 58*J^TYpES OF MACHINE learning j

Types of supervised learning

Regression and classification are two types of
supervised machine learning. They can be under­
stood as:
• Regression, is the type of machine learning in
which we feed the model with data like ‘/\3 (input,
i.e. X) has value of 65 (output, i.e. Y), fB' has
value of 66, etc. Based on the given data, the
model learns the relation between the input and
output (here fA' & 65).
Once the machine is trained with sufficient data
we provide a input let's say fC' and let the model
predict the output, but you must know the real
output of that input. You check the prediction
with the real value and check whether it is
correct or wrong. If the predictions are correct
we pass the model. If the predictions aren't
correct we keep training until is reaches a
 Z—• -------------------------------------------- *
59 TYPES OF MACHINE LEARNING
—

Regression inturn have different ypes like linear

and logistic regression, which we will learn in
itJs separate lesson.
• Classification, is the type of machine learning
in which we feed data and the model classifies the
data into different groups. Consider the following
example,

the data has different type of shapes in it. We

will teach the model which is what shape or what
are the different groups in the data. We will
provide the groups with their features like:
 Z—• -------------------------------------------- *
60 TYPES OF MACHINE LEARNING
- —

Now the trained model can classify any data after

learning how the groups are formed. If a new shape
is passed it will classify it according to what it
has learned. Like regression, we will keep feeding
it data until it classifies the data correctly.

Classification has also different types like

decision tree, Naive Bayes classification, support
vector machines, etc.
We will learn about them in the lesson dedicated
for this topic.
Unsupervised learning
Unlike supervised learning, we
don't teach or check the
predictions made by the models,
instead we feed the data and ask
for predictions directly. And it
is obvious that much data you'll
feed the results will be much
accurate. Unsupervised learning is
used in artificial intelligence
applications like face detection, object detection,
etc.

Reinforcement learning
Reinforcement learning consists of learning
models which doesn't require any input or output
data instead they learn to perform certain tasks
like solving puzzles or playing games. If the
model performs steps correctly it is rewarded
points, unless it penalized. The models learns the
more it performs from it's mistakes. The model
creates data as it performs the functions unlike
receiving data at the beginning.
For example consider the following model:
 Z—• ------------------------------------------ *
62 TYPES OF MACHINE LEARNING
—

Deep Reinforcement learning

As the name suggests this is the combination of
reinforcement and deep learning. Reinforcements
algorithms are combined with deep learning to
create powerful Deep Reinforcement learning models
that is used in fields like robotics, video games,
finance and healthcare. Many unsolvable problems
are solved by these models. DRL models are still
at new to us and there is a lot to learn about it
 ■x

1 IMPORTING
IV DATA
• Importing CSV
Data
• Importing JSON I I
Data I I
I
• Importing XLS
Doto

o J
z Oz A
10 J IMPORTING DATA
__________________________________________ >

Installing Anaconda
Head to anaconda.com/products/individuat to
download the latest version of Anaconda.

Anaconda Installers

Windows ■■ MacOS « Linux A

64-Bit Graphical Instcliter (466 MB) 64-Bit Graphical Installer (462 MB) 64-Bit (x86) Inst.liter (550 MB)

32-Bit Graphical Instailler (397 MB) 64-Bit Command Line Installer (454 MB) 64-Bit (Power8 ;>nd Power9) Installer (290
MB)

You can download the anaconda-installer for your

system, whether it is Windows, Mac or Linux. After
installing it, just run the installer and install
it as usual.
After installation, search for anaconda prompt
and run it.
All Apps Documents Web More ▼ $

Best match

Anaconda Prompt (anaconda3)

App

Search the web

Anaconda Prompt (anaconda3)
A anaconda Prompt (anaconda3) - See App
web results

P anaconda prompt anaconda3

CT Open
P anaconda prompt anaconda3 conda
c0 Run as administrator

P anaconda prompt anaconda3

uninstall
> Open file location

“P3 Pin to Start

"P* Pin to taskbar

® Uninstall

P anaconda Prompt (anaconda3)

 ' •/
65 IMPORTING DATA
---'•
This is the Anaconda Command Prompt, from where we
can run programs or perform other operations using
code's as commands.
Anaconda Prompt (anaconda3)

(base) C:\Users\Rahul>

Jupyter Notebook
The Oupyter Notebook is an open-source
web application that allows you to create
and share documents that contain live
code, equations, visualizations and jupyter
explanatory text. We will use it to perform
our data processing, analytics and
visualization, etc. on the go.
To open iupter Notebook, write jupyter notebook in
the anaconda command prompt and press enter.
5 Anaconda Prompt (anaconda3) □ >
(base) C:\Users\Rahul>jupyter notebook
 Anaconda will redirect you to your browser, [it may
ask you, in which browser to host your jupyter
notebook if you have more than one browsers] a new
tab will appear with your Jupyter notebook hosted.
You can host your Python files here, and also run the
code on the fly.
P’ jupyter Quit Logout

Files Running Clusters

Select items to perform actions on them. Upload New ■»

□ 0 ~ Name Last Modified

□ CD 3D Objects 5 days ago

□ CD anaconda3 4 days ago

□ CD ansel 5 days ago

□ CD Contacts 5 days ago

□ Co Creative Cloud Files 4 days ago

□ C3 Desktop 3 days ago

□ CD Documents a day ago

□ CD Downloads 10 minutes ago

□ CD Favorites 5 days ago

□ CD Links 5 days ago

□ CD Music a day ago

□ CD OneDrive an hour ago

□ CD Pictures 5 days ago

□ CD Saved Games 5 days ago

□ CD Searches 5 days ago

□ CD Videos 4 days ago

□ Newipynb 4 days ago

In Jupyter Notebook, we donJt need to install any

other module, package or library externally
everything we need is already present here and the
best thing is that you can code online without
installing any IDE or the Python Interpreter, which
makes it the best choice for data scientists
 Z—• F*-----------------------------------------------------------------------------------------

67 IMPORTING DATA
•

Working with Jupyter Notebook

To start coding, click on New and select Python 3 to
open a new Python file.
JUpyter Quit Logout

Files Running Clusters

Select items to perform actions on them. Upload New - 0

1 0 ■» fc/ Name* Last Modified File size

CD 3D Objects 5 days ago

CD anaconda3 4 days ago

CD ansel 5 days ago

CD Contacts 5 days ago

CD Creative Cloud Files 4 days ago

CD Desktop 3 days ago

rn r-» _______ _

0 jupyter Quit Logout

Files Running Clusters

Select items to perform actions on them. ____ I 1 —1 I____

Create a new notebook with Python 3
0 "V] B / Name 4
—1----- J ---------- Python 3
CD 3D Objects

CD anaconda3 Text File

CD ansel Folder
Terminal
CD Contacts

CD Creative Cloud Files 4 days ago

This is the place where we will write our code [in

the cell] and run it.
C jupyter Untitled Last Checkpoint: a few seconds ago (unsaved changes) Visit repo Copy Binder link

If you cannot create new file or encounter any

error, you can head directly to jupyter.org/try and
choose Python.
Try Classic Notebook Try JupyterLab Try Jupyter with Julia

jupyter

A tutorial introducing basic features JupyterLab is the new interface for A basic example of using Jupyter
of Jupyter notebooks and the Jupyter notebooks and is ready for with Julia.
I Python kernel using the classic general use. Give it a try!
Jupyter Notebook interface.
 We can rename our file, by clicking the name
[untitled]
3 jupyter Untitled Last Checkpoint: a few seconds ago (unsaved changes) Visit repo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted | Python 3 O

E + ® ft *4- H Run ■ C » Code v Q A Download A A O GitHub % Binder Memory: 168/2048 MB

I In [ ]: Q

We have only one code cell, in this cell we will

write our code
3 jupyter New Last Checkpoint: 2 minutes ago (autosaved) f® Visit repo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted | Python 3 0

B + »: (?) t 4-4’ H Run ■ C » |code v| H ± Download A A O GitHub % Binder Memory: 219/2048 MB

In [ ]:

There are three type cells - code cells, markdown

cells and raw cells.
We can use markdown cells to display headings or
titles.
J U py ter New Last Checkpoint: 7 minutes ago (unsaved changes)

File Edit View Insert Cell Kernel Widgets Help Trusted / | Python 3 O

B + | >5t 4 4> H Run ■ C » Markdown v E3 A Download <1 Ct O GitHub % Binder Memory: 219/2048 MB

I # Jupyter Notebook

Now run the cell by clicking the run button on the

header.
JUpyter New Last Checkpoint: 7 minutes ago (unsaved changes) Visit repo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted Z | Python 3 O

B + § C 4 4 H Run ■ C H code v ra A Download d Ci O GitHub % Binder Memory: 219/2048 MB

Jupyter Notebook
in [ ]:
 f •
69 IMPORTING DATA
---- ------------------
In code cells, we can write Python codes and execute
them instantly.
^JUpyter New Last Checkpoint: 12 minutes ago (autosaved) Visit repo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted / | Python 3 0

B + Run ■ C H Code v q A Download d Q O GitHub % Binder Memory: 119/2048 MB

Jupyter Notebook
In [ ]: 123*525|

^JUpyter New Last Checkpoint: 13 minutes ago (unsaved changes) Visit repo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted | Python 3 O

B + ® (5 + H Run ■ C H Code v s ± Download Ci Q O GitHub % Binder Memory: 119 / 2048 MB

Jupyter Notebook
In [1]: 123*525

Out[l]: 64575

To insert a new cell below the selected cell, press

b on your keyborad or click the + icon.
3 jupyter New Last Checkpoint: 17 minutes ago (unsaved changes) f® Visitrepo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted | Python 3 O

B | ♦ ® (5 ♦ 4’ H Run ■ C Code v E A Download a O GitHub % Binder Memory: 119 / 2048 MB

Jupyter Notebook
In [1]: 123*525

Out[l]: 64575

In [ ]:

You can select [blue] or edit [green] a cell, by

clicking outside the text feild or inside the text
feild respectively.
3 JUpyter New Last Checkpoint: 19 minutes ago (autosaved) Visitrepo Copy Binder link

File Edit View Insert Cell Kernel Widgets Help Trusted | Python 3 O

B + 3. ® C) M Run ■ C » code * S A Download A A O GitHub % Binder Memory: 119 / 2048 MB

Jupyter Notebook
 Z—• F*-----------------------------------------------------------------------------------------

70 IMPORTING DATA
• -J

We have more access to the markdown cells, to diplay

texts more gracefully. We can add headings,
sub-headings and lower-headings, using # 1, 2 and 3
times followed by space and then text respectively.

# Jupyter Notebook
## IPython
###Data

In [ ]: |

Jupyter Notebook

IPython

Data

in [ ]:

We can create ordered and bulleted lists

To create an ordered list, use 1 for the first list

item and then use tabspace for the sub-list items and
use correct numbering. [The text should be written
followed by space after the numbers]
To create a bulleted list, use - for square bullets
and * for round bullets, and same manner as above for
list and sub-list items.
 71 IMPORTING DATA
-

We can also links, using [] & (). Write the display

text in [] and put the link in (), you can also add a
hover text inside of ( ) using " ” quotes.

TJupter Notebook for Pythonl(httDs://iUDVter.ore/try "Try it!")

Jupter Notebook for Python

We can also use **<text>** or <text> to render

bold text and *<text>* or _<text>_ to render
italicized text

We can also insert images by going to the

Edit>Insert Image and browse your image to enter it
File Edit View Cell Kernel Widgets Help | Python 3

Create tables using | and strictly following the

below example
|Product|Price|Quantity|
|------------ 1-— |-................|
|Biscuits|5|2|
|Milk|7|5L|

Product Price Quantity

Biscuits 5 2

Milk 7 5L
 Z—• F*-----------------------------------------------------------------------------------------

72 IMPORTING DATA
—
HereJs a complete list of shortcuts of various
operations with cells.
r
Operations Shortcut
change cell to code y
change cell to markdown m
change cell to raw r
close the pager Esc
restart kernal 0 + 0
copy selected cell c
cut selected cell X
delete selected cell d + d
enter edit mode Enter
extend selection below Shift + j
extend selection above Shift + k
find and replace f
ignore Shift
insert cell above a
insert cell below b
interrupt the kernal i + i
Merge cells Shift + m
paste cells above Shift + v
paste cells below V
run cell and insert below Alt + Enter
run cell and select below Shift + Enter
run selected cells Ctrl + Enter
save notebook Ctrl + s
scroll notebook up SHIFT + Space
scroll notebook down Space
select all Ctrl + a
show keyboard shortcuts h
toggle all line numbers Shift + 1
toggle cell output 0
toggle cell scrolling Shift + o
toggle line numbers 1
undo cell deletion z
 Z—• F*-----------------------------------------------------------------------------------------

73 IMPORTING DATA
• -J

Data files
Often we have data in multiple file formats like,
data of sales of any product, number of
subscribers, etc. There are a lot of ways to import
data but we will use pandas library here
Importing CSV Data
Reading data from CSV(comma separated values) is a
fundamental necessity in Data Science. Often, we get
data from various sources which can get exported to
CSV format so that they can be used by other
systems. To work with csv files we need one first,
you can download the sample file from here
http://bit.Ly/csv_dt
To import it, you need move the csv file in the
place where your lupyter Notebook is hosted, to find
it use the below code

In [32]: import os
print(os.getcwd())

C:\Users\Rahul

Move your file there, and use the read_csv

function of pandas library to import the csv file

In [38]: import pandas as pan

dt = pan.read_csv("csv_data.csv")
dt

id name price sales brand

0 101 biscuits 5.00 227 HomeFoods

1 102 cookies 7.25 158 TBakery

2 103 cake 12.00 50 TBakery

3 104 whey_supplement 34.90 24 MusleUp

4 105 protein_bars 4.90 85 MusleUp

5 106 potato_chips 1.75 121 HomeFoods

 Z—• F*-----------------------------------------------------------------------------------------

74 IMPORTING DATA
• -J

We can access a single column of the csv data using

slicing like DataFnames

In [39]: import pandas as pan

dt = pan.read_csv("csv_data.csv")
dt['sales’]

Out[39]: 0 227
1 158
2 50
3 24
4 85
5 121
Name:: sales, dtype: int64

We can extract only 2 or more columns from the data

using the toe[:,[<*cotumns>]] syntax

In [44]: import pandas as pan

dt = pan.read_csv("csv_data.csv")
dt.loc[:,['name','sales']]

0ut[44]:
name sales

0 biscuits 227

1 cookies 158

2 cake 50

3 whey_supplement 24

4 protein_bars 85

5 potato_chips 121

or just with some rows

In [46]: import pandas as pan

dt = pan.read_csv("csv_data.csv")
dt.loc[4:6,['name','sales']]

0ut[46]:
name sales

4 protein_bars 85

5 potato_chips 121
 Z—• F*-----------------------------------------------------------------------------------------

75 IMPORTING DATA
• -J

To access a single element, we can use its

row-column index with the values function
In [49]: import pandas as pan
dt = pan.read_csv("csv_data.csv")
dt

id name price sales brand

0 101 biscuits 5.00 227 HomeFoods

1 102 cookies 7.25 158 TBakery

2 103 cake 12.00 50 TBakery

3 104 whey_supplement 34.90 24 MusleUp

4 105 protein_bars 4.90 85 Muslellp

5 106 potato_chips 1.75 121 HomeFoods

In [50]: biscuits_sales = dt.values[0,3]

biscuits_sales

Out[50]: 227

0 12 3 4
id name price sales brand
V
dt. values J
/1O1 biscuits 5^^(227) HomeFoodsJ

dt. values [ 0 1 102 ^^ceoRies 7.25 158 TBakery

2 cake 12.00 50 TBakery

(dt. values [0,3]}*-" 3 104 whey_supplement 34.90 24 MusleUp

4 105 protein_bars 4.90 85 MusleUp

5 106 potato_chips 1.75 121 HomeFoods

'dt,vatues[Q,3]}

The data values are stored as ndarrays so, to access

single elements we can using slicing similar to that
of DataFrames
 IMPORTING DATA

Importing JSON Data

JSON file stores data as text in human-readable
format. ISON stands for JavaScript Object Nota­
tion. Get your sample JSON data here
http://bit.ty/json_data
Pandas can read JSON files using the reacLjson
function
In [2]: import pandas as pan
dt = pan.read_json("json_data.json")
dt

ID Name Price Sales Brand

0 101 Biscuits 5.00 227 HomeFoods

1 102 Cookies 7.25 158 TBakery

2 103 Cake 12.00 52 TBakery

3 104 Whey Supplement 34.90 24 Muslellp

4 105 Protein Bars 4.90 85 MusleUp

5 106 Potato Chips 1.75 121 HomeFoods

Similar to the CSV files, we can perform all the

slicing and data extraction with JSON data files

In [6]: import pandas as pan

dt = pan.read_json("json_data.json")
print(dt.loc[:,["ID","Name","Sales"]])
print(dt["Name"])
print(dt.values[5,4])

ID Name Sales
0 101 Biscuits 227
1 102 Cookies 158
2 103 Cake 52
3 104 Whey Supplement 24
4 105 Protein Bars 85
5 106 Potato Chips 121
0 Biscuits
1 Cookies
2 Cake
3 Whey Supplement
4 Protein Bars
5 Potato Chips
Name: Name, dtype: object
HomeFoods
 IMPORTING DATA

Importing EXCEL Data

Microsoft Excel is a very widely used spread
sheet program. Its user friendliness and appealing
features makes it a very frequently used tool in
Data Science. Get your sample ISON data here
http://bit.Ly/xtsx_data
The reacLexcet function of the pandas library is
used read the content of an Excel file into the
python environment as a pandas DataFrame.

In [9]: import pandas as pan

dt = pan.read_excel("xlsx_data.xlsx")
dt

id name price sales brand Unnamed: 5

0 101 biscuits 5.00 227 HomeFoods NaN

1 102 cookies 7.25 158 TBakery NaN

2 103 cake 12.00 50 34 TBakery

3 104 whey_supplement 34.90 24 MusleUp NaN

4 105 protein_bars 4.90 85 MusleUp NaN

5 106 potato_chips 1.75 121 HomeFoods NaN

As execel sheets are imported as Pandas Data-

Frames, we can perform all the tasks on the excel
data like Data Frames.
You may notice., we have a Unnamed: 5 column with
NaN values [except dt. value[2,5] ]. Let's clean up
our data.
First we need to remove the Unnamed: 5 column,
which we can do using the det keyword

In [10]: import pandas as pan

dt = pan.read_excel("xlsx_data.xlsx")
del dt["Unnamed: 5"]

As we have learned earlier the det keyword

removes the whole column we don't need to deal
with the Data Cleansing
 Z—• F*-----------------------------------------------------------------------------------------

78 IMPORTING DATA
—
We have removed the Unnamed: 5 column

In [11]: import pandas as pan

dt = pan.read_excel("xlsx_data.xlsx")
del dt["Unnamed: 5"]
dt

id name price sales brand

0 101 biscuits 5.00 227 HomeFoods

1 102 cookies 7.25 158 TBakery

2 103 cake 12.00 50 (34)

3 104 whey_supplement 34.90 24 MusleUp

4 105 protein_bars 4.90 85 MusleUp

5 106 potato_chips 1.75 121 HomeFoods

Now, we need to replace dt. valueL2,5] i.e. 34

with TBakery. We can use the replace method

In [12]: import pandas as pan

dt = pan.read_excel("xlsx_data.xlsx")
del dt["Unnamed: 5"]
dt.replace({34:"TBakery"})

id name price sales brand

0 101 biscuits 5.00 227 HomeFoods

1 102 cookies 7.25 158 TBakery

2 103 cake 12.00 50 TBakery

3 104 whey_supplement 34.90 24 MusleUp

4 105 protein_bars 4.90 85 MusleUp

5 106 potato_chips 1.75 121 HomeFoods

So our data is clean with no errors. Try recaping

the chapter and attempt the Exercise, where youJll
be provided with sample data files [links] with
lots of errors and you have to perform all the data
cleansing practised in the previous lesson, this
will be a very good exercise to help you understand
about data processing and cleansing more
 ■x

DATA
U OPERATIONS
• NumPy
operations
• Pandas
operations
• Cleaning data

o
 —11T------------
data operations
A

'--- '□----------- y

Python handles data of various formats mainly

through the two libraries, Pandas and Numpy. We
have already seen the important features of these
two libraries in the previous chapters. In this
chapter we will see some basic examples from each
of the libraries on how to operate on data and
perform different tasks like cleaning the data,
analytics, etc.

NumPy Operations
To start working with NumPy, we need to import
numpy to create NumPy arrays.

In [ ]: import numpy

Now let's create an array, using the arrayO

function and print it.

In [2]: import numpy

ar = numpy.array([l,5,7])
print(ar)

[1 5 7]

ar is a 1-Dimensional array, we can also create a

2-Dimensional array by creating one or more
1-Dimensional array inside of another array

In [8]: import numpy

ar = numpy.array([[1,5,7], [2,3,9]])
print(ar)

[[1 5 7]
[2 3 9]]

1-D array

r2l-D array
 1
81 data operations
-

We can specify the dimension of an array during

creation using the ndmin parameter

In [10]: import numpy

ar = numpy.array([l,5,7]> ndmin = 2 )
print(ar)

[[1 5 ?]]

Although we passed a 1-Dimensional array, it

became a 2-Dimensional array because of the
specification of the dimensions of the array in the
ndmin parameter
We created an array with integers so, let's create
arrays with strings and floats using the dtype
parameter with the same values
In [11]: import numpy
ar_str = numpy.array([1,5,7], dtype = str )
ar_flt = numpy.array([l,5,7], dtype = float)
print(ar_str)
print(ar_fIt)

[■r '5' ■?■]

[1. 5. 7.]

ar_str is an array of string literals and an_fit

is an array of floats. We can also change these
numbers to complex numbers the same way using
complex as dtype

In [13]: import numpy

ar_str = numpy.array([l,5,7], dtype = str )
ar_flt = numpy.array([l,5,7], dtype = float)
ar_cmx = numpy.array([l,5,7], dtype = complex )
print(ar_str)
print(ar_flt)
print(ar_cmx)

[T '5' ’7’]
[1. 5. 7.]
[1.+0.J 5.+0.j 7.+0.j]

ar.str, ar_f“Lt and ar.cmx are arrays created

with same data., but with different data types as
strings, floats and complex numbers repectively.
 Z—•
82 data operations
•

Pandas Operations
Pandas handles data through Series,Data Frame,
and Panel. We will learn to create each of these.

Pandas Series
We already know what Pandas Series is. A pandas
Series can be created using the SeriesO function
so, let's import pandas and create series.

In [14]: import pandas

sr = pandas.Series([l, 5,7])
print(sr)

0 1
1 5
2 7
dtype: int64

As you can see our data is indexed form 0 to 2

with the data type printed as integer, we can
specify our own indexes in the index parameter

In [16]: import pandas

sr = pandas.Series([l,5,7], index = ['A', 'B', 'C'])
print(sr)

A 1
B 5
C 7
dtype: int64

Like ndarrays, we can also specify the data type

in pandas series using dtype parameter during
series creation

In [18]: import pandas

sr = pandas.Series([1,5,7], dtype = complex )
print(sr)

0 1.000000+0.000000j
1 5.000000+0.000000j
2 7.000000+0.000000j
dtype: complexl28
 Z—•
83 data operations
•
We can use a ndarray to create a pandas series

In [19]: import numpy

import pandas
ar = numpy.array([1,5,7])
sr = pandas.Series( data = ar, copy = True )
#is same as sr = pandas.series(ar, copy = True)
print(sr)

Q 1
1 5
2 7
dtype: int32

We passed the ar ndarray as the data for the

series [use of the data parameter isnJt necessary,
its just for better understanding] and also used
the copy parameter to create a copy of the data.
If you want to get the data, without the indexes
use the values function

In [21]: import numpy

import pandas
ar = numpy.array([l,5,7])
sr = pandas.Series(ar)
print(sr.values)

[1 5 7]

You can print a more detailed version of the

above using the array function

In [22]: import numpy

ray Type
import pandas
ar = numpy.array([1,5,7]) <PandasArray>
sr = pandas.Series(ar) [1, 5, 7] £V|alues
print(sr.array) Lengtl

<PandasArray> ta type
[1, 5, 7]
Length: 3, dtype: int32

You can use values or array function according

to your needs whether you want just the values or
summarized detail of the arrays in that panda
series. Also note the difference in the array
function in NumPy and Pandas.
 • /*
data operations

Pandas Data Frames

Pandas Data Frames aligns data in a tabular
fashion of rows and columns. A pandas DataFrame
can be created using the DataFrameO function, we
need pass a dictionary as the data

In [23]: import pandas

df = pandas.DataFrame({"Product":['CookiesBiscuits'],
"Sales":[157,227]})
print(df)

Product Sales
0 Cookies 157
1 Biscuits 227

Dictionary keys are the columns and their values

are the content of the rows of the Data Frame. We
can also use index parameter here

In [24]: import pandas

df = pandas.DataFrame({"Product":['CookiesBiscuits'],
"Sales":[157,227]}, index = [1,2])
print(df)

Product Sales
1 Cookies 157
2 Biscuits 227

We can define the columns and it-’s data

seperately using ndarrays
In [42]: import pandas
import numpy
ar = numpy.array([[l,3],[6,2]])

df = pandas.DataFrame(data = ar,
index = ['A','B'],
columns = [,C1','C2'])
print(df)

Cl C2
A 1 3
B 6 2

The data is stored in the ndarray and the columns

are defined in the DataFrame-’s columns parameter.
Note that, a 2-Dimensional ndarray with 2
1-Dimensional arrays in it is passed to the data
parameter to act as the data
 85 data operations

We can add columns to the DataFrame using the

<DataFrame> [< column_name >] = <val_ues>
syntax

In [44]: import pandas

import numpy
ar = numpy.array([[1,3],[6,2]])

df = pandas.DataFrame(data = ar,
index = ['A','B'],
columns = ['Cl','C2'])
df['C3'] = (df['Cl']*5)
print(df)

Cl C2 C3
A 1 3 5
B 6 2 30

We can delete columns from the DataFrame using

the det function

In [45]: import pandas

import numpy
ar = numpy.array([[1,3],[6,2]])

df = pandas.DataFrame(data = ar,
index = ['A','B'],
columns = [,C1,,'C2'])
df['C3‘] = (df['Cl']*5)
del df['C2']
print(df)

Cl C3
A 1 5
B 6 30

We can print a column of the DataFrame using the

<DataFrame>[< column_name >] syntax

In [46]: import pandas

import numpy
ar = numpy.array([[l,3],[6,2]])

df = pandas.DataFrame(data = ar,
index = ['A','B'],
columns = [,C1',,C2'])
print(df['Cl'])

A 1
B 6
Name: Cl, dtype: int32
 data operations

Slicing Syntax
To get a single element from a ndarray or pandas
series or pandas dataframes, we need to use the
slice syntax <array>[start:end:step(optional)]
Let's extract some elements from the arrays we
have created so far.

In [59]: import numpy as npy

arl = npy.array([1, 5])
ar2 = npy.array([[1, 3],
[5, 2]])
ar3 = npy.array([[[1, 3],
[5, 2]],
[[2, 4],
[4, 6]]])

#SLicing 1-Dimensional, array

print(arl[0])

#SLicing 2-DimensionaL array

print(ar2[0,l])

#SLicing 3-Dimensional, array

print(ar3[l,0j1])

1
3
4

We use a comma , to slice further in 2 or more

dimensional arrays, the following figure will help
you understand the slicing of the 3-Dimensional
array more better

ar3[] [[[1, 3],

[5, 2]],
[[2, 4],
Full orray [4, 6]]]
 Z—•
87 data operations
•
[[[1, 3],
[5. 2]]j
[[2, 4]j
[4, 6]] ]

[[[1, 3]j
[5, 2]]j
l[2, 4]j
[4j 6]] ]

[[[1, 3]j
[5, 2]]j
[[2> 4 ],
[4, 6]] ]
ar3[l,0,l]

Final Slice

Slicing may seem a bit tough for beginners due to

the dimensions, that's why I created the figure to
help you understand slicing better. If you are
confident try solving the slicing questions in the
Exercise
 88 data operations

To get a element from a pandas series., we use the

<series>[<expticit_index> or <impticit_index>]
syntax

In [4]: import pandas as pan

sr = pan.Series([1, 3, 5], index = ['a','b','c'])

print(sr['a']) #impLicit indexing

print(sr[0]) #expLicit indexing

1
1

If you have indexes like numbers like these

In [6]: import pandas as pan
sr = pan.Series([1, 3, 5], index = [2,4,6])

If you want the second element using the implicit

index [indexing defined in index parameter] use
<series> .Loc [<LabeL>] syntax and using the
explicit indexing [ 0,1,2,... ] use
<series>. itoc [<index>] syntax

In [7]: import pandas as pan

sr = pan.Series([1, 3, 5], index = [2,4,6])

print(sr.loc[4])
print(sr.iloc[l])

3
3

we can modify or delete tne elements using slicing

In [9]: import pandas as pan
sr = pan.Series([1, 3, 5], index = [2,4,6])

sr[4] = 7
print(sr)

del sr[6]
print(sr)

2 1
4 7
6 5
dtype: int64
2 1
4 7
dtype: int64
 Let's say we have a DataFrame like this

In [27]: import pandas as pan

sr = pan.DataFrame({'Product':['Biscuit','Cookies'],
'Sales':[227,158]}, index = [1,2])
sr

0ut[27]:
Product Sales

1 Biscuit 227

2 Cookies 158

And want the Sales Column only, so use the

<DataFrame> [< column_labet >] syntax
In [32]: import pandas as pan
sr = pan.DataFrame({'ProductBiscuit','Cookies'],
'Sales':[227,158]}, index = [1,2])
sr['Sales']

Out[32]: 1 227
2 158
Name: Sales, dtype: int64

or to get the second row only, so use the

<DataFrame>.toe [< row_index>] syntax
In [33]: import pandas as pan
sr = pan.DataFrame({'Product':['Biscuit','Cookies'],
'Sales':[227,158]}, index = [1,2])
sr.loc[2] #You can aLso use sr.iLoc[l]

Out[33]: Product Cookies

Sales 158
Name: 2, dtype: object

or to get the sales of cookies only, so use the

< Dat a Frame> .values [< index >] syntax
In [37]: import pandas as pan
sr = pan.DataFrame({'Product':['Biscuit','Cookies'],
■Sales':[227,158]}, index = [1,2])
sr.values[l,l]

Out[37]: 158

The values are stored as ndarrays, that's why it

used slicing similar to that of 2-Dimensional
ndarrays
We can delete a whole column from the DataFrame
In [41]: import pandas as pan
sr = pan.DataFrame({'Product':['Biscuit','Cookies’],
'Sales':[227,158]}, index = [1,2])
sr

0ut[41]:
Product Sales

1 Biscuit 227

2 Cookies 158

In [45]: del sr[’Sales']

sr

0ut[45]:
Product

1 Biscuit

2 Cookies

but we cannot delete a value

In [47]: import pandas as pan
sr = pan.DataFrame({'Product':['Biscuit','Cookies']}
'Sales':[227,158]}, index = [1,2])
del sr.values[l,l]

ValueError Traceback (most recent call last)

<ipython-input-47-4422f0e71c59> in <module>
2 sr = pan.DataFrame({'Product':['Biscuit','Cookies'],
3 'Sales':[227,158]}, index = [1,2])
------ > 4 del sr.values[l,l]

ValueError: cannot delete array elements

nor you can modify a value

In [48]: import pandas as pan
sr = pan.DataFrame({'Product':['Biscuit','Cookies'],
'Sales':[227,158]}, index = [1,2])
sr

0ut[48]:
Product Sales

1 Biscuit 227

2 Cookies 158

In [51]: sr.values[l,l] = 162

sr.values[l,l]

Out[51]: 158
 data operations

More with ndarrays

We can reverse a ndarray using <array>[ ::-1] syntax
In [56]: import numpy as npy
ar = npy.array([1,2,3,4])
ar

Out[56]: array([l, 2, 3, 4])

In [55]: ar = ar[::-1]
ar

Out[55]: array([4, 3, 2, 1])

We can broadcast a whole ndarray without doing it

the long way

In [63]: import numpy as npy

ar = npy.array([5,1,3,9])
ar

Out[63]: array([5, 1, 3, 9])

In [64]: ar.sort()
ar

Out[64]: array([l, 3, 5, 9])

There are many built-in ndarray methods that will

not be discussed now, but will be used in the future
lessons in various steps, you may go to the
documentation to find all the functions and their
roles, as we don't require every function for our
data processing and analyzing, all the miscellaneous
functions are not discussed in this book
 data operations

Data Cleansing
Let's consider a situtation like below

In [71]: import pandas as pan

import numpy as npy
ar = npy.array([[1,2,3],[4,7,2],[4,9,1]])
df = pan.DataFrame( data = ar, index = ['a','c','e'],
columns = ['Cl','C2','C3'])
df

0ut[71]:
C1 C2 C3

a 1 2 3

c 4 7 2

e 4 9 1

In [72]: df = df.reindex(['a','b','c','d','e'])
df

0ut[72]:
C1 C2 C3

a 1.0 2.0 3.0

b NaN NaN NaN

c 4.0 7.0 2.0

d NaN NaN NaN

e 4.0 9.0 1.0

The reindexed Data Frame has NaN values in the b and

d rows. This happened because, there is no data for b
and d rows. Using reindexing, we have created a Data-
Frame with missing values. In the output, NaN means
Not a Number. To make detecting missing values easier
(and across different array dtypes), Pandas provides
the isnuULO and notnullO functions, which are also
methods on Series and DataFrame objects

In [73]: df['Cl'] .isnullO

Out[73]: a False
b True
C False
d True
e False
Name: Cl, dtype: bool
 z—• /
93 data operations
-
Pandas provides various methods for cleaning the
missing values. The fillna function can filAna NaN
values with non-null data in a couple of ways like
replacing NaN values with 0

In [74]: df.fillna(O)

Out[74]:
C1 C2 C3

a 1.0 2.0 3.0

b 0.0 0.0 0.0

c 4.0 7.0 2.0

d 0.0 0.0 0.0

e 4.0 9.0 1.0

We can copy the value above or below that data using

pad or bfitt in method parameter of filtna function

In [75]: df.fillna( method = 'pad' )

C1 C2 C3

a 1.0 2.0 3.0

b 1.0 2.0 3.0

c 4.0 7.0 2.0

d 4.0 7.0 2.0

e 4.0 9.0 1.0

We can drop the rows with missing values with dropna

function

In [76]: df.dropna()

0ut[76]:
C1 C2 C3

a 1.0 2.0 3.0

c 4.0 7.0 2.0

e 4.0 9.0 1.0

 f
94 data operations
-
If we want to change a single value in a Data
Frame,we can use the replace function

In [78]: import pandas as pan

import numpy as npy
ar = npy.array([[l,2,3],[4,7,2],[4,9,1]])
df = pan.DataFrame( data = ar, index = ['a'>'c', 'e']
columns = ['Cl','C2',’C3’])
df.replace({3:13})

Out[78]:
C1 C2 C3

a 1 2 13

c 4 7 2

e 4 9 1
 1 O DATA ANALYSIS
lu S PROCESSING
• Doto Analytics
• Correlations
between
attributes
• Skewness of
the data

o
 A
data analysis s processing
k ________________________________________________ J

We need to a lot of analytics or statistics of our

data to know more about the data and it's
attributes
Like central tendency i.e. mean, median and mode
are the basic statistics of our data which tells us
about the average of the data, 50% or middle value
and the most occuring value in the whole data
Likewise we will analyze our data and as mentioned
earlier we don't need to calculate them manually or
through formula's, there's plenty of functions
present in different libraries to conduct the
analysis

Data analytics
Before training any model we need to check the
data and it's details. We will use the rtress.csv'
as your data for now. You can get the
rC bit.ly/trees csv dt
file either scanning the qr-code or http://bit.ly/trees_csv_dt
the link given -
Make sure to move the file your home
directory of lupyter Notebook and then
import the csv data. Before doing any
further action, let's have a look at
our raw data Scan Me!

In [2]: import pandas

dt = pandas.read_csv('trees.csv’)
dt

Out[2]:
Index "Girth (in)" "Height (ft)" "Volume(ftA3)"

0 1 8.3 70 10.3

1 2 8.6 65 10.3

2 3 8.8 63 10.2

3 4 10.5 72 16.4

4 5 10.7 81 18.8
 97 DATA ANALYSIS & PROCESSING
-------- '9
The first analysis is to know the shape of the
data or how amny rows and columns are present in
the data. We can do so by using the shape
attribute of the dataframe object

In [4]: import pandas

dt = pandas.read_csv('trees.csv')
dt.shape

0ut[4]: (31, 4)

So our data has 31 rows and 4 columns i.e. 124

values in total. If we want we can just inspect
the first 10 values using the head() function and
passing 10 as argument

In [5]: import pandas

dt = pandas.read_csv('trees.csv')
dt.head(10)

Out[5]:
Index "Girth (in)" "Height (ft)" "Volume(ftA3)"

0 1 8.3 70 10.3

1 2 8.6 65 10.3

2 3 8.8 63 10.2

3 4 10.5 72 16.4

4 5 10.7 81 18.8

5 6 10.8 83 19.7

6 7 11.0 66 15.6

7 8 11.0 75 18.2

8 9 11.1 80 22.6

9 10 11.2 75 19.9

To get a statistical overview of the whole data

we can use the describe^ function which provides
8 properties i.e. count, mean, standard deviation,
minimum value, maximum value, 25% (first
interquartile sperator), 50% (median) and 75%
(third interquartile seperator)
In [6]: import pandas
dt = pandas.read_csv('trees.csv')
dt.describe()

Index "Girth (in)" "Height (ft)" "Volume(ftA3)"

count 31.000000 31.000000 31.000000 31.000000

mean 16.000000 13.248387 76.000000 30.170968

std 9.092121 3.138139 6.371813 16.437846

min 1.000000 8.300000 63.000000 10.200000

25% 8.500000 11.050000 72.000000 19.400000

50% 16.000000 12.900000 76.000000 24.200000

75% 23.500000 15.250000 80.000000 37.300000

max 31.000000 20.600000 87.000000 77.000000

If you want the values rounded-off to say 2

decimal places we can use the pandas set_option()
function and specify the precision as 2. We can
specify a lot of options through this function

In [7]: import pandas

dt = pandas.read_csv('trees.csv')
pandas.set_option('precision',2)
dt.describe()

Index "Girth (in)" "Height (ft)" "Volume(ftA3)"

count 31.00 31.00 31.00 31.00

mean 16.00 13.25 76.00 30.17

std 9.09 3.14 6.37 16.44

min 1.00 8.30 63.00 10.20

25% 8.50 11.05 72.00 19.40

50% 16.00 12.90 76.00 24.20

75% 23.50 15.25 80.00 37.30

max 31.00 20.60 87.00 77.00

 ' •/ '
99 DATA ANALYSIS S PROCESSING
-------- '•

Correlation between attributes

The relation between two attributes (feature or
label) in a data is called relationship. It is
important to know the relations between the
attributes. We can do so using the corr() function
and using the Pearson's Correlation Coefficient to
calculate that. The Pearson's Correlation
Coefficiet can be understood by the following:
• 1 represents positive correlation
• 0 represents no relation at all
• -1 represents negative correlation

In [2]: import pandas

dt = pandas.read_csv('trees.csv')
pandas.set_option('precision',2)
dt.corr(method='pearson')

Index "Girth (in)" "Height (ft)" "Volume(ftA3)"

Index 1.00 0.97 0.47 0.90

"Girth (in)" 0.97 1.00 0.52 0.97

"Height (ft)" 0.47 0.52 1.00 0.60

"Volume(ftA3)" 0.90 0.97 0.60 1.00

Note that we used the precision of the values as

2 to keep the values rounded-off to 2 decimal
places. In the corr() function we specified
pearson in the method parameter.
As we already know that Girth, Height and Volume
of tree are correlated that's why we get the
values around 0.5 - 1.0 which represents positive
correlationship i.e. if Height is changed the
volume will be affected, if the Girth is changed
the volume will be affected and vice versa
 ' •/
100 DATA ANALYSIS S PROCESSING
-------- '•

Skewness of the data

Skewness of a data is the situation when the data
appears to have normal distribution but it may be
skewed to either left or right. We need the
skewness of a data to correct the data during it's
preparation. The more the value is close to 0 it
is less skewed and more the value is close it -1
or 1 it is more skewed to either left or right
side, let's check the skewness of our tress data
using the skewO function

In [3]: import pandas

dt = pandas.read_csv('trees.csv')
pandas.set_option('precision',2)
dt.skew()

Out[3]: Index 0.00

"Girth (in)" 0.55
"Height (ft)" -0.39
"Volume(ftA3)" 1.12
dtype: float64

As the index column has values from 1 to 31 it's

skewness is 0 i.e. not skewness at all. On the
other hand Girth can be said to be skewed to the
right side, Height is skewed to the left side and
Volume is highly skewed to the right side i.e.
beyond 1. While data preparation we must consider
the skewness and keep it close as much as possible
to 0

Data Processing
Before feeding the data to models we need to
pre-process the data because the algorithms are
completely depended on the data so it must be
clean and appropriate as much as possible. While
finding skewness we found that you data is skewed
i.e. it needs to be closer to 0 for better
results, so let's look at some processes to ready
our data
 ----------------------------------------
^101 DATA ANALYSIS S PROCESSING

Scaling
Our data is spread over a wide range with
different scales i.e. not suitable to train
models. We need to bring our data in a more
appropriate scale, we can do so using the
MinMaxScater class and itJs fit_transform()
method of the scikit-learn library. We can scale
our data in the range of 0 to 1 which is the most
appropriate range for the algorithms

In [28]: import pandas

from sklearn import preprocessing
dt = pandas.read_csv('trees.csv')
ar = dt.values # array
# Scoter Object
Sclr = preprocessing.MinMaxScaler(feature_range=(0,l))
skl_ar = Sclr.fit_transform(ar) ^Seating
# Seated data
skl_dt = pandas.DataFrame(skl_ar_,
columns=['S.No.','Height','Height','Volume'])

skl_dt.round(1).loc[5:10]

Out[28]:
S.No. Height Height Volume

5 0.2 0.2 0.8 0.1

6 0.2 0.2 0.1 0.1

7 0.2 0.2 0.5 0.1

8 0.3 0.2 0.7 0.2

9 0.3 0.2 0.5 0.1

10 0.3 0.2 0.7 0.2

You can compare the values S.No. Girth Height Volume

with the values beside i.e.
5 5 10.7 81 18.8
unsealed. If you want you can
change the range to say 0-100 6 6 10.8 83 19.7

through the feature_range 7 7 11.0 66 15.6

parameter in MinMaxScater 8 8 11.0 75 18.2
while the scaler class 9 9 11.1 80 22.6
intialization
10 10 11.2 75 19.9
 ' •/ '
102-- DATA ANALYSIS S PROCESSING
--- '•
Normalization
Normalization is used to rescale each row of data
to have a length of 1. It is mainly useful in
Sparse dataset where we have lots of zeros. There
are two types of normalization process namely LI
and L2. With the LI method, all the values in each
row will sum upto 1. We can demonstrate the same
using the Normalizer class and it's transform
method. To use the LI method specify rll' in the
norm parameter of the class

In [3]: import pandas

from sklearn import preprocessing
dt = pandas.read_csv('trees.csv')
ar = dt.values
Nm = preprocessing.Normalizer(norm='11')
nm_ar = Nm.transform(ar)
print(nm_ar[:5])
# Sum of the rows
for i in [0,1,2,3,4]:
print(sum(nm_ar[i]))

[[0.01116071 0.09263393 0.78125 0.11495536]

[0.02328289 0.10011641 0.75669383 0.11990687]
[0.03529412 0.10352941 0.74117647 0.12 ]
[0.03887269 0.10204082 0.69970845 0.15937804]
[0.04329004 0.09264069 0.7012987 0.16277056]]
1.0
1.0
1.0
0.9999999999999999
1.0

We created the Nm object of the Normalizer class

with the normalizing method as 11 in the norm
parameter while intialization and normalized our
ar data values with the transform method of the
Normalizer class and stored it in nm_ar variable.
Then we printed the 5 rows of the normalized data
values
We also created a for loop to print the sum of
each row of the normalized data i.e. l(except for
the 4th row i.e. 0.99). Note that we didn't
 ' •/ '
103-- DATA ANALYSIS S PROCESSING
--- '•
perform any rounding-off. In the next method i.e.
L2 Normalization^ all the squares of values in
each row sum upto 1. So letJs use r12J in the norm
parameter and check their sums

In [12]: import pandas

from sklearn import preprocessing
dt = pandas.read_csv('trees.csv')
ar = dt.values
Nm = preprocessing.Normalizer(norm='12')
nm_ar = Nm.transform(ar)

print('L2 Normalization\n')
print(nm_ar[:5])

# Sum of vaLues in the rows

print('\nSum of the values in each row\n')
for i in [0,1,2]:
print(sum(nm_ar[i]))

# Sum of the squares of the vaLues in the rows

sm_row = '\nSum of squares of the values in each row\n'
for i in [0,1,2,3]:
print(sm_row)
sm_row = 0
for val in nm_ar[i]:
sm_row += val*val

L2 Normalization

[[0.01403589 0.11649791 0.98251251 0.1445697 ]

[0.03012017 0.12951675 0.97890567 0.1551189 ]
[0.04651593 0.13644674 0.97683459 0.15815417]
[0.05355175 0.14057333 0.96393143 0.21956216]
[0.05953254 0.12739964 0.96442719 0.22384236]]

Sum of the values in each row

1.2576160130346932
1.2936614951212533
1.3179514295489714

Sum of squares of the values in each row

1.0
1.0
1.0
 ' •/
104 DATA ANALYSIS S PROCESSING
--- '•
The code may a bit hard to understand because of
the for loop but let's try to understand it. We
normalize the data as we did we before but this
time we used the f12' method and printed the data
values. Then we printed the sum of the values of
first three rows of the normalized data but they
didn't sum upto 1. Next as the L2 method states,
we printed the sum of the squared values in the
first three rows using for loop which turned out
to be exactly 1
Before the for loop, we created an sm_row
varaible in which we will add our squared values
in the rows but we stored a string at start. Then
we created the outer loop in which we will get the
index of the rows. Also we entered one more number
in the list because at the first iteration the
string in the sm_row will be printed and after
printing it we changed it's value to 0 and then we
create the innner in which we will perform
addition. In each iteration of the inner loop we
will add the square of each element in the rows
with += compound assignment operator. After all
the values are sumed up, we return to the outer
loop and print it and again revert the value to 0
to store teh sum for the next row until all the
values are printed
----------------------- [Sum holder variable (vessel)]

(sm_row) = (’\nSum of squares of..?)

for i in [0,1,2,3]: [ 1st.Run ) nm_ar[0]
print(sm_row) > [0.01403589,

(1st Run) sm_row = 0 ____ ___ 0.11649791,

for (val ]in (nm_ar[i j: > 0.98251251,
(sm_row)+= val* val L * 0.1445697]
<_________________________________ 2

Z ----
[Sum of. )

2nd Run)-
i-{1.0 )«---------
1.0
1.0 Output
\______________ ✓
 ' •/ '
105-- DATA ANALYSIS S PROCESSING
--- '•
Binarization
In binarization we binarize our data i.e. reduce
differences to only two to leave crisp vales with
a threshold. For exaple if we set the threshold to
10, all the value in a data set under 10 will be
converted to 0 and above 10 will be converted into
1. Let's binarize our data with Binarizer class
and transformO method
In [21]: import pandas
from sklearn import preprocessing
dt = pandas.read_csv('trees.csv')
ar = dt.values
Nm = preprocessing.Normalizer(norm=111')
nm_ar = Nm.transform(ar)

Bin = preprocessing.Binarizer(threshold=0.1)
bin_ar = Bin.transform(nm_ar)
bin_ar[10:16]

array([[0., 0., i-L

[0., 0-, i.],
[1-, 0-, 1., i-L
[1., 1-, 1-, i-L
[1-, 0-, 1-, i-L
[1-, 1-, 1., i-]])

As you can see used the normalized (LI) that had

a range of 0 to 1 which made things easier to set
a threshold which is specified in the threshold
parameter i.e. 0.1
So all the values below 0.1 are changed to 0 and
all the values above 0.1 are changed to 1

0
 ' •/ '
106-- DATA ANALYSIS S PROCESSING
--- '•
Standardization
Standardization or Standard scaling is the method
of changing the distribution of data arttributes
to Gausiann distribution (Normal distribution). In
this mthe mean is changes to 0 and standard devia­
tion is changed to 1. Let's standardize our data
using the StandardScater class and it's fit()
and transform() methods
In [14]: import pandas
import numpy
from sklearn import preprocessing
dt = pandas.read_csv('trees.csv')
ar = dt.values
# Standardizer
Std = preprocessing.StandardScaler().fit(ar)
std_ar = Std.transform(ar)
print(std_ar[0:3])

print('Mean:',round(numpy.mean(std_ar),2))
print('Std.Deviation:',round(numpy.std(std_ar),2))

[[-1.67705098 -1.60291968 -0.9572127 -1.22883711]

[-1.56524758 -1.50574137 -1.75488995 -1.22883711]
[-1.45344419 -1.44095583 -2.07396086 -1.23502119]]
Mean: -0.0
Std.Deviation: 1.0

While the StandardScater object intialization

we also called the fit() function to fit the
scaler to our ar array and also transformed it, if
you don't call the fit you'll get an error like
This StandardScater instance is not fitted
yet. Catt 'fit' with appropriate arguments
before using this estimator
you can also use the fit & transform functions in
the previous preprocessing methods, for
demonstration purpose they aren't used in previous
examples but make sure to use them in it's
applications
Note that we used meant) and std() functions of
the numpy package to calcualte the mean and
standard deviation i.e. 0 and 1
 DATA ANALYSIS 8 PROCESSING

Label encoding
In many cases our data has more labels (word)
than features (numeric) but using words (strings)
in processing limits many activities. For that
purpose we need to change those labels into
numeric notations or features like the following
example

In [15]: import pandas

from sklearn import preprocessing
dt = pandas.DataFrame({'Questions':['A','B','C','D', 'E'],
'Answers':['True','True','False','True','False' ]})
dt

Out[15]:
Questions Answers

0 A True

1 B True

2 C False

3 D True

4 E False

We can use the Label-Encoder class for label encoding

In [17]: import pandas

from sklearn import preprocessing
dt = pandas.DataFrame({'Questions':['A','B','C','D','E'],
'Answers':['True','True','False','True','False']})
Enc = preprocessing.LabelEncoder( )
Enc.fit(dt['Answers'])
# Encoded LabeLs
dt['Answers'] = Enc.transform(dt['Answers'])
dt

0ut[17]:
Questions Answers

0 A 1

1 B 1

2 C 0

3 D 1

4 E 0
 ' •/ '
108 DATA ANALYSIS S PROCESSING
--- '•
As you can see that we had the Questions as A-E
and Answers as True or False. But we encoded the
Answers label to be 0(False) or l(True)
If we want we can get the label for the value or
decode the 0 or 1 values using the
inverse_transform() function
In [18]: import pandas
from sklearn import preprocessing
dt = pandas.DataFrame({'Questions':['A','B','C','D','E'],
'Answers':[’True', 'True','False','True','False']})
Enc = preprocessing.LabelEncoder()
Enc.fit(dt['Answers'])
# Encoded LabeLs
dt['Answers'] = Enc.transform(dt['Answers'])
# Decoding LabeLs
print(Enc.inverse_transform([0,1]))

['False' 'True']

By encoding we can hide the true values and

perform a lot operations with them because they are
numerical values. In this data we had less only two
label values i.e. True and False, but when there
are more values the encoding will range from 0 to
their respective lengths
 DATA
VISUALIZATION
• Plotting data
• Univariate plots
• Multivariate
plots

o
 13 data visualization
J
X______ / □------------------
Python has excellent libraries for data
visualization. A combination of Pandas., numpy and
matplotlib can help in creating in nearly all types
of visualizations charts. We can use the visuals to
better understand our data

Plotting data
We use numpy library to create the required
numbers to be mapped for creating the chart and the
pyplot module of matplotlib which draws the
actual chart

In [11]: import numpy as npy

import matplotlib.pyplot as pit

x = npy.arange(0;10) ^outputs [0123456789]

y = x A 2
plt.plot(x,y)

Out[ll]: [<matplotlib.lines.Line2D at 0x26clalbcl00>]

The arange(0,10) function creates a ndarray of

numbers from 0 to 10 [excluding 10] and the plot
function plots a simple chart of the data we
provided. We can also plot any imported data using
the same
 H DATA VISUALIZATION

y
Editing labels and colors
As we already know matplotlib uses MATLAB symbols
as formatted strings to customize the colors [refer
to page 32] and how to add labels to the plots

In [15]: import numpy as npy

import matplotlib.pyplot as pit
x = npy.arange(0,10)
y = x A 2
# Editing LabeLs
pit.title("Matplotlib")
pit.xlabel("X_Axis")
pit.ylabel("Y_Axis")
# Editing Line styLe and coLor
plt.plot(x,y,'c')
plt.plot(x,y,■*')

Out[15]: [<matplotlib.lines.Line2D at 0x26cla312130>]

We added a title for the plot Matplolib using the

title function, X axis and Y axis labels using
xlabel and ylabel functions respectfully.
The ‘c3 represents cyan which is the color of the
line with ‘*3 as symbol for stars
Note that we didn't passed these values in any
parameter because they are treated as formatted
strings matplotlib.pyplot. plot still interprets
it as positional argument
 VISUALIZATION

Univariate plots
Univariate plots are the type of plots or visuals
with different plots for each variable to
understand them individually. We will use the
trees.csv data, which we used in the previous
lesson. First of all let's plot histogram for each
variable using the hist() function of the pandas
library

In [1]: import pandas

dt = pandas.read_csv('trees.csv')
dt.hist()

Out[l]: array([[cmatplotlib.axes._subplots.AxesSubplot object

<matplotlib.axes._subplots.AxesSubplot object
[<matplotlib.axes._subplots.AxesSubplot object
cmatplotlib.axes._subplots.AxesSubplot object
dtype=object)

As you can see we have histograms plotted for

each variable i.e. Girth, Height, Volume and
Index(which can be ignored). Inspecting the
visuals we can get a lot of information about the
data, it's distribution, maximum value, minimum
value, etc. We can also visualize individual
variables using the hist() function and the
sliced variable or the column
 z—• /
113 DATA VISUALIZATION
-
In [2]: import pandas
dt = pandas.read_csv('trees.csv',
names=['Index'>'Girth','Height'>'Volume'])
dt.drop(0,inplace=True)
dt['Height'].hist()

70 65 63 72 81 83 66 75 80 79 76 69 74 85 86 71 64 78 77 82 87

Note that we renamed the columns while importing

the csv data using the names parameter of the
reacLcsvO function., but the default column names
will be stored as the first row, we need to remove
that, to do so we use the dropO function and
provide the 0 as the index i.e. first row and also
True for the inptace parameter which will remove
that from the actual data
Next is density plots which are similar to
histograms but have smooth lines like the below
chart
 114 DATA VISUALIZATION
- •
We craete density plots using the plot()
function of pandas library and specifying
density in the kind paramter
In [7]: import pandas
dt = pandas.read_csv('trees.csv')
del dt['Index']
dt.plot(kind='density')

0ut[7]: <matplotlib.axes._subplots.AxesSubplot at 0x261f2dc

If you want seperated plots, specify True in

subplots parameter
dt.plot(kind='density',subplots=True)

0ut[8]: array([<matplotlib.axes._subplots.AxesSubplot object

<matplotlib.axes._subplots.AxesSubplot object
<matplotlib.axes._subplots.AxesSubplot object
dtype=object)
 115 DATA VISUALIZATION
- •

Another univariate plot called box and whiskers

plot can be used too. We can specify box in the
kind parameter of ptot() function this time

In [9]: import pandas

dt = pandas.read_csv('trees.csv')
del dt['Index']
dt.plot(kind='box'>subplots=True)

Out[9]: "Girth (in)" AxesSubplot(0.125,0.125;0.227941

"Height (ft)" AxesSubplot(0.398529,0.125;0.227941
"Volume(ftA3)" AxesSubplot(0.672059,0.125;0.227941
dtype: object

The box plots have the following features:

'Girth (in)'
 z—• /
116 DATA VISUALIZATION
-
Multivariate plots
As it sounds, through multivariate plots we can
understand the realtions btween different
attributes or variables in a data. One of the
multivariate plots is a correlation matrix. We can
plot correlation matrix for our data like so:

In [16]: from matplotlib import pyplot

import pandas
dt = pandas.read_csv('trees.csv')
del dt['Index']
cor = dt.corr() # correLations
# PLotting correLation matrix
vis = pyplot.figure()
# Adding the coLor meter
ax = vis.add_subplot(lll)
cax = ax.matshow(cor, vmin=-l, vmax=l)
vis.colorbar(cax)
pyplot.show()

0 12

I
We used the figure() function of the matlplotlib
library, along with the matrix we also added a color
bar to indicate the value of colors using the
correlations data i.e. calculated using the corr()
function. We added the matrix using the
add_subplot() function and 111 as argument i.e. it's
position. Then we used the matshow() function and
passed our correlations and min and max values.
 117 DATA VISUALIZATION
- •

Finally we plotted our matrix with the colorbar

We can also label the axes as our column names
using the set_xticks() and set_yticks()
functions to set the positions and
set_xticktabet0 and set_yticktabetO functions
to label them accordingly

In [19]: vis = pyplot.figure()

ax = vis.add_subplot(lll)
cax = ax.matshow(cor, vmin=-l, vmax=l)
vis.colorbar(cax)
ax.set_xticks([0,1,2])
ax.set_yticks([0,1,2])
ax.set_xticklabels(['Girth','Height','Volume'])
ax.set_yticklabels(['Girth','Height','Volume'])
pyplot.show()

We can easily understand the relations between

the variables i.e. Girth & Volume is close to 1
correlation value(full positive) and Height &
Volume have a bit less realtion than Girth &
Volume. We can also view scatter matrix plots to
understand realtions between variables or attri­
butes using dot graphs. We can do so using the
pandas scatter_matrix0 function. We need to use
the pandas.plotting.scatter_matrix() syntax to
access the function and pass the data inside of
the parenthesis of the function
 z—• /
118 DATA VISUALIZATION
-

In [1]: from matplotlib import pyplot

import pandas
dt = pandas.read_csv('trees.csv')
del dt['Index']
pandas.plotting.scatter_matrix(dt)

Out[l]: array([[<matplotlib.axes._subplots.AxesSubplot object

<matplotlib.axes._subplots.AxesSubplot object
<matplotlib.axes._subplots.AxesSubplot object
[<matplotlib.axes._subplots.AxesSubplot object
<matplotlib.axes._subplots.AxesSubplot object
<matplotlib.axes._subplots.AxesSubplot object
[<matplotlib.axes._subplots.AxesSubplot object
<matplotlib.axes._subplots.AxesSubplot object
<matplotlib.axes._subplots.AxesSubplot object
dtype=object)

s 20

Ju.
••
• •
15
§ • • • • * .s 7’
(3
10
L •

80

70
mil
25

9
• V •’

a a R
•
e• •
•• •
------- ;-------- -—
•

8
11.
a a £
"Girth (in)" "Height (ft)’ "Volume(ft/K3)"

Girth & Volume

X

Ju.
• e •••
• •
•••

t a ’

uiL
• V

L-r----------- 1------------ r-1 i kJ

29998 a r a
"Girth (in)" "Height (ft)" "Vblume(ft/'3)"
 1 /I REGRESSION
14 ALGORITHM

• What is
Regression
• Regressor
model
• Linear regression

o J
v
14 J REGRESSION
J
□------------
What is regression?
As we already know about the regression algorithm
which is a case of supervised machine learning
where we feed input data(numeric) and the algorithm
learns the patterns in the data and predicts the
output data. The performance of the algorithm is
entirely based on it's learning so we need to do
our best to feed the best data to our model. So
letJs create a regressor with the scikit-learn-’s
readymade algorithms

Regressor Model
LetJs create a simple regressor model to predict
the weight of a person if the height is given as
input value. You can download the file from here.
Then jump to your jupyter notebook
and import the following modules to
start the model http://bit.ly/HaitWet

import pandas
from sklearn import Linear_model
import sklearn.metrics
import matplotlib.pyplot

In [1]: import pandas

from sklearn import linear_model
import sklearn.metrics
import matplotlib.pyplot

We need pandas library to import our csv data,

linear_model to create the regressor model,
sklearn.metrics to evaluate the accuracy of our
model and matplotlib to visualize our data in
multiple steps
Now we can import the csv data and have a look at
it before training the model
 z—• /

121 REGRESSION

In [2]: import pandas

from sklearn import linear_model
import sklearn.metrics
import matplotlib.pyplot
dt = pandas.read_csv('height_and_weight.csv')
dt.head()

Out[2]:
Index Height(ln) Weight(lbs)

0 1 65.78 112.99

1 2 71.52 136.49

2 3 69.40 153.03

3 4 68.22 142.34

4 5 67.79 144.30

we imported tne csv data and inspect tne Tirst

five rows using the headO function (by default
if oyu don't pass any value it will show the first
5 rows). We already have indexes so we can remove
the Index column. We can check the distribution
of the data by plotting a histogram of the data
using the pandas hist() function

In [3]: import pandas

from sklearn import linear_model
import sklearn.metrics
import matplotlib.pyplot
dt = pandas.read_csv('height_and_weight.csv')
del dt['Index']
dt.hist()

0ut[3]: Height(ln) Weight(lbs)

40

20

0
A 65 70

We have a normal distributed data so we don't

0
100 120 140 160

need to do any preprocessing, we can move onto

training the model
 z—•
122 REGRESSION
•
But we need to seperate the values into input
data i.e. X and output data i.e. y. For our input
data we will use the height data and for the
output we will use the weight data. So moving onto
a new cell (Hotkey : b) let's seperate the data
into inp_X and out_y variables

In [4]: inp_X = dt['Height(In)']

out_y = dt['Weight(lbs)' ]

Now we can create our regression model RegModel

using the LinearRegression class and use to
fit() method and pass the input value inp_X and
output value out_y as arguments

In [5]: inp_X = dt.drop(columns=['Weight(lbs)'])

out_y = dt.drop(columns=['Height(In)'])
RegModel = linear_model.LinearRegression()
RegModel.fit(inp_X,out_y)

0ut[5]: LinearRegression()

And our model is ready! Yes, we can ask the model

to make predictions for weight of a person if it's
height is 60 inches. We will use the predict()
method to do the same

In [6]: RegModel.predict([[60]])

Out[6]: array([[99.93286131]])

So as you can see if a person has height of 60

inches then the weight of the person will be
approx 99.93 pounds, is what our model predicted.
As you can see we are in a new cell and used the
predictO function to predict the value and we
also passed the value as [[60]], because actually
the value will be treadted as numpy.array([[60]])
and if you pass 60 or [60] you'll encounter an
error because the the dataset for training is a
2-D array so we need the same for predictions. And
if a question is tinkering you that how did the
 z—•
123 REGRESSION
•

model made prediction just review the first

lesson. But we don't know whether the value is
correct or not? Then we can't calculate the accu­
racy of the model either. Then we need to divide
our dataset into training set say 90% of the data
to train the model and 10% as testing set whose
input values will be provided to the model to make
predictions and then we will compare the predic­
tions with the real ones. In the cell where we
trained the model, we will divide the input values
i.e. Height into 90% in inp_X and 10% in tst_X
and the same with the output values

In [1]: import pandas

from sklearn import linear_model
from sklearn.model_selection import train_test_split
import sklearn.metrics
import matplotlib.pyplot
dt = pandas.read_csv('height_and_weight.csv')
del dt['Index']

In [2]: height = dt.drop(columns=['Weight(lbs)'])

weight = dt.drop(columns=['Height(In)'])
inP_Xj tst_Xj out_y, tst_y = train_test_split(
heightjweightj test_size=0.1)
RegModel = linear_model.LinearRegression()
RegModel.fit(inp-XjOut-y)

Out[2]: LinearRegression()

To ease up the data spliting task we used the

train_test_sptit () function in the
sklearn.model_selection. That's why we imported
the function in the first cell where we imported
data and the required libraries and modules. In
the second cell we divided the input data and
output data into height and weight. Then we
created four variable inp_X, tst_X, out_y and
tst_y to store input for training, input for
testing, output for training and output for
comparing the predictions respectively using the
 124 REGRESSION

train_test_sptit() function. We passed the input

data height, ouput data weight as arguments and
0.1(10%) to the test_size parameter. The spliting
and assigning of the data can be understood from
the below figure:

after that as we did earlier we create our model

and train it. Now we can ask our RegModet to
predict the weight of the tst_X values and
compare it with the tst_y values

In [3]: pred_y = RegModel.predict(tst_X)

cmp = pandas.DataFrame({'Predictions':pred_y.flatten(),
'Actual values':tst_y['Weight(lbs)'].values})
cmp.plot(kind='bar',figsize=(7.5,6))

Out[3]: <matplotlib.axes._subplots.AxesSubplot at 0x2038d756cl0>

 z—•
125 REGRESSION
•
We used the predict() method and passed the test
height values tst_X as arguments and stored the
predictions in the pred_y. Then we created a data
frame using the pred_y values (we flattened the
array using the flatten () function to change it
to 1-D array from 2-D array before passing it as
values) and tst_y actual values (we sliced the
Weight(lbs) column and extracted itJs values) and
plotted a grouped bar graph using the ptot()
function and ban to kind parameter and also
specified the size of the plot using figsize
parameter. As you can observe, the predicted
values as blue and actual values are orange so we
can tell visually what is the performance of our
model. Most of the bars are close to each other
but some are way taller or shorter than the other
i.e. our model is performing good but in some
cases it predicted wrong. We can also visualize
the regression line and data points

In [4]: pyplot.scatter(tst_X, tst_y, color='black')

pyplot.plot(tst_X, pred_y, color='yellow')
pyplot.show()

140

130

120

110

100

64 65 66 67 68 69 70 71

We used the scatter() function to plot the data

points and plot() function to plot the regression
line
 z—•
126 REGRESSION
•

To know how much error is their in the

predictions or the MAE (Mean Absolute Error) we
can use the mean_absotute_error() function in
sktearn. metrics. It will return an average of
the differences between the predicted and actual
values
In [6]: metrics.mean_absolute_error(tst_y, pred_y)

Out[6]: 11.529538265252379

We passed our actual values tst_y, predicted

values pred_y as arguments to the function. So at
an average the error values will have 11.5 pounds
of difference from the actual weights with our
current model

Linear Regression
As learnt earlier, regression algorithms find
relationship between two variables and predict the
values based on that relation. In linear
regression, the algorithm finds the linear
relationship between two variables i.e. if a
independent varaible is changed the dependent
variable will be affected. For example consider
the following dataset, we have data of different
squares with itJs length of one side and itJs
area. And it is clear that if the side
(independent variable) is changed say increased
the area (dependent variable) will
change(increase) too because area is calculated
from the length of the side

Side 1_____ Area_____ 1

 z—• /

127 REGRESSION

Mathematically the relation can be expressed as:

Y = mX + b
where., Y is the dependent(side), X is the
dependent variable(area) we are using to make
predictions, m is the slop of the regression line
which represents the effect X has on Y and b is a
constant, known as the Y-intercept. If X = 0,Y
would be equal to b
The relationship can be either positive or
negative

Simple Linear Regression

Simple linear regression or SLR is the types of
regression in which we predict a value using only
one feature like we did before, we predicted the
weight of a person using the height

140

130

120

no

100

64 65 66 67 68 69 70 71
Multiple Linear Regression
Multiple Linear Regression or MLR is the type of
regression in which we predict a value using two
or more features like predicting the weight of a
person using the age and height values. Here, the
regression line is calculated using the following
formula:
h(xj = bo + bixu + ... + bpXiP+

where p is the number of the features, h(x) istthe

predicted value, bo, bi, etc. are regression
coefficients and e aretthe residual errors i.e.
errors in the data. This also gives rise to the
following formula

yt= h(xi) + ei

where y is the actual value or the dependent

variable
The training process for a MLR model is same as
we did before, but this time we have more features
from which predictions will be made. LetJs create
a model to predict the salary of person if we get
the age and gender as input. You can download the
dataset from here
We need to need to import the same
things as we did we before bit.ly/MallCust

http://bit.ly/MallCust
import pandas
from sktearn import tinear_modet
from sktearn import metrics
from matptotlib import pyptot
from sktearn.modet_setection -
-import train_test_sptit

we will import train_test_split before

 z—• /

129 REGRESSION

and also import the data set and inspect the first
five rows using the headO function

In [1]: import pandas

from sklearn import linear_model, metrics
from sklearn.model_selection import train_test_split
from matplotlib import pyplot
dt = pandas.read_csv( 'Mall_Customers.csv')
dt.head()

CustomerlD Genre Age Annual Income (k$) Spending Score (1-100)

0 1 Male 19 15 39

1 2 Male 21 15 81

2 3 Female 20 16 6

3 4 Female 23 16 77

4 5 Female 31 17 40

Our data has 200 rows and 5 columns. As you can

see the Male and Female are not numeric values
which is fine, but it is best time to pratice our
preprocessing skills. We will encode them into
numeric values. If you don't remember how to do
so, go back to the Data analysis and processing
lesson
In [2]: import pandas
from sklearn import linear_model_> metrics
from sklearn.model_selection import train_test_split
from matplotlib import pyplot
from sklearn import preprocessing
dt = pandas.read_csv('Mail-Customers.csv')
Enc = preprocessing.LabelEncoder( )
Enc.fit(['Male','Female'])
dt['Genre'] = Enc.transform(dt['Genre'])
dt.head()

CustomerlD Genre Age Annual Income (k$) Spending Score (1-100)

0 1 1 19 15 39

1 2 1 21 15 81
2 3 0 20 16 6
3 4 0 23 16 77

4 5 0 31 17 40
 z—•
130 REGRESSION

We have encoded our Male and Female labels into 1

and 0 features respectively. Now we can move onto
a new cell and split our data

In [3]: Input = dt.drop(columns=['CustomerlD'}

'Annual Income (k$)',
'Spending Score (1-100)'])
Output = dt.drop(columns=['CustomerlD',
'Genre'>'Age',
'Spending Score (1-100)'])
inp_X,tst_X,out_y?tst_y = train_test_split (
Input., Out put, test_size=0.1)

The input values are stored in Input and the

output are stored in the Output. Then we splitted
the data for training and testing using the
train_tets_sptit() function. Now we can create
our model and train it

In [4]: Input = dt.drop(columns=['CustomerlD',

'Annual Income (k$)',
'Spending Score (1-100)'])
Output = dt.drop(columns=['CustomerlD',
'Genre', 'Age',
'Spending Score (1-100)'])
inp_Xjtst_X,out_y,tst_y = train_test_split(
Input4OutputJtest_size=0.1)
# Training Model.
RegModel = linear_model.LinearRegression()
RegModel.fit(inp_X,out_y)

0ut[4]: LinearRegression()

Our RegModet is ready to predict the annual

income of people if the gender and age are passed
as input. But we don't know the range of age here
so we can execute the following code to know so

In [31]: dt['Age'].max() - dt['Age'],min()

Out[31]: 52

The range of the age in the dataset is 52

 z—• /

131 REGRESSION

Before comparing the predictions and actual

values we can ask the model to predict some values
like how much a 30 years old female is earning and
how much a 42 years old male is earning

In [5]: RegModel.predict([ [0,30], [1,42] ])

Out[5]: array([[60.5504497 ],
[62.09356734]])

Note the way we passed the values. As Female and

Male are encoded as 0 and 1, we pass [ [0,30],
[1,42] ] and our model is telling that a 30 years
old female earns about 60.5k and 42 years old male
earns about 62k. Well let's predict the test input
and compare the results with actual ones
In [6]: pred_y = RegModel.predict(tst_X)
cmp = pandas.DataFrame({'Predictedpred_y.flatten()j
'Actual':tst_y['Annual Income (k$)'].values})
cmp.plot(kind='bar',figsize=(7.5,6))

0ut[6]: <matplotlib.axes._subplots.AxesSubplot at 0xl402ccc89a0>

As you can observe, most of the values are

 z—• /

132 REGRESSION

incorrect because of the distribution of the data,

so what do we do now?
There are lot of ways to improve the performance
of model, we can increase the data as in our case
we have only 200 rows, to predict values precisely
we need at least 10 times the data we have now
because the distribution isn't noraml in this case
In [7]: cmp['Predicted'].plot(kind='density')
pyplot.show()
cmp['Actual'].plot(kind=*density',color='orange')
pyplot.show()

We will learn about more methods to improve the

performance of our models in detail in the
performance and metrics lesson
<Or
1 E CLASSIFICATION
!□ ALGORITHM

• Decision Tree I 0 n
"I I
• Logistic In 0
regression
• Nolve Boyes
I
I

o J
 ar A

15 CLASSIFICATION
_____ / □-------------- J

We already know what is a classification

algorithm, but there are two types of
classification algorithms - lazy learners and eager
learners i.e. lazy learners learns less during
trainging but more in predicting like KNN
algorithms but eager learners learns in training
and less in testing like decision tree, Naive
Bayes, etc. Now let's create a classifier using the
decision tree

Decision tree
Let's create a classifier to classify a customer
into which falvour he/she likes if the age and
gender are provided as input. To do so we will use
the decision tree, an algorithm that can perform
both classification and regression tasks. It learns
the categories in a dataset and creates categories
using a decision tree and then predicts the
category of an input
We will use the scikit learns
DecisionTreeClassifier to create our
model. You can download the data set
from here -
Now jump onto your jupyter notebook
and import the packages and modules
needed in this project

import pandas
from sklearn.tree import -
-DecisionT reeCtassifier
from sklearn.model_selection import
train_test_split (optional)

In [1]: import pandas

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
 —•
135 CLASSIFICATION

Now we can import our dataset and previwe it

using the head() function
In [2]: import pandas
from sklearn.tree import DecisionTreeClassifier

dt = pandas.read_csv('flavour.csv')
dt.head()

Age Gender Flavour

0 6 Male Chocolate

1 6 Female Strawberry

2 7 Male Chocolate

3 8 Female Strawberry

4 11 Male Butterscotch

As you can see we need to encode the Gender

labels into numeric values. So let's import the
preprocessing module and encode the labels

In [3]: import pandas

from sklearn.tree import DecisionTreeClassifier
from sklearn import preprocessing
dt = pandas.read_csv('flavour.csv')
Enc = preprocessing.LabelEncoder().fit(['Male',
'Female'])
dt['Gender'] = Enc.transform(dt['Gender'])
dt.head()

Out[3]
Age Gender Flavour

0 6 1 Chocolate

1 6 0 Strawberry

2 7 1 Chocolate

3 8 0 Strawberry

4 11 1 Butterscotch

So the Mate and Female labels are encoded into 1

and 0 respectively using the LabetEncoder. Note
that we fitted Mate and Female while intialization
 z—• /

136 CLASSIFICATION

Now we can move onto a new cell and split our

data into input and output. Then we will create
our classifier CModel using DecisionTreeClassifier
and training the model using the input and ouput
values
In [4]: inp_X = dt.drop(columns='Flavour')
out_y = dt.drop(columns=['AgeGender'])
# CLassifier
CModel = DecisionTreeClassifier()
CModel.fit(inp_X,out_y)

0ut[4]: DecisionTreeClassifier()

So we trained our model using the fit() function

and our model is ready to make predictions. Let's
ask our model which flavour will a 7 years old boy
and 9 years old girl will prefer:

In [5]: CModel.predict([ [7,1], [9,0] ])

Out[5]: array(['Chocolate', 'Strawberry'], dtype=object)

According to our model a 7 years old boy [7,1]

will prefer chocolate and a 9 years old girl [9,0]
will prefer strawberry which is pretty much
correct. But we trained our model with only 20
rows of data but we are getting some descent
results. But what if we ask the model to predict
what a 30 years old women will prefer, i.e. beyond
the range of age we provided in the data

In [8]: CModel.predict([[30,0]])

Out[8]: array(['Coffee'], dtype=object)

According to our model a 30 years old women [7,1]

will prefer coffee. Which maybe not correct
because the maximum age for women provided in the
dataset is 24. So how did the model predicted that
value? Let's find out how our model is performing
classifications
 z—• /

137 CLASSIFICATION

To view the the decision tree or the algorithm

which is used by the model to classify the values
can be seen using the below code

In [1]: import graphviz

from sklearn.tree import export_graphviz

exP data = export_graphviz(CModel,

out_tiie= ’at-tree.aor ’ ,rearure_names= [ 'Age', 'Gender' ],
class_names=['Chocolate', 'Strawberry',
'Butterscotch', 'Vanilla', 'Mango',
'Almond_Choco', 'Coffee'],
filled=True, rounded=True,special_characters=True)
graph = graphviz.Source(data)
graph

Age < 20.5 ]

gini = 0.825
samples = 20
value = [3, 6. 2, 3, 2. 2, 2]
class = Strawberry
True .False

f Age < 9.0 Age < 23.0 ~\

gini = 0.781 gini = 0.375
samples = 16 samples = 4
value = [2. 6, 2. 0, 2, 2. 2] value = [1.0. 0, 3, 0, 0,0]
class = Strawberry class = Vanilla
J

Gender < 0.5 Gender < 0.5 Gender < 0.5 s

gini = 0.0
gini = 0.5 gini = 0.667 gini = 0.5
samples = 2
samples = 4 samples = 12 samples = 2
value = [0. 0. 0, 2, 0, 0, 0]
value = [0, 0, 2, 0, 0, 2, 0] value = [2. 6, 0. 0. 2, 0, 2] value = [1,0, 0,1,0, 0,0]
class = Vanilla
class = Butterscotch y class = Strawberry class = Chocolate

' Age < 15.5 Age < 13.5

gini = 0.0 gini = 0.0 gini = 0.0 gini = 0.0
gini = 0.444 gini = 0.667
samples = 2 samples = 2 samples = 1 samples = 1
samples = 6 samples = 6
value = [0. 0, 0. 0. 0, 2, 0] value = [0, 0. 2, 0, 0. 0. 0] value = [0, 0. 0,1,0. 0,0] value = [1.0, 0, 0. 0, 0. 0]
value = [0. 4, 0, 0. 0, 0, 2] value = [2. 2, 0, 0, 2, 0, 0]
k class = Almond_Choco class = Butterscotch class = Vanilla class = Chocolate
class = Strawberry class = Chocolate

Age < 12.0

gini = 0.0 r gini = 0.0
Age < 18.0
gini = 0.444 gini = 0.5
samples = 3 samples = 2
samples = 3 samples = 4
value = [0. 3, 0. 0. 0, 0. 0] value = [0. 2, 0. 0. 0, 0. 0]
value = [0,1, 0, 0, 0, 0, 2] value = [2, 0, 0, 0, 2, 0, 0]
class = Strawberry class = Strawberry
class = Coffee J y class = Chocolate

gini = 0.0 gini = 0.0 r gini = 0.0 gini = 0.0

samples = 1 samples = 2 samples = 2 samples = 2
value = [0, 1, 0, 0, 0, 0, 0] value = [0, 0, 0, 0, 0, 0, 2] value = [0. 0. 0. 0. 2. 0. 0] value = [2, 0. 0. 0. 0. 0. 0]
class = Strawberry j ,class = Coffee class = Mango class = Chocolate

Before visualizing the decision tree you need to

install the graphviz using the below code in your
anaconda prompt
conda install -c conda-forge python-graphviz
Then import the export_graphviz and graphviz.
Using export_graphviz we visualize the decision
tree and store it in data and using Source function
of graphviz we view the tree
But as you can see the tree is much big to fit in
the page so let's understand it by breaking it down
 z—•
138 CLASSIFICATION
• -J

Let's see what will happen if [7,1] (7-year old

boy) is the input

Starting from the root the first comparison is

whether the age is less than or equal to 20.5 Age
20.5 and the age of the input is 7 so we move
on to the True side

Next the algorithm compares whether the age is

smaller than or equal to 9, and our age is smaller
so we move down (straight to green)
 z—•
139 CLASSIFICATION
• -J

Now we are checking whether the gender is smaller

than or equal to 0.5 i.e. 0 or not. But here our
gender is 1 so we move to the non green side

Once again we compare age, whether age is smaller

to equal or smaller than 18 and 7 is so we finally
stop at the orage box (orange = True, blue =
False) and choose the class i.e. Chocolate

Starting from the root we reach the conclusion

that the input has a class chocolate i.e. a 7
years old boy likes chocolate
But you may what are the other attributes present
there like gini, samples, etc. Gini is the name of
the cost function that is used to evaluate the
binary splits in the dataset and works with the
categorial target variable “Success” or “Fail­
ure”.A perfect Gini index value is 0 and worst is
0.5 which used to split further or not. You can
see gini score with 0 aren't splitted further,
samples is the number of data points in the given
dataset with the respective characterestics
 /

140 CLASSIFICATION

Here is the full decision tree for the

classification of the 7 years old boyJs
preference. If you want to look at the whole tree
in more quality execute the code
Z" Age < 20.5
gini = 0.825
samples = 20
value = [3, 6. 2, 3. 2, 2. 2]
class = Strawberry y

z Age <9.0 "x z Age S 23.0 >

gini = 0.781 gini = 0.375
samples = 16 samples = 4
value = [2. 6. 2, 0. 2. 2. 2] value = [1.0. 0. 3. 0. 0,0]
class = Strawberry
J class = Vanilla j

Gender < 0.5 Gender < 0.5

gini = 0.5 gini = 0.667
samples = 4 samples = 12
value = [0. 0. 2. 0. 0. 2. 0] value = [2. 6. 0, 0. 2, 0. 2]
class = Butterscotch class = Strawberry
y

z Ages 15.5 Ages 13.5

■x
gini = 0.444 gini = 0.667
samples = 6 samples = 6
value = [0. 4. 0. 0. 0. 0. 2] value = [2, 2. 0. 0. 2. 0. 0]
class = Strawberry y class = Chocolate

z gini = 0.0
Ages 18.0
gini = 0.5
samples = 2
samples = 4
value = [0. 2. 0. 0. 0. 0. 0]
value = [2. 0. 0, 0. 2. 0. 0]
class = Strawberry y
class = Chocolate j

z gini = 0.0 ■> gini = 0.0

samples = 2 samples = 2
value = [0. 0. 0, 0. 2, 0. 0] value = [2. 0. 0. 0. 0. 0, 0]
class = Mango class = Chocolate

Likewise we can use the decision tree to solve

different kind of problems based on classification
But you may also ask how does the tree creates
those comparisions or splits? It isn-’t necessary
to know but you should. First the algorithm
calculates the gini index for each attribute using
the below formula:
p2 + q2
which is the sum of the square of probability for
success(p) and failure(q). Then the dataset is
splitted into two lists of rows having index of an
attribute and a split value of that attribute.
Then it finds the best possible split by
evaluating the cost(gini) of the split
 sCLASSIFICATION

Logistic Regression
Logistic regression is a type of model that
predicts the outcome of output values as Yes or no
as numeric values 1 or 0 respectively. We can use
these type of models to classify a day as rainy or
not, a person as healthy or sick, etc. But there
are different types of logistic regression used
for to different situations.

Binomial Logistic Regression

Binomial or binary logistic regression used to
predict exactly two outcomes i.e. either
l(positive) or 0(negative)
Let's use an dataset to predict whether it will
rain or not if the temperature and humidity
percent are provided as input. You can download
the data set from here -
Let's import the modules and the http://bit.ly/RainBin
dataset together. This time we will
import linear_modet and
train_test_sptit from sklearn

import pandas
from sklearn import Linear_model
from sklearn.modet_setection import -
- train_test_split
from sklearn.metrics import accuracy_score

In [1]: import pandas

from sklearn import linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

We also imported the accuracy_score() function

from sklearn. metrics to calculate the accuracy
of our model. Now we can import our dataset and
this time let's view it as it is
 /
142 CLASSIFICATION

In [2]: import pandas

from sklearn import linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
dt = pandas.read_csv('Rainfall_data.csv')
dt

Out[2]:
Unnamed: 0 Temperature Humidity% Rain

0 0 34 74.2 Yes

1 1 19 68.2 No

2 2 28 67.2 Yes

3 3 29 66.6 Yes

4 4 26 57.9 Yes

19995 19995 30 77.9 Yes

19996 19996 20 74.8 Yes

19997 19997 14 69.4 No

19998 19998 20 60.6 No

19999 19999 22 64.8 No

20000 rows x 4 columns

As you can see we have 20000 rows and 4 columns

worth of data! Now we can move on to a new cell
and perform the splitting of the data into
train-test input and train-test output
In [3]: Input = dt.drop(columns=['Unnamed: O'^'Rain'])
Output = dt['Rain']
inp-X, tst_X, out—y, tst_y = train_test_split(
Input.values,Output,test_size=0.01)

We stored the input features i.e. Temperature in

Input and the output i.e. Rain (Yes or No) in
Output. Then we passed these values to the
train_test_sptit () function and splitted the
data into training input, testing input, trainging
output and testing output where the test size is
 143 CLASSIFICATION
•

0.01 (1% i.e. 200). Now we can create out logistic

regression CModel and train it
In [4]: Input = dt.drop(columns=['Unnamed: 0','Rain'])
Output = dt['Rain']
inp_Xj tst-X, out_y, tst_y = train_test_split(
Input.values,Output,test_size=0.01)
CModel = linear_model.LogisticRegression()
CModel.fit(inp_X,out_y)

0ut[4]: LogisticRegression()

So our model is ready to make predictions, let's

move onto a new cell and let the model predict.
Then we will compare the values and print the
accuracy score
In [5]: from sklearn import preprocessing
pred_y = CModel.predict(tst_X)
Enc = preprocessing.LabelEncoder().fit(['Yes'No'])
cmp = pandas.DataFrame({'Predicted':Enc.transform(pred_y),
'Actual':Enc.transform(tst_y)})
print('Accuracy Score:',accuracy_score(tst_y,pred_y))
cmp.plot(kind='density')

Accuracy Score: 0.91

0ut[5]: <matplotlib.axes._subplots.AxesSubplot at 0xlfb7a395d30>

So the model has accuracy score of 0.91 i.e. 91%,

which is really good! You can also see the density
plot where only 9% of values are predicted wrong
by the model
 144 CLASSIFICATION
•
So how did our model predicted teh values or how
do the logistic regression works? To understand we
will see what is the mathematics behind the
algorithm., if you want you can move ahead or give
it read. The followings are the steps of linear
function of binomial logistics regression:
• We already know that the output will be either
0(No) or l(Yes). For that the linear function is
basically used as an input to another function
such as g in the following relation

he(x) = g(0Tx) [0 he sS 1 ]
gis the logistic or sigmoid function which can be
found with the following formula:

where z is 0Tx
• We can visualize the sigmoid curve can be
understood by the following graph

the classes can be divided into positive or

negative. The output comes under the probability
of positive class if it lies between 0 and 1. For
our implementation, we are interpreting the output
of hypothesis function as positive if it is bigger
than or equal to 0.5 (>0.5), otherwise negative
• We also need to define a loss function to
measure how well the algorithm performs using
the weights on functions, represented by 9 and h
is equal to g(X0):
 145 CLASSIFICATION
•

after defining the loss function our prime goal is

to minimize the loss function
• It can be done with the help of fitting the
weights which means by increasing or decreasing
the weights. With the help of derivatives of the
loss function with respect to each weight, we
would be able to know what parameters should
have high weight and what should have smaller
weight. The following gradient descent equation
tells us how loss would change if we modified
the parameters:
3J(0)
=—XT (g(Xf)) — y)
SOj m

Multinomial Logistic Regression

As the name suggest this time we will have to pre­
dict outputs more than 2 times. In multinomial lo­
gistic regression we perform classification into 2
or more categories also the categories can be just
different types like Rain, Hailstorm, Snow, etc. or
ordinal like Heavy rain, moderate rain or low rain­
fall
Let's consider the previous situation where we
predicted whether it will rain or not,
bit.ly/RainMulti
so let's create a model to predict
whether it will rain heavy, moderate http://bit.ly/RainMulti
or low. You can download the dataset
from here -
and import the modules as we did
while creating model to predict the
rainfall

In [1]: import pandas

from sklearn import linear_model, metrics
from sklearn.model_selection import train_test_split
 —•
146 CLASSIFICATION
•
Now we can import our data and preview it without
the headO function

In [2]: import pandas

from sklearn import linear_model, metrics
from sklearn.model_selection import train_test_split
dt = pandas.read_csv('RainfallData.csv')
dt

Out[2]:
Temperature Humidity% Rainfall

0 34 74.2 Low

1 19 68.2 No Rain

2 28 67.2 Moderate

3 29 66.6 Moderate

4 26 57.9 Low

...

17996 31 89.7 No Rain

17997 21 84.7 No Rain

17998 28 74.7 No Rain

17999 30 78.2 No Rain

18000 34 80.4 Low

18001 rows x 3 columns

We have the same temperature, Humidity percent

columns but the rain is classified into No rain,
low, moderate and high. Now we can move onto the
next i.e. splitting the data

In [3]: Input = dt.drop(columns='Rainfall')

Output = dt.drop(columns=['Temperature','Humidity%'])
I inp_X,tst_X,out_y,tst_y = train_test_split(
Input,Output,test_size=0.1)

Next we need to scale our Input data (optional)

or we may encounter error. We will import
preprocessing module and scale our input data.
Then we can split our data into training and
testing sets and train our model after creating it
 147 —
CLASSIFICATION

In [4]: from sklearn import preprocessing

Input = preprocessing.scale(dt.drop(columns='Rainfall').values)
Output = dt['Rainfall']
inp_X,tst_X,out_y,tst_y = train_test_split(
Input,Output,test_size=0.2)
CModel = linear_model.LogisticRegression()
CModel.fit(inp_X,out_y)

0ut[4]: LogisticRegression()

Our model is trained. Now we can test out model's

predictions with actual values. To visualize it we
need to use the LabetEncoder and encode the
Rainfall labels into numeric values. We will also
print the accuracy of our model

In [5]: pred_y = CModel.predict(tst_X)

Enc = preprocessing.LabelEncoder().fit(['No Rain',
'Low','Moderate','High'])
cmp = pandas.DataFrame({'Predicted':Enc.transform(
pred_y),'Actual':Enc.transform(tst_y)})
acc = metrics.accuracy_score(tst_y,pred_y)
print('Accuracy:’,acc)
cmp.plot(kind='density*)

Accuracy: 0.435156900860872

Out[5]: <matplotlib.axes._subplots .AxesSubplot at 0x239fe078.

so our model didn't performed well. So here's a

question for you - why is the accuracy of our
model is below 50%? (without reading further)
 148 —
CLASSIFICATION

So what do you think? Is it because we scaled the

data? Yes, you are obviously wrong. The scaling
wasn't necessary but it was included to lure you
to think that it would've been the reason but
that's not it. Instead it is a good practice to do
so. The actual reason is the distribution in the
data. If you observe the plotted graph of the
predicted and actual values

you'll notice that the distribution is really

different. Our dataset has a spike i.e. less 'high
rainfall' data only concentrated at one place and
no or low rainfall a lot. We provided our model
with 18000 rows worth of data but the distribution
wasn't good i.e. we didn't get enough data for
moderate or high rainfall. The places where the
lines are together or overlapping are the
predictions made correct by our model i.e. mostly
no rainfall and low rainfall. Our model didn't
predicted moderate or high rainfall for any value
at all. Encountering errors like this helps us to
counter problems in actual situations. During this
time we need to come up with different methods to
improve our model (that will be covered in the
Performance and metrics lesson) or change the
algorithm, so let's look at another classification
algorithm
 CLASSIFICATION J

Naive Bayes
Naive Bayes algorithm is based on the Bayes
theorem which we already learned in previous
lessons. We have three types of Naive Bayes
algorithms:
• Gaussian, is used when the data in labels is
drawn from a gaussian distribution
• Multinomial, is used when the data in labels is
drawn from a multinomial distribution
• Bernoulli, is used when we have to predict
binary features like 0 or 1
So let's use the naive bayes algorithm to create
to model to predict whether it will rain or not
with the dataset used in the Binomial logistic
regression. We will use the Gaussian Naive Bayes
algorithm for this model

In [1]: import pandas

from sklearn import naive_bayes, metrics
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
dt = pandas.read_csv('Rainfall_data.csv')

This time we imported naive_bayes from sklearn

to create our model and LabetEncoder to encode
the labels while comparing the predictions. Next
we can split out data into training input, testing
input, training output and testing output
respectively using the train_test_sptit()
function and the test_size as 10%

In [2]: Input = dt.drop(columns=['Unnamed: 0','Rain'])

Output = dt[’Rain’]
inp_X,tst_X,out_y,tst_y = train_test_split(
Input,Output,test_size=0.1)

Now to create our model, we will use the

GaussianNB class from naive_bayes and train it
with the fit() method
 150 —
CLASSIFICATION

In [3]: Input = dt.drop(columns=['Unnamed: 0','Rain'])

Output = dt['Rain']
inp-Xjtst-X, out_y,,tst_y = train_test_split(
InputjOutput>test_size=0.1)
CModel = naive_bayes.GaussianNB()
CModel.fit(inp_X,out_y)

Out[3]: GaussianNB()

Our CModel is ready to predict. So letJs test

our model with the predict() method and compare
the answers using the density plot. We will also
print the accuracy score

In [4]: pred_y = CModel.predict(tst_X)

Enc = LabelEncoder().fit(['Yes','No'])
cmp = pandas.DataFrame({'Predicted':Enc.transform(
pred_y),'Actual':Enc.transform(tst_y.values)})
acc = metrics. accuracy_score(tst_y.» pred_y)
print('Accuracy:'>acc)
cmp.plot(kind='density')

Accuracy: 0.9275

0ut[4]: <matplotlib.axes._subplots.AxesSubplot at 0x26al30ee>

The model has an accuracy of 92% i.e. the

predicted line and the actual line in the graph is
almost overlapping each other. Our naive bayes
model has performed well than the logistic
regression model by 1%
 i o logical
IO PROGRAMMING
• Creoting Logics
• Collections
• Heirorchical
Logics

o
 X
16
_____ /
LOGICAL PROGRAMMING
__________________________________________________________ j

We have created different machine learning with

different algorithms. Now let's look at how
problems are solved with Logic programming i.e.
discussed in Al systems where problems can be
solved with facts (knowledge base) like 6 is an
even number and Rules like evry number divisible by
2 is even. We can create these type of facts and
rules for logic programming

Creating Logics
We will use kanren package for logic programming
so open your anaconda prompt and install pip with
conda install pip
□ Anaconda Prompt (anaconda!)

(base) C:\Users\Rahul>conda install pip

and then install kanren with pip using

pip install kanren

~ Anaconda Prompt (anaconda!)

(base) C:\Users\Rahul>pip install kanren^

With machine learning models like regressors we

predicted unknown values, the same is with logic
programming. First of all let's create logic
variables i.e. variables used in logic programming.
We can do so by creating a variable like x as we
usually create and use the var() function from
kanren

In [1]: from kanren import var

x = var()
 153 LOGICAL PROGRAMMING

We can also pass a token(label) to the logical

variable by passing a string in as argument ot the
var() function

In [2]: from kanren import var

x = var('log')
x

Out[2]: ~log

The ~ symbol specifies that "Log is a token of a

logical variable. Let's get a value for x that it
satisfies the rule that it equals 5 i.e. 5 itslef
We need import eq to create condition and run to
get our answer

In [3]: from kanren import var, eq, run

x = var()
run(l, # number of output
x, # Logical. variabLe
eq(x,5)) # ruLe i.e. x == 5

Out[3]: (5J

So we want a value that is equal to 5 i.e. 5

itself. First of all we call the run() function
and the first argument is the number of desired
outputs i.e. 1 here. Then we pass the logical
variable i.e. x here and followed by the logical
variable we specify the conditon that is equals
which is created using the eq () function in which
we pass the logical variable first and then the
value it is equal to i.e. also called the goal
constructor. The run() function evaluated the
condition and returns a tuple containing 5. Note
that the tuple has only one element thus it is
followed by a comma. This was a very simple
condition so let's create another condition like
we want a number that is floating point number.
For that we need to know how to create
 154 LOGICAL PROGRAMMING

collections. Collections can be created using the

memberoO function. We can simply get an element
from the collection

In [4]: from kanren import var, membero, run

x = var()
run(l, x, membero(x,('A','X','E')))

0ut[4]: ('A',)

Everything is same but now we changed the

condition that x needs to be member from (f A\
fXJ> fEJ) and we get fAJ
So back to our main problem we can use the
memberoO function to pass multiple values and
the goalifyO function to specify a particular
goal i.e. the type as float

In [5]: from kanren import var, goalify, run, membero

X = var('x')
goal = goalify(type)
run(l,
x,
membero(x, (0, 1, 2.4, 3)), # members
(goal, x, float)) # needs to be fLoat

Out[5]: (2.4,)

We used another logical variable i.e. goal which

we created using goalify() and method is type. So
our first condition is it needs to be an element
of CO/ 1, 2.4, 3) and our second condition is
the it needs to be float which is constructed by
goal or goalifyO indirectly
Note the memberoO is function and takes two
arguments i.e. logical variable and collection and
the float comparison is done in a seperate
parentheses (goal, x, float) not like goat(x,
float)
 155 LOGICAL PROGRAMMING

Heirarchical logics
We can also create facts with heirarchies or
parent-child relationship. For that we need to use
the Relation class and create fact using factsC)
function

In [1]: from kanren import var, Relation, facts, run

x = var()
parent = Relation()
facts(parent, # ReLating
('Milk', 'Curd'), # MiLk -> Curd
('Milk', ’Cheese'), # MiLk -> Cheese
('Wheat', 'Bread')) # Wheat -> Bread
run(l, x, parent(x, 'Cheese'))

Out[l]: ('Milk',)

We created a parent object of the Relation class

Then we created parent-child facts using the
factsO function like the (‘ Milk' Curd') where
Milk is parent of Curd. We can get an value for x
where it is parent of Chesse
We can also reverse the condition and ask for 2
child of Milk by using the x after Milk and 2 as
first argument

In [2]: from kanren import var, Relation, facts, run

x = var()
parent = Relation()
facts(parent, # ReLating
('Milk', 'Curd'), # MiLk -> Curd
('Milk', 'Cheese'), # MiLk -> Cheese
('Wheat', 'Bread')) # Wheat -> Bread
run(2, x, parent('Milk',x))

Out[2]: ('Curd', 'Cheese')

We get Curd and Cheese. Similary we can create

more parent-child realtionships. These methods can
be used to build facts and rules to get the
desired output
17 NATURAL LAN-
1/ GUAGE PROCESSING
• Tokenization
• Stemming
• Feature
extraction
• Rating Model

o J
 A

17
_____ /
natural language processing
1________________________________________________________ J

As the name suggests, NLP is used to find patters

and relationship in text data. In this lesson we
will learn about how to extract individual texts
from a data and perform different tasks on them

Tokenization
Tokenization is the simplest form of breaking down
text data into smaller parts like words,
punctuations or sentences

(The product was good, lightweight and value for money! j

value [money
was] [and

To do the same, we need to install the NLTK

package(Natural Language Tool Kit). Use the
following code to install it

□ Anaconda Prompt (an a con da 3] - con da install nltk

(base) C:\Users\Rahul>conda install nltk

Then open your jupyter notebook and import the

nttk package and run the download () function to
download necessary files automatically

In [*]: import nltk

nltk.download()

showing info https://raw.githubusercontent.com/nltk/

 NLTK Downloader

File View Sort Help

Collections Corpora Models All Packages

Identifier Name Size Status
all All packages n/a partial
all-corpora All the corpora n/a partial
all-nltk All packages available on nltk_data gh-pages bran n/a partial
book Everything used in the NLTK Book n/a partial
popular Popular packages n/a partial
tests Packages for running tests n/a not installed
third-party Third-party data packages n/a not installed

Refresh

Server Index:

After the download finishes we can move onto

tokenizing our text data. So let's tokenize our
data on the basis of words

In [3]: from nltk import word_tokenize

txt = 'The product was good, lightweight\
and value for money!'
nip = word_tokenize(txt)
nip

Out[3]: ['The',
'product',
'was',
'good',
■ ■
'lightweightand',
'value',
'for',
'money',
■!']

We used the worcLtokenize() function to tokenize

our text data. As you can see we have a list of
each word and punctuation marks as elements. We
can also tokenize it on the basis of punctuation
marks using wordpunct_tokenize() function
 In [8]: from nltk import wordpunct_tokenize
txt = 'The product was good, lightweight\
and value for money!'
nip = wordpunct_tokenize(txt)
nip

Out[8]: ['The',
'product',
'was',
'good',
I I
J J

'lightweightand',
'value',
'for',
'money',
■!']

We can also tokenzie on the basis of sentences

using the sent_tokenize0 function

Stemming
The text data can contain different forms of
verbs like go, gone or went. For a unified
approach we need to bring back those into base
forms i.e. called stemming

In [9]: from nltk import word_tokenize

from nltk.stem import PorterStemmer
txt,vsl = 'The product is fairly good but it has\
scratches', ' '
wrd = word_tokenize(txt)
stm = PorterStemmer()
for w in wrd:
vsl = stm.stem(w)
if w == vsl:
print(vsl)
elif w != vsl:
print(w,'>', vsl)

The > the

product
is
fairly > fairli
good
but
 ✓
160 NATURAL LANGUAGE PROCESSING
--------

it
has > ha
scratches > scratch

First we need to tokenize our words which we did

using the worcLtokenizer() function and then we
imported the PorterStemmer class and created the
stm object. Then using a for loop we iterated over
the tokenized words and stemming them using the
stem() method simultaneously. Look at how the
tokenized words are stemmed. As you can see except
scratch the Stemmer has changed things unnecessarily
So letJs use another stemmer i.e. SnowbatLStemirier
in which we need to specify the language while
object intialization i.e. english

In [10]: from nltk import word_tokenize

from nltk.stem import SnowballStemmer
txtjvsl = 'The product is fairly good but it has\
scratches'> ' '
wrd = word_tokenize(txt)
stm = SnowballStemmer('english')
for w in wrd:
vsl = stm.stem(w)
if w == vsl:
print(vsl)
elif w != vsl:
print(w,'>',vsl)

The > the

product
is
fairly > fair
good
but
it
has
scratches > scratch

This time we have accurate stemming. You can use

different stemmers for different situations. Note
the use of if statements to print the stems
 161 NATURAL LANGUAGE PROCESSING
i J-

We can also use a lemmatizer to the get the base

form of tokens. We can use the WordNetLemmatizer
for that purpose

In [11]: from nltk import word_tokenize

from nltk.stem import WordNetLemmatizer
txt,vsl = 'The product is fairly good but it has \
scratches',''
wrd = word_tokenize(txt)
stm = WordNetLemmatizer()
for w in wrd:
vsl = stm.lemmatize(w)
if w == vsl:
print(vsl)
elif w != vsl:
print(w,'>',vsl)

The
product
is
fairly
good
but
it
has > ha
scratches > scratch

As you can see the lemmatizer removed the fs? or

fes? from the words i.e. the difference between
lemmatizing and stemming
So we can use tokenization and stemming or
lemmatization to find pattern in text data

Feature extraction
In the 12th lesson we learned about label
encoding i.e. changing labels or string attributes
in a data into features or numeric notations.
Doing so helps the machine learning algorithms to
perform various activities with them. The same can
be said for the text data, even if we tokenize
text data it is still a raw text which cannot be
processed by the algorithms
 So we need to encode these tokens into numeric
values to be useful of the algorithms. We can do
so using the Bag of Word Model or feature
extraction i.e. a process which involves
extracting features(unique vocabulary) from the
text data and then it creates a model or matrix
with the weight of each feature in each sentence
in the text data. This matrix is the feature
vectors. So let's say we have the following text
Oranges are orange
But black current are purple
So let's extract the features first and for the
same we will use CountVectorizer from sklearn

In [1]: from sklearn.feature_extraction.text import CountVectorizer

txt = ['Oranges are orange',
'But black currants are purple']
enc = CountVectorizer()
enc_txt = enc.fit_transform(txt)
enc.get_feature_names()

Out[l]: ['are', 'black', 'but', 'currants', 'orange', 'oranges', 'purple']

First of all we created an enc object of the

CountVectorizer class and then we transformed
our text into the matrix. We can see the feature
names using the get_feature_names() method
Now we can view the matrix by converting it into
an array using the toarrayO function

In [2]: from sklearn.feature_extraction.text import \

CountVectorizer
txt = ['Oranges are orange'>
'But black currants are purple']
enc = CountVectorizer()
enc_txt = enc.fit_transform(txt )
enc_txt.toarray()

Out[2]: array([[l, 0, 0, 0, 1, 1, 0],

[1, 1, 1, 1, 0j 0, 1]], dtype=int64)
 7 V
163 natural language processing I
----- '• '
You can compare the matrix with the feature names
and the text data to understand the weight of each
word and the matrix itself

Text:
Oranges are orange
But black current are purple

Features:
are, black, but, currants, orange, oranges, purple

Matrix:

[[i, a, e, 0, i, i, e],
[1, 1, 1, 1, 0, 0, 1]]

Features:
[are,] black, but, currants, [orange,] [oranges,] purple

Sent|ence 1; :
[Oranges] [are] [orange]
I i~~~
Matrix: \

[ 0, 0j
[1, 1. b

As you can see we have 7 feature names in total

in the text data with 2 sentences. The matrix is a
2-D array with 1-D arrays equivalent ot the number
of sentences i.e. 2 here. Inside of the 1-D arrays
we have weight or feature vectors of each feature
with respective to the text data. The first array
which is representing the first sentence. We have
1 for present 1 time and 0 for not present
 164 natural language processing I
----- '• '
Now we can get the encoded features of the text
from the vocabulary- attribute of the
CountVectorizer class

In [3]: from sklearn.feature_extraction.text import \

CountVectorizer
txt = ['Oranges are orange'>
'But black currants are purple']
enc = CountVectorizer()
enc_txt = enc.fit_transform(txt)
enc.vocabulary_

Out[3]: {'oranges': 5,
'are': 0,
'orange': 4,
'but': 2,
'black': 1,
'currants': 3,
'purple': 6}

The order of the indicig is that of the order of

the feature names (fareJ is the first in feature
names) and the features is the text data(like
oranges is the first word)
Except for these there are some statistics like
Term Frequency(TF) i.e. the number of times a word
appear in a data and Inverse Document
Frequency(IDF) i.e. how much unique a word is to a
dataset
But a question may bug your mind that why we are
doing all these like we can just tokenize the text
data and use there index as features? But we all
know that a sentence is a combination of different
words that sounds meaningful with respect to the
grammar. If we just pass the indexes of each work
to an algorithm it will not work properly because
there is order or combination, we need the weight
of each word, features, TF and IDF to help the
algorithms to learn and analyze our text data as
it is
Rating Model
We have learned how to process text data to be
useful for algorithms. So let's create a model
that will rate a seller on the basis of reviews
First of all we need the sample dataset of
reviews, which you can download bit.ly/NLP rev

from here. Now we need to import http://bit.ly/NLP_rev

the necessary modules:

import pandas
from sktearn.feature_extraction.text
import CountVectorizer, Tfidf-
-T ransformer Scan Me!

In [1]: import pandas

from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction.text import \
TfidfTransformer., CountVectorizer
dt = pandas.read_csv('reviews.csv')

This time we also imported the TfidfTransformer

to transform our encoded values with TF i.e. Term
Frequency and IDF i.e. Inverse Document Frequency.
Let's preview our dataset with the head()
function

In [2]: dt.head()

Out[2]:
Reviews Rating

0 The product is fairly good but it has scratche... Average

1 Not same as told in the description Poor

2 It is worth the money! Good

3 Delivered product is not genuine Poor

4 I'm not satisifed with the build quality Poor

 •/
166 NATURAL LANGUAGE PROCESSING I

Now we can move onto the next step i.e. data

processing

In [3]: # Dividing input and output

rev = dt['Reviews']
y = dt['Rating'].tolist() # Output
# Feature extraction
enc = CountVectorizer()
enc.fit(rev)
enc_rev = enc.transform(rev)
# TF & IDF
tf = TfidfTransformer()
tf.fit(enc_rev)
X = tf.transform(enc_rev) # Input

First of all we divided the data into input i.e.

the reviews and output i.e. the ratings. Next we
need to extract features using the
CountVectorizer and pass the rev reviews to
fit 0 and then to transform^ function and store
it in enc_rev. Then we can extract the TF and IDF
using the TfidfTransformer. This time pass the
enc_rev to fit() and the to transformO function
and finally store it as input. Now we can move
onto creating our classification model

In [4]: CModel = DecisionTreeClassifier ()

CModel.fit(X,y)

0ut[4]: DecisionTreeClassifier()
____________________________________________

So our model is ready to rate comments! Now we

can create a function in which we can pass reviews
as strings and get their ratings. But we need to
make sure that we extract the features from the
comments(strings) using the same Countvectorizer
and TfidfTransformer used while processing the
data i.e enc and tf here. So let's create the
function
 ✓
167 NATURAL LANGUAGE PROCESSING I
-

In [5]: def rate(*comment):

enc_com = enc.transform(comment)
com = tf.transform(enc_com)
pred = CModel.predict(com)
for ele,ratin in zip(comment,pred):
print(ele?'\nRating:',ratin)

We defined the rate function in which we will

recieve a list of comments(*args) and then
transformed it using enc and tf. Note that we used
transformO instead of fit_transforni(). Using
for loop we printed the comment and itJs rating
So letJs call our rate() function and pass
comments like:

In [6]: rate('Not worth the money',

'Somewhat satisfying')

Not worth the money

Rating: Poor
Somewhat satisfying
Rating: Good

As you can see we have created a NLP model that

can rate reviews by analysing texts similarly, we
use these techniques to create more NLP models
IQ SPEECH
IO RECOGNITION

• Reoding on
audio file
• Extracting
features
• Speech
recognizer

o
 A

18
k ______ /
SPEECH RECOGNITION
T )

Speech recognition is one of the most important

aspects in Al. It allows humans to interact with
machines by speaking. But speech recognition is not
easy because of a lot of factors Hike the
vocabulary of the model, audio quality
i.e. use of which microphone to
capture audio or noises in the
audio, speech characterestics
like speaker style, accent, voice
features or subject and object of
the speech. By taking all these
into consideration a speech
recognition model is created

Reading an audio file

The first step is to import the
audio file. You can download the http://bit.ly/aud_data
sample audio file from here -
We will import the audio or .wav
file using the scipy package's
wavfite.read 0 function, so
import the wavfile module from
scpipy.io and import the audion Scan Me!
file

In [1]: from scipy.io import wavfile

freq, sig = wavfile.read('aud_data.wav')

The readO returns two elements i.e. frequencies

and the audio signals that's why we stored them in
freq and sig respectively. While recording audio
the file is stored in digitized form but like text
we need these in discrete numeric values which is
called sampling. The two returned elements by the
readO function are results of sampling
 z—•
170 SPEECH RECOGNITION

We can also visualize our audio file using

matpLottib

In [2]: from matplotlib import pyplot

pyplot.plot(sig)

0ut[2]: [<matplotlib.lines.Line2D at 0xl68f7677610>>

<matplotlib.lines.Line2D at 0xl68f7677640>]

As you can see we simply plotted the signals data

suing the ptotO function. This shows the
amplitude (dB) of the wav file, though I already
performed functions like normalization, noise
reduction, etc. on the audio we don't need to do
any

Extracting features
After importing the audio we need extract
features from the file. There are a lot of ways to
extract features from audio files like MFCC i.e.
Mel-Frequency Cepstral Coefficients. Simply you
need to know that MFCC is the Coefficients that
make up the Mel-frequency cepstral where
Mel-frequency is a scaling process which scales
the audio at human audible range and cepstrum is
the information of rate of change in spectral bands
 171 SPEECH RECOGNITION

You may not be familiar with all these terms and

it's not necessary too, you just need to
understand what's going behind the curtains and
you can always skip ahead and use the models but I
would like you to atleast give it a read or two
So install python_speech_features module using
pip instaii python_speech_features
in your anaconda prompt and import mfcc()
function from python_speech_features

In [8]: from python_speech_features import mfcc

from matplotlib import pyplot
coe = mfcc(sig[:2500]jfreq)
pyplot.matshow(coe)

0ut[8]: <matplotlib.image.Axeslmage at 0xlc81el6bl00>

0 2 4 6 8 10 12

Using the mfcc we can analyze the audio and

perform a lot of tasks on it. You may still not
get what is mfcc but you should know that these
coefficients are used to classify, predict or
recommend the audio file. Now let's predict what
is said in the audio file
 172 SPEECH RECOGNITION
- n

Speech Recognizer
Previously we imported audio files, visualized it
and also extracted it's feature but even though if
we want to create a speech recognizer from scratch
it will take a lot of time that's why we will use
the Speech Recognition package
Install it using the below code bit.ly/Recog aud
K
in your anaconda prompt : http://bit.ly/Recog_aud

pip instaUL SpeechRecognition

You can download the audio file

for speech recognition from here
i.e. a .flac file - Scan Me!
Now we can import the
Recognizer and AudioFite class
speech_recognition to create a recongnizer and
read the .flac file

In [2]: from speech_recognition import Recognizer, AudioFile

SRec = Recognizer()

with AudioFile('aud_recog.flac') as flac:

aud = SRec.record(flac)

SRec.recognize_google(aud)

Out[2]: 'what do you call someone who sells themselves in

exchange for Spaghetti'

As you can see we created a SRec object of the

Recognizer class. Then we opened the .flac file
using with(which will close the file after the
block of code is executed) and read it with the
recordO method. Our recognizer has read the
audio so we can ask it to recognize the audio
using the recognize_googie() method i.e. Speech
recognizer by google
 z—•
173 SPEECH RECOGNITION

I would like you to go ahead and listen to the

audio file and check it with the recognized text
You can also use a microphone for the input and
recognize that using Microphone class. But you
need to use the adjust_for_ambient_noise0
function to tell the recognizer that your audio
has background noise and adjust accordingly. And
instead of recordO use ListenO method. If you
have a mic then give it a go!

In [3]: from speech_recognition import Recognizer, Microphone

SRec = Recognizer()

with Microphone() as vois:

SRec.adjust_for_ambient_noise(vois)
aud = SRec.listen(vois)
1 Q ARTIFICIAL INTELLI-
io GENCE ALGORITHMS
• Sotisfying
constraints <
• Finding unknown I
values
• Unique
combinations

o
r A

Al ALGORITHMS
k ____________________________)
As mentioned earlier we can use Al algorithms for
solving many problems i.e. called heuristic search.
Like we have some conditions needed to be met and
then find the answer i.e. Constraint Satisfaction
Problems. We can use search algorithms to solve
problems with constraints which have a lot of
possible anwers and finding them manually will take
a lot of time. The search algorithm will search
through every possible answer and return the best
answer

Satisfying constraints
Let's say we have a problem like -

2x + 3 = y

where we want to solve the equation for x and y. To

do so we need to install the python-constraint
package with the following code in the anacond
prompt:

□ Anaconda Prompt (anaconda^)

(base) C:\Users\Rahul>pip install python-constraint.

Then we can import the constraint module and

create a object of Problem class, which will be
used to create and solve the problem

In [1]: import constraint

prb = constraint.Problem()
 z—• /
176 Al ALGORITHMS

If you remember, we created logical variables

while logical programming similarly we will create
variables using the addVariabte() method in which
we need to pass the variable name i.e. x and y
here and a range of their value using the range ()
function which can be assumed to be 10 here

In [2]: import constraint

prb = constraint.Problem()

prb.addVariable('x',range(10))
prb.addVariable('y'>range(10))

We created the x and y variables. Now we can

create constraint using the addConstraint () • We
will pass an anonymous function i.e. a lambda
function
In [3]: import constraint
prb = constraint.Problem()
prb.addVariable('x',range(10))
prb.addVariable('y',range(10))
# ProbLem : 2x + 3 = 9
prb.addConstraint(lambda x, y: 2*x + 3 == 9)

Using the lambda function we recieve x and y as

arguments and return the expression 2*x + 3 == y.
Note the use of == instead if = because we are
comparing not assigning
So now we can get the answers of our problem
using the getSotutions() method which will
return a list of dictionaries of possible answers
in the range of the variable

In [4]: for sol in prb.getSolutions():

print(sol)

{'x' ■y.; 9}
3,
{'x* ■y.; 7}
2,
{’X* ■y.; 5}
1,
{•x‘ 8, ■y': 3}
 177 Al ALGORITHMS
•

Note the use of for loop to iterate over the

solutions and print them. So with respect to the
equation or constraint i.e. 2x + 3 = y; value of x
can be 3, 2, 1, 0 and y can be 9? 7, 5, 3
respectively. Let's test the answers using the
first pair of solutions i.e. 3 & 9 for the
equation:

2x + 3 = y
2(3) + 3 = (9)
6 + 3 = 9
9 = 9

You can see the solutions satisfy our equation

Finding unknown values

We have learn't how the search algorithm works so
let's solve the following problem :

Problem: Jake bought 5 cakes and 7 chocolates for

88 and Mike bought 2 cakes and 6 chocolates for 48.
So, what is the cost of each cake and chocolate?

So we need to find the cost of an individual cake

and chocolate if we have the following system of
equations:
(Jake) 5x + 7y = 88
(Mike) 2x + 6y = 48

where x represents the cost of an individual cake

and y represents cost of an individual chocolate.
We obtained the above set of equations from
problem in hand i.e. lake bought 5 cakes(x) and(+)
7 chocolate(y) for(=) 88 and Mike bought 2
cakes(x) and (+) 6 chocolates(y) for(=) 48. Using
these two equations or constraints we can
pin-point the values
 Al ALGORITHMS

So let's import our constraint package and

create an object of the Problem class
Problem: Jake bought 5 cakes and 7 chocolates for
88 and Mike bought 2 cakes and 6 chocolates for 48.
So, what is the cost of each cake and chocolate?

In [1]: from constraint import Problem

cost = Problem()

Now let's add two variables to the problem

representing the cost of each cake and chocolate.
And let's set the range to 20 for now

In [2]: from constraint import Problem

cost = Problem()
cost.addVariable('cake1,range(20))
cost.addVariable('choco',range(20))

After creating the variables we can add the

constraint to the problem and we will use the
first equation(lake's)

In [3]: from constraint import Problem

cost = Problem()
cost.addVariable('cake',range(20))
cost.addVariable('choco',range(20))
cost.addConstraint(lambda x,y:5*x+7*y==88)

Let's check on the possible cost of a cake and

chocolate in the range of 20 for the first
equation

In [4]: from constraint import Problem

cost = Problem()
cost.addVariable('cake'>range(20))
cost.addVariable('choco',range(20))
cost.addConstraint(lambda x,y:5*x+7*y==88)
cost.getSolutions()

0ut[4]: [{'cake': 12, 'choco': 4}, {'cake': 5, 'choco': 9}]

 z—•
179 Al ALGORITHMS
• -J

So Dake would have bought eack cake for 12 or 5

and each chocolate for 4 or 9. We have narrowed
down our answers to two possible solutions then
let's use these values in the second equation
(Mike's) and the pair of solution which will
satisfy the equation will be the correct answer

In [5]: sol = cost.getSolutions()

eq2 = 0
for i in range(len(sol)):
eq2 = 2*sol[i][’cake’] + 6*sol[i][1choco’]
if eq2 == 48:
print(sol[i])

{'cake': 12, 'choco': 4}

Moving onto a new cell we stored the solutions

for Dake's equation in sol and create an variable
eq2 to store the answer of Mike's equation. Using
a for loop we passed the cake's and choco's values
to the equation 2*cake(i.e. sot[i] ['cake']) +
6*choco(i.e. sot[i]['choco']) and compare
whether their total is 48 and the first pair has
satisfied the equation. Note the slicing of the
solutions of the first equation like
sot[i] ['cake'] where i is the index of pairs of
solutions(dictionaries) and the second slice is
the key i.e. cake whose value we need
So the price of each cake is 12 and each
chocolate is 4. Alternatively we could have used
two different Problem objects for both the
equations i.e. Dake's and Mike's. Then we could
have compared the same solution pair in their
solutions and print it
The following code implements the above
alternative method to find the price for each cake
and chocolate where we find pssible solution pairs
for each of the equation and then print the same
pair using if statements
 z—• /
180 Al ALGORITHMS

In [2]: from constraint import Problem

# SoLutions for Jake's
cost = Problem()
cost.addVariable('cake',range(20))
cost.addVariable('choco',range(20))
cost.addConstraint(lambda x,y:5*x+7*y==88)
soil = cost.getSolutions()
# SoLutions for Mike's
cost2 = Problem()
cost2.addVariable('cake',range(20))
cost2.addVariable('choco',range(20))
cost2.addConstraint(lambda x,y:2*x+6*y==48)
sol2 = cost2.getSolutions()
# Finding the same pair
for sol in soil:
if sol in sol2:
print(sol)

{'cake’: 12, 'choco*: 4}

Unique combinations
Previously we learned how to use the search
algorithm to find out unknown values. But let's
say we want a unique combinations of day and night
duty of lake, Mike and John where they share equal
work and holidays in a week (except Sunday). We
can use the AULDifferentConstraint0 and get the
unique combinations

In [1]: from constraint import *

matrix = Problem()
matrix.addVariables(['day','night'],
['Jake','Mike','John'])
matrix. addConst raint(AHDiff erentConst raint())
matrix.getSolutions()

[{'day': 'John', 'night': 'Mike'},

{'day': 'John', 'night': 'Jake'},
{'day': 'Mike', 'night': 'John'},
{'day': 'Mike', 'night': 'Jake'},
{'day': 'Jake', 'night': 'Mike'},
{'day': 'Jake', 'night': 'John'}]
 z—• /
181 Al ALGORITHMS

As you see we get a unique combination of day and

night shifts of lake, Mike and lohn. Let's create
a dice matrix. Do you know what is dice matrix?
A dice matrix is 2-D array of 3 1-D
arrays of unique 2 values whose sum
is exactly 7. To better understand
it grab a dice and find the sum of
two opposite sides. You'll find that
every opposite sums up to 7 like 1 is
opposite to 6. Let's create this
dice matrix from scratch using basic
loops and decision control

In [1]: import random

num = [1,2,3,4,5,6]
matrix = []
pair = []
# Appending unique combinations
while True:
# Getting random pair of 2 vaLues
pair = random.choices(num,k=2)
# Checking if sum is 7
if sum(pair) == 7:
# Checking if pair is new or not
if pair not in matrix:
matrix.append(pair) # adding pair
else:
continue

# If 3 unique combinations are present

# then terminate the Loop
if len(matrix) == 3:
break

Without freaking out I would like you to observe

the code along with comments and then understand
what's going on
First-off we imported the random module. Then we
created a list of numbers in a die. Then we
created empty lists to store value in them in the
loop
Then we created a white loop and passed True as
 z—•
182 Al ALGORITHMS
-4
•

the condition, you may think that it will create

an indefinite loop! But we will take care of it by
terminating the loop when our conditions will be
met. In the loop we start by creating a list of 2
elements picking from the num list using the
choicest) function. Note that we passed 2 to the
k parameter that means it will return a list with
2 randomly chose elements
Them we check whether the selected pair sums upto
7 or not using if statement. And again check that
if the pair is unique to the matrix and then
append the pair in the matrix if yes or we
continue to the next iteration
After appending pairs to the matrix we check that
whether the matrix has reached 3 pairs or not and
if yes then terminate the loop using the break
keyword
So after executing the code print the matrix and
look at the pairs

In [2]: matrix

0ut[2]: [[3, 4], [5, 2], [6, 1]]

Similarly I would like you to create a matrix of

pairs that sum upto 9 with numbers from 1 to 8
 artificial neural
CM NETWORKS
• Creating neural
networks
artificially
• Layers of
networks

o
 A

20
______ /
ARTIFICIAL NEURAL NETWORKS
T_______________________________________________________________________________ J
Have you ever wondered how do you remember
everything or reading this book now? It's pretty
obvious that our brain is behind everything but
what's inside it that make it happen? Speaking
biologically, our brain consists of
trillions of neurons or nerve cells
that create a neural network.
These neurons work together to
help us think, decide, etc. But
we are not going that deep here,
instead what if we have the
machines have the same ability as
our neural networks to learn, think,
decide and perform tasks like us. Well
that's known as Artificial Neural Networks or ANN.
We aren't going to learn how they work or things
behind the scene because it needs another dedicated
book about it but it falls under Al which we are
learning in this book so let's use it

Creating neural networks

artificially
To create neural networks artificially we will use
the NeuroLab package. So let's install using
anaconda prompt

~ Anaconda Prompt (anacondal)

(base) C:\Users\Rahul>pip install NeuroLab

Let's create a simple perceptron(basic unit of an

ANN) classifier with one neuron or node where we
will provide two input values like 0,0 and classify
it into the maximum value i.e. 0 as output
 •/
185 ARTIFICIAL NEURAL NETWORKS
----- '• '
Start-off by importing the newpO function from
the neurotab package which we will use to create
the classifier. Also create the input and output
values

In [1]: from neurolab.net import newp

X = [[0, 0], [0, 1], [1, 0]]
y = [[0], [1], [1]]

You can see that we pass two numbers as input and

the output is the maximum number. Now let's create
our classifier using the newpO function

In [2]: from neurolab.net import newp

X = [[0, 0]j [0, 1], [1, 0]]
y = [[0], [1], [i]]
ann = newp([[0, 1],[0, 1]], # 2 inputs
1) # 1 node or neuron

We added a new perceptron by specifying the

number of input i.e. an array with two 1-D arrays
and they contain the range of the input
(min, max) and the number of nodes i.e. 1. So
let's train our classifier

In [3]: from neurolab.net import newp

x = [[0, 0], [0, 1], [1, 0]]
y = [[e]. [i], [i]]
ann = newp([[0, 1],[0, 1]], 1)
ann.train(X, y, show = 1)

Epoch: i; Error: 1.0;

Epoch: 2; Error: 0.5;
Epoch: 3; Error: 0.5;
Epoch: 4; Error: 0.0;
The goal of learning is reached

Out[3]: [1.0, 0.5, 0.5, 0.0]

We trained our classifier using the tnainO

method. Passed the input and output as usual but
 • s
186--- ARTIFICIAL NEURAL NETWORKS
--- '•
you may see that we also passed 1 to the show
parameter and got something printed like.,
Epoch: 1; Error: 1.0;

So what is it? Here, we are watching the way our

classifier is learning. Epoch refers to the weight
of the varaible(input) to change the node (logic)
and Error is the error rate from actual value. As
you can see when epoch was 1 i.e. the node can be
changed only once the error rate was 1.0. Then the
epoch is raised to 2 i.e. the node can be changed
twice and the error rate is lowered to 0.5 and
agian the epoch is raised to 3 but the the error
rate remains 0.5. Finally when the epoch reaches 4
the error reduces to 0.0 i.e. The goat of
learning is reached
We also get an array printed
[1.0, 0.5, 0.5, 0.0]
presenting the error values. We passed 1 to the
show parameter means that the epoch value will
start from 1 and print the results of each
iteration like,
Epoch: 1; Error: 1.0;

So let's test the classifier by passing an input

it hasn't trained with like [1, 0] using the
sim() method
I In [4]: ann.sim([[l,0]])

0ut[4]: array([[1.]])

This was the simplest example of working of an

ANN. Here the nodes form logics which depends on
the epoch i.e. how much the variables will effect
the decision control. Think that you want to buy
something by watching it's advertisement (epoch 1)
but asked someone(epoch 2) who has it but says not
to and you finally decide not to.
 z
187 ARTIFICIAL NEURAL NETWORKS
-----

Layers of networks
Previously we created a network with one node or
neuron with single layer i.e. iy will evaluate
both the input and output but let's say we want to
add more neurons or layers. We can achieve so
using the newffC) function so let's import it and
create our input and output data
I
In [1]: from neurolab.net import newff
X = [[e,o],[i,0],[i,i]]
y = [[©],[i],[2]]

This time we passed the same input but changed

the output as the sum of the inputs. Though
according to the number of data our network will
not work well which is fine because it is for
learning purpose only. So then create the network
like earlier but use the newffQ function instead

In [2]: ann = newff([[0> 1], [0, 2]],

[4, 1])
err = ann.train(X>y->epochs=20)

The maximum number of train epochs is reached

We passed the shape of the input and output as

first argument i.e. an 2-D array like,
[[0, 1], [0, 2]]
where the first 1-D array is the range of input
[0, 1]
and the second 1-D array is the range of output
[0, 2]
and as second argument we passed the shape of our
nodes or neurons i.e
K 1]
where we specified 4 nodes for input and 1 for
output. In the train() function we also specified
the maximum number of epochs in the parameter
 188 ARTIFICIAL NEURAL NETWORKS
I X

Let's visualize the error rate effected by the

number of epochs

In [3]: from matplotlib import pyplot

pyplot.plot(err)
pyplot.xlabel(1 Epochs')
pyplot.ylabel('Error')

0ut[3]: Text(0, 0.5, 'Error’)

As you can see the error rate dropped from 1 to

0.5 by 10 number of epochs and stayed the same
i.e. our current error rate is around 0.5. So
let's simulate the network and pass [1,0] as input

In [4]: ann.sim([[0, 1]])

Out[4]: array([[0.99982076]])

As to my surprise we get an answer very close to

what we want but it may be fluke. Because the
error rate is 0.5 and we don't what the network is
deciding on the basis of. You can say that it is
really easy to just pass the sum of the input but
why it's making errors?
 •z
189 ARTIFICIAL NEURAL NETWORKS
--- '•--------------------
Well we are passing the network some numbers
[[0, 0], [1, 0],[1, 1]
and telling that these inputs has the following
output
[[0], [1], [2]]
without specifying any intentions behind it. It is
like telling a child that there is this number and
this number and anyhow we get that number. Will it
be possible to tell the result of another numbers?
BUt itJs the same we are doing. These artificial
neural networks need more raw data to find more
accurate intention behind the output and as
mentioned earlier ANN is a topic that needs an
seperate book dedicated to it but because it falls
under Al we are discussing about it. Anyhow so you
should get a thorough idea of ANN
 DEEP
Cl LEARNING
• Deep learning
vs machine
learning
• Deep learning
models
• Regressoion using
deep learning
o---------------
 A

21
___________ /
DEEP LEARNING
__________________________________________ J

We already know what is deep learning and how it

works or if you forgot you should go back to the
Types of Al lesson to go through it again

Deep learning vs Machine

Learning
Deep learning and machine learning are a lot alike
or more specifically Deep learning is a powerful
machine learning technique. But
even though they are alike, but
their results are the major
differences between them. Let's
look at the differences between
Deep Learning and Machine
learning:
• Machine learning models can
perform with very less data
but Deep learning models need
comparatively high data than Machine learning to
produce more precise and accurate results
• Machine learning models can work low-profile
systems but Deep learning models require modern
and high-end systems to work
• In machine learning all the inputs are taken into
account to determine the output but deep
learning models extracts the best variables
(features) with high information gain
Most of Machine learning
algorithms are lazy learners
i.e. they take more time during
predicting but deep learning
models takes less time but in
turn takes more time in
training than machine learning
 192 DEEP LEARNING

Deep learning models

Previously we have created machine learning
models with various algorithms but how do we
create Deep Learning Models? As mentioned earlier
deep learning models scale as the data increases
and requires more computational speed. Taking that
into account these models can be created using
Convolutional Neural Networks or CNN that sounds
similar to the ANN we leared earlier but is a more
advanced. Convolutional Neural Networks are also
made up of layers and neurons like ANN but they do
not input data like ANN who converts the input
into 1-D array and feed to the network. Though it
is well enough for numeric data but if the input
is in the form of images it becomes difficult for
the network. But CNN takes that difficulties into
account where they neurons are connected to a
small patch of output of the previous layer unlike
the ANN

Regression with deep learning

We used macine learning algorithms like
LinearRegression to create regressors so let's
use tensforftow and keras i.e. high-level neural
network API to build a regressor(using deep
learning). So let's install keras and tensorftow
using anaconda prompt

~ Anaconda Prompt (anaconda!)

(base) C:\Users\Rahul>pip install keras_

Anaconda Prompt (anaconda!)

(base) C:\Users\Rahul>pip install tensorflow,

 Z—•

193 DEEP LEARNING

•

For this regressor we will use the height and

weight dataset used while creating the linear
regressor(Page no. 120)
So let's import the modules and functions we need
to build the regressor:

from pandas import read_csv

from tensorfiow.keras.models import \
Sequential
from tensorflow.keras.layers import Dense

We will use the read_csv() function to import

our csv data. We will use Sequential class to
create the deep learning model and Dense to add
layers in the model

In [1]: from pandas import read_csv

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

dt = read_csv('height_and_weight.csv')

We also imported our dataset. So let's preview at

the values in the dataset which we are going to
use

In [2]: dt.values[:5]

array([[ 1. , 65.78, 112.99],

[ 2. , 71.52, 136.49],
[ 3. , 69.4 , 153.03],
[ 4. , 68.22, 142.34],
[ 5. , 67.79, 144.3 ]])

We pass our model these values where the first

column is the index which should be ignored, the
second column is the height data and the third
column is the weight data. So let's split the data
into input and output
 Z—•
194 DEEP LEARNING
•

In [3]: dt = dt.values
X = dt[:, 1]
y = dt[:, 2]

Each value in all the 1-D arrays with index 1 are

height data and each value in all the 1-D arrays
with index 2 are weight data. So let's create a
deep learning model of the Sequential class

In [4]: RegModel = Sequential()

Now we can move onto the next step i.e. adding

the layers using the add () method and Dense

In [5]: # Input Layer

RegModel.add(Dense(4, input_shape=(200,)}
activation='relu'))
# Layer
Re^fetfeP.add(Dense(2, activation='linear’))

The first layer should be the input layer where

the first argument is the number of outputs
produced by that layer(nodes or neurons), then the
input_shape parameter where we need to mention
the shape of the input data i.e. (200,1) here
(You can check the same using X.shape command)
and atlas we specify the activation method i.e.
retu(rectified linear activation function).The
second layer i.e. a hidden layer is added too
where the output produced by that layer is 2 and
the activation function is linear. This is the
layer where decision takes place but the actions
are hidden. We can check the number of layers i.e.
2 using the layers attribute

In [6]: # Number of Layers in the deep Learning modeL

len(RegModel.layers)

0ut[6]: 2
 Z—•

195 DEEP LEARNING

•

Now we can compile our model using the compile ()

method where we need to specify the loss function.
The loss function is the type of metrics like mean
absolute error(Average of Absolute difference
between predicted value and actual value), mean
squared error(Average of Squared difference
between predicted value and actual value), etc.
which will be take into to account to monitor the
performance of the model. For this regressor we
will use the MAE (mean_absoiute_error) and pass
it to the toss parameter. Next comes the
optimizer i.e. an algorithm or method which helps
to reduce the losses by changing the weights of
different attributes and features in the neural
network. We will use the adam algorithm which can
be specified in the optimizer parameter. And we
will also specify MAE in the metrics parameter to
print them in the terminal while training

In [7]: # CompiLing the modeL

RegModel.compile(loss='mean_absolute_error',
optimizer='adam',
metrics=['mean_absolute_error'])

Next we can train our model using the fit()

method and pass the input i.e. X and the output
i.e. y as arguments. Along with the data we need
to specify some more parameter, while creating a
neural network we came upon a term epoch i.e. the
weight of each feature to change the network or
decision control and we need to specify the same
here too(because the deep learning model is
created using neural networks). So we will specify
10 in the epochs parameter. We can see the
training progress bar using the verbose parameter
where 0 means nothing, 1 means animated progress
bar (===) and 2 means only number of epochs (1/10)
and the vatidation_spiit parameter which will
 DEEP LEARNING

split and shuffle the datasets at each epoch

iteration. The batch_size is the number of data
points or samples given at a time. The more the
batch_size the more memory is required so we will
use 1 as the batch size

In [8]: # Training the modeL

RegModel.fit(X, y,
epochs=10.» batch_size=l>
verbose=l, validation_split=0.1)

Epoch 1/10
180/180 [------------------------------------------------------- ] -
- val_mean_absolute_error: 139.4882
Epoch 2/10
180/180 [------------------------------------------------------- ] -
- val_mean_absolute_error: 121.8028
Epoch 3/10
180/180 [==============================] -
val_mean_absolute_error: 90.9186
Epoch 4/10
180/180 [------------------------------------------------------- ] -
al_mean_absolute_error: 64.9408
Epoch 5/10
180/180 [-=============================] -
al_mean_absolute_error: 51.0282
Epoch 6/10
180/180 [=————-------------- =-=----------- ] -
al_mean_absolute_error: 33.7814
Epoch 7/10
180/180 [-====-============-=-===-====-] -
al_mean_absolute_error: 12.2725
Epoch 8/10
180/180 [---------- =--------- =------------------------------- ] -
mean_absolute_error: 8.9072
Epoch 9/10
180/180 [------------------------------------------------------- ] -
mean_absolute_error: 8.8819
Epoch 10/10
180/180 [---------- =--------- =------------------------------- ] -
mean_absolute_error: 8.8243

0ut[8]: tensorflow.python.keras.callbacks.History

Some portion of the output is clipped so please

refer to actual by running the code
 Z—•
197 DEEP LEARNING
•

We get the progress and MAE for out model

printed. You can see that the error rate falls
from 139 to 8 during 10 iterations, which can be
visualised as the below graph:

6 1D
No of Epochs

If you want to extract the loss values (like MAE

here) you can create a LossHistory class (child
of Callback class) and get the loss values by
passing an object of the class to the callback
parameter while training the model

In [ ]: from keras.callbacks import Callback

# Getting Losses
class LossHistory(Callback):
def on_train_begin(self, logs={}):
self.losses = []

def on_batch_end(self, batch, logs={}):

self.losses.append(logs.get('loss'))

# Training
MAE = LossHistory()
 Z—•

198 DEEP LEARNING

•

RegModel.fit(X, y,
epochs=10, batch_size=l,
verbose=l, validation_split=0.1,
callbacks=[MAE])

# Loss vaLues
MAE.losses

You can also view a summary report of the model

using the summaryO function

RegModel.summary()

Model:: "sequential"

Layer (type) Output Shape Param #

dense (Dense) (None, 4) 8

dense__1 (Dense) (None, 2) 10

Total params: 18
Trainable params: 18
Non-trainable params: 0

One a side note, you may get different or not

same at all results using the same datasets and
parameters because the model is based upon neural
networks whose reasoning or decision control is
un-accountable to us. If you get errors while
importing the dependencies, make sure to install
the pachages through anconda prompt and then
upgrade all the packages like numpy, scipy,
scikit-learn, etc using the following code -
pip install —upgrade <package>
like:
~ Anaconda Prompt (anacondal)

(base) C:\Users\Rahul>pip install --upgrade scipy

22 COMPUTER
VISION
• Processing
images
• Tweaking
images
• Face
recognition

o
 Till now we have learned how to replicate human
thinking with various methods but let's look at
the another end of the story i.e. visuals.
Training machines to view the world as we see i.e.
in 3-D and interpret on that information is called
computer vision

Processing images
You may think how can machines
view the world as we see? How is
that is even possible? So let's
move a bit back to the NLP lesson
where we learned to teach models
to work with human language. It
seemed a question to that the
computers which only works with
binary digits (0 and 1) can
understand human language and now that machine
has vision too. Based on the NLP lesson, what do
you think enables machines to have vision?
First off, how do you think you are able to see
images in your devices. Image files are stored as
.png, .jpg, .svg, etc. formats but how do they
store an image digitally? I want you to naviage to
any image on your desktop

LJ • This PC ♦ Dataset (X:) • Image v &

monitor.png

* Dataset (X:)

monitor.pnq date taken 17.7 KB

 (—•f
201 COMPUTER VISION
-----

Then right-click on the image and choose Open

With > Choose another app
How do you want to open this file?

Keep using this app

Photos
Photos are easily enhanced, organized for you, and
ready to edit and share,

Other options

Adobe Illustrator 2020

Adobe Photoshop 2020

Notepad

Paint
v
R'Mrt* on

3 Always use this app to open .png files

Choose the Notepad and hit OK

JU monitor.png - Notepad • #

File Edit Format View Help

^PNG

IHDR OD oxOu DsBITmElDd* pHYs

x
xOB(>x OtEXtSoftware www.inkscape.org>i<Q FIDATxUiYyPVyziue2}»«#&::&::b^»izs&
V 2Ea0"*"0(0*l"<i«aA.DE^Er™B9c§l 2dD-c9IAd®6>0’ei,Sar6u|i9<giu@D~ I.JD

0 @0 D e5•yYVkc?eYeZ“Yd’"}§-rB6YPL 0, >C9eYMO03©{:u 'vx%3en3,EhOyotCy3 !0@VKC<in

,ZeOkiz16miEAASXayx-0€Qu0«]6YEez<rup(HI]€iU{A'wA[]-S3R-i-trn2od£E0€-.U] 1YO/HD ”•0' /eOw'F*
LI|{saeE%/ Y-Ow¥yNON| x.0D’0)8EoAd7.}#KR 2idAaeu> uI5&ya O?>zi'OOUh§UUO;’< uAo0K
tD€-[Y$oX$D(]€wY[I],,„aDnO>=’kDE%Yp6e IuD'6M“Lhfe'~2V£KDEjiuyd=c-e-> ~EaDKya,w FUU 2Z:h&-
@Vsu, [ur,[lAt ‘yfl6pa-,,Ezzuu<£DA«0y-o?U daaOTWuDxO4>yzQ0 »-'-“D aeO<EXyp}p#(]={]* j£Y%XfDNn <
UMyBdOu'D,rci&'6 I-dUO On§auEeYD€voy-io(] dSE«(D.£EXfg+ WioYaAK1 «(E0uOn|D'iAEiKUPjOoeU
s«0cd4»°$3eO3i
@NxaQNn ~odaOc%#AR ODy®02u a?R:2IxA0O> 0}»{?'7 I-rED0sjn{euD: '7 Iluo~ED0P< vyO€yO<Q:
~Klwy@0 6€>gb( '>J( '27™uD
~U"-p 0 Lnd„E0 €e4LX”P 32)Ea?P A|^CA0
€eoyY”0@(] S)$0@0 3isdd( '*0CaJJ( '6pI™uD
~odA
 z
202 COMPUTER VISION
--- '•
You can see some weird text written all over
which you don't understand at all. That's fine but
the main goal of this was to show you that images
store the images digitally as texts. Now the image
you selected try to scale it until you see it

Every image consists of several millions of

pixels with one color value which together forms
an image as we see. And as machine learns through
patters in data the same applies here i.e.
machines finds relations in images on the basis of
their several properties which also includes each
and every pixel. It may sound weird but that's the
idea behind computer vision
 • ----------------------------------------
203 COMPUTER VISION
-J
•

Tweaking images
So let's import images first and perform a little
bit of tweaking on it. To do so we need to install
the opencv library using anconda

□ Anaconda Prompt (anacondal)

(base) C:\Users\Rahul>conda install opencv.

Now we can import images using the opencv

libraries imreadO function, so let's import an
image. You can pick any image you want but let me
show you how the feet of a mosquito looks like
when scaled up

In [1]: import cv2

img = cv2.imread('mosquito_feet.jpg’)
cv2.imshow("Mosquito Feet",img)

A new Mosquito Feet window will open where the

imported image can be seen
 •f
204 COMPUTER VISION

If you want we can also visualize the image in

the jupyter notebook with matplotlib but we need
to change the color mode from BGR to RGB

In [2]: from matplotlib import pyplot

cimg = cv2.cvtColor(img., cv2.COLOR_BGR2RGB)
pyplot.imshow(cimg)

0ut[2]: <matplotlib.image.Axeslmage at 0x2345e4a0df0>

100

200

300

400

500

600

0 100 200 300 400 500 600 700

You may believe or not but these is the magnified

view of a mosquito's leg. But let's not stray to
talk about it much. So let's say we want to
convert the image into black and white using the
cvtCotorO function where the first parameter is
the image and the second is conversion i.e.
C0L0R_BGR2GRAY here. Earlier we used BGR2RGB
In [3]: gimg = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
cv2.imshow('BlacknWhite',gimg)
 z
205 COMPUTER VISION
--- '•
We can also save the grayscale image using the
imwriteO function and pass the filename and the
image itself

In [4]: gimg = cv2.cvtColor(img>cv2.C0L0R_BGR2GRAY)

cv2.imwrite('BW_mosquito_feet.jpg',gimg)

0ut[4]: True

Check your jupyter notebook homepage where the

image will be saved. Let's say we want to create
an image effect like linearts. We can do so using
the CannyO function

In [7]: edg = cv2.Canny(gimg,100,200)

pyplot.imshow(edg)

Out[7]: <matplotlib.image.Axeslmage at 0xlb0cd0b5d60>

100

200

300

400

500

600

As you can see the image is converted into

lineart or more specifically the edges. The
CannyC) function does edge detection on images
using a gradient function where 100 and 200 are
it's parameters. But the color is blue because
matplotlib expects images to be in RGB mode but
opencv returns an BGR version
 • -------------------------------------------------------------

206 COMPUTER VISION

—

Face recognition
Using opencv we create a face recognition model.
You can choose any image you want. For
demonstration, I will use the following image:

Now we need to load the dataset for the Face

Recogniser, which you will find here -
anaconda3\Lib\site-packages\cv2\data\
haarcascade_frontalface_default.xml

In [1]: import cv2

dt = r"anaconda3\Lib\site-packages\cv2\. ..
FaceRecog = cv2.CascadeClassifier(dt)

Make sure to load the dataset properly or the

face recongniser will not work. And also the file
path should be stored as a raw string i.e.
prefixing it with p" so the \ escape characters
are treated as part of the string
 9/ s
207 COMPUTER VISION
- J—

We used the CascadeCLassifier class to create

the FaceRecog object. Now we can load our image
and change it to grayscale because the FaceRecog
will only accept grayscale images

In [2]: img = cv2.imread('peoples.jpg')

gimg = cv2.cvtColor(img, cv2.C0L0R_BGR2GRAY)

So our input is ready and we can pass it to the

FaceRecog. Along with the image we need to pass
1.3 and 5 as additional arguments to help in the
face detection. If face is detected the recogniser
will pass four values which will form an rectangle
around the face. Using the for loop we can plot
the rectangle onto the image using the
rectangle() function along with the additional
numeric values

In [3]: fes = FaceRecog.detectMultiScale(gimg, 1.3,5)

for (x,y,w,h) in fes:
img = cv2.rectangle(img,(x,y),(x+w, y+h),
(255,0,0),3)

Now let's save the img and view it

In [4]: cv2.imwrite('Detected-Faces.png',img)

0ut[4]: True
 • -------------
208 COMPUTER VISION
—

Let's enlarge the image to see the rectangle

around the face clearly

Using opencv we can perform all sorts of image

related tasks like these
 Have you every played Tic Tac
Toe? It is a fun game consiting of
two players playing in turns to
create a combination of three
symbols in a row. So let's create
an Al to play it with you, sounds
fun right?
To create the game we need to
import an package called easyAI which
will ease our task of creating the Al. So let's
install it through anaconda prompt

Anaconda Prompt (anaconda^)

(base) C:\Users\Rahul>pip install easyAI^

So jump onto your jupyter notebook and open up a

new notebook. Let's import the dependencies we
need to create the game

In [1]: from easyAI import TwoPlayersGame, AI_Player> Negamax

from easyAI.Player import Human_Player

We need the TwoPlayersGame class as the parent

class of our tic tac toe game. The AI_Player is
the machine which will play with us and Negamax is
the algorithm going to be used by the AI_Player.
We also imported Human_Player so we can play
againt the Al. Now I want to you to think to the
requirements in a tic tac toe game like, the 3 by
3 matrix or board where we will play, the symbols
used, number of turns, possible combinations,
winning combinations, etc. Then you will be able
to connect with the flow properly
 • F*--------------------------------------------------------------

211 Al APPLICATION
•

First-off all let's create the TicTacToe_game

class and inherit the TwoPtayersGame too

In [2]: class TicTacToe_game(TwoPlayersGame):

def init (self, players):
# Starting PLayer
self.players = players
# Symbol, of the pLayer
self.nplayer = 1
# Creating the matrix
self.board = [0] * 9

After creating the class comes the init ()

intialization where we will intialize the players
i.e Al and we. The we will create the symbol for
the player and the 3*3 borad i.e.

where we will pass the value for each box or dot

from the number 1 to 9. So if we want to cross in
the middle box we need to pass 5

. X .

Or if we want to cross on the left-bottom we need

to pass 7

X . .

let's define a function for the same i.e. creating

the moves
 • F*--------------------------------------------------------------

212 Al APPLICATION
•

In [2]: def possible_moves(self):

return [x + 1 for x, y in\
enumerate(self.board) if y == 0]

We defined a new method possible_moves() adn as

it's name suggests we will define the possible
moves. So what can be the posible moves in tic tac
toe? There can be 9 possible moves and as
mentioned earlier herer they will be denoted as
1-9. So first of all let's say you pass 7 i.e.
bottom-left, is it possible? Well ofcourse it is.

X . .

and then the Al passes 1 that means it want to use

the first box, is it possible? Well it is too.

0 . .

X . .

then again it's your turn and you pass 5 this time

0 . .
. X .
X . .

but what if the Al passes 5 too because it's the

most favourable move. Or how can we stop ourselves
from passing 1? To do so we will use the return
statement(i.e. x + 1 for x, y in enumerate (self,
board) if y==0) where we use the enumerate
function on our borad that passes [(0,0), (1,0),
(2,0), ..., (8,0)] i.e. a list of tuples in
 • F*--------------------------------------------------------------

213 Al APPLICATION
•

def possible_moves(self):
return [
enumerate(self.board) ]

which the first elements of the tuples are the

possible moves(which will be raised by 1 later
after for loop) and the second is whether it is
usable(0) or not(l), so when a move let's say 1 is
used the 0 next it will be raised by one(function
defined later when getting moves in real). And as
we added a if statement to compare the value of
y(second element) and pass on the tuples with y of
0 value

def possible_moves(self):
return [
enumerate(self.board) if y == 0]

Now we need a list of the possible moves with

possible moves at each turn so we will use list
comprehension i.e. a for loop to generate it's
values
def possible_moves(self):
return [x + 1 for x, y in\
enumerate(self.board) if y == 0]

where we will get the x(first element i.e. moves)

and y. Note that first of all the control is with
the enumerate() function which creates a list of
tuples of possible moves and their usability, then
we iterate over the tuples using the for loop and
if y == 0 i.e. the x next to y is still usable
then pass it and atlas we raise the x by 1. If you
didn't get it don't freak, just give it a read
again and you'll get the idea
So now we need the moves i.e. 1, 7, 5 etc. from
the players so let's make function for that
 • F*--------------------------------------------------------------

214 Al APPLICATION
-J
•

In [2]: def make_move(self, move):

# Store the move in npLayer
self.board[int(move) - 1] = self.nplayer

We defined the make_moveO function to get the

move from the player. As you can see arrays are
indexed starting from 0 i.e. the range is 0-8 but
we will pass moves in range 1-9 that's why, we
will subtract from it and then use that indexed
element from the board or matrix and store it to
self.nptayer(i.e. cross X)
Now we need to define the same function for the
Al i.e. umake_rnove(). The box or place where the
move is made will be chnaged to 0(i.e. circle 0)
def umake_move(seIfj move):
# TeLLing when the pLayer is making
# moves to boost the Al
self.board[int(move) - 1] = 0

Now we need to specify the conditions for win or

lose. So let's figure out different possible
combinations to win the game:

XXX 12 3

XXX 4 5 6

XXX 7 8 9

Horizontally we have three possible pairs i.e.

[1, 2, 3], [4, 5, 6] and [7, 8, 9]. Then also,
 9/ s
215 Al APPLICATION
- J—

X . . 1 . .
X . . 4 . .
X . . 7 . .

. X . . 2 .
. X . . 5 .
. X . . 8 .

. . X . . 3
. . X . . 6
. . X . . 9

Vertically we have three possible pairs i.e.

[1, 4, 7], [2, 5, 8] and [3, 6, 9]. Then also,

X . . 1 . .
. X . . 5 .
. . X . . 9

. . X . . 3
. X . . 5 .
X . . 7 . .

Diagonally we have two possible pairs i.e. [1, 5,

9] and [3, 5, 7]. So let's create a function to
define the losing condition for the Al
def condition_for_lose(self):
# Wining combinations
possible_combinations = [[1,2,3], [4,5,6], [7,8,9],
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]
return any([all([(self.board[z-l] == self.nopponent)
for z in combination])
for combination in possible_combinations])

So we created the condition_for_l_ose()

function. Let's break up the function to
understand it better
 • F*--------------------------------------------------------------

216 Al APPLICATION

First of all we created the

possibte_combinations list which contains all
the winning combinations
def condition_for_lose(self):
# Wining combinations
possible_combinations = [[1,2,3], [4,5,6], [7,8,9],
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]

Again we use list comprehension to create the

mapping of the board or the matrix like if a
winning combination say [2, 5, 8] is made we will
use those values and map them into

. X .
. X .
. X .

First of all we will get each pair from the

possibte_combinations using for loop and store
the pair in combination
possible_combinations = )[[1,2,3],
def condition_for_lose(self : [4,5,6], [7,8,9],
# Wining combinations

| return
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]
([

for combination in possible_combinations])

Then using another for loop or inner loop we get

each move from the combination
def condition_for_lose(self): as z
# Wining combinations
possible_combinations = [[1,2,3], [4,5,6], [7,8,9],
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]
return ([
for z in combination])
for combination in possible_combinations])
 • F*--------------------------------------------------------------

217 Al APPLICATION
•

Then we need to check each move individually with

the player i.e. us. For the same we will use ==
operator for each move z in the combination pair
in the possibte_combinations. You may say that
we can pass the combination-*s directly but it
can't be because the moves are single integral
values not lists thatJs why we need to go through
the loops to check each move in a particular
combination
def condition_for_lose(self):
# Wining combinations
possible_combinations = [[1,2,3], [4,5,6], [7,8,9],
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]
return ( ([(self.board[z-l] == self.nopponent)
for z in combination])
for combination in possible_combinations])

The comparison will be done with the move made by

player i.e. stored in sett. nopponent. You may
say we haven't defined any attribute like so;
that's correct. The self. nopponent is inherited
from the TwoPtayensGame class which asks the
users input and stores it there, you'll see it
when we will run the program. Now if all the moves
z in a pair combination matches with opponent we
will pass True using att() function i.e. True if
all the moves z in a pair combination matches
with opponent
def condition_for_lose(self):
# Wining combinations
possible_combinations = [[1,2,3], [4,5,6], [7,8,9],
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]
return ([all([(self.board[z-l] == self.nopponent)
for z in combination])
for combination in possible_combinations])

Getting out of the inner loop i.e. comparison of

one combination we need to use the any()
function which will pass True if any of the aU()
 9/ s
218 Al APPLICATION
-

passes True for any combination in the whole

process
def condition_for_lose(self):
# Wining combinations
possible_combinations = [[1,2,3], [4,5,6], [7,8,9],
[1,4,7], [2,5,8], [3,6,9], [1,5,9], [3,5,7]]
return any([all([(self.board[z-l] == self.nopponent)
for z in combination])
for combination in possible_combinations])

passes True for any combination in the whole

process
Now we can move onto the next step i.e. when will
the game end defining is_over()
In [2]: def is_over(self):
# End game if 9 turns are pLayed or
# the opponent wins
return (self.possible_moves() == []) or\
self.condition_for_lose()

The game will end when all the boxes in the borad
are filled i.e. after 9 turns or if a winning
combination is made. Note that we are calling the
condition_for_Tose() method inside of another
method. Now after we have defined the rules and
all the conditions we need to print the board or
matrix too so let's define the showO function
def show(self):
# Printing the board
print(’\n'+'\n'.join([' ’. join( [ [' . ’, 'O', ’X1]
[self.board[3*j + i]]
for i in range(3)]) for j in range(3)]))

Once again we will use list comprehension. We

will get 3*3 values values from the for loops and
then use them as index for the board and use the
'O', 'X'] symbols as mapping the board
using joint) method which will concatenate the
symbol whether 0 or X to the ' ' string
 • F*--------------------------------------------------------------

219 Al APPLICATION
•

Atlas we need to define the scoring 0 function

where we need to return 0 when player hasn't made
a winning combination or -100 if we win. The
algoirthm will learn from the mistakes and try to
revert back to 0

def scoring (self):

return -100 if self.condition_for_lose() else 0

So our TicTacToe_game is ready to be played

with. So let's call our game and pass
Human_Player and AI_Ptayer to it. Also pass
Negmax(7) for the AI_Ptayer intialization. Then
call the ptayO method

In [*]: Game = TicTacToe_game([Human_Player()>

AI_Player(Negamax(7))])
Game.play()

Player 1 what do you play ?

We created the Game object of our

TicTacToe_game class and passed the players.
Then we called the ptayO method to play the
game., you may think we didn't defined any ptayO
method that's correct it's inherited from
TwoPtayersGame. You can see the board or the 3
by 3 matrix printed there

and it's asking for our move. So let's play the

game and I'm passing 7 i.e. bottom-left
 • F*--------------------------------------------------------------

220 Al APPLICATION
__________

Player 1 what do you play ? 7

Move #1: player 1 plays 7 :

0 . .

Move #2: player 2 plays 5 :

After I passed 7 and hit Enter; it printed

Move #1: player 1 plays 7 and the borad with my
move

0 . .

Next it is AIJs turn where the Al passes 5 as itJs

move i.e. middle box, pretty smart. And again we
are asked to pass our next move, so IJ11 pass 1
this time and see if the Al manages to block we or
not

Move #2: player 2 plays 5 :

. X .
0 . .

Player 1 what do you play ? [~l[

1
 • F*------------------------------------------
221 Al APPLICATION
—

Player 1 what do you play ? 1

Move #3: player 1 plays 1 :

0 . .
. X .
0 . .

Move #4: player 2 plays 4 :

0 . .
X X .
0 . .

Player 1 what do you play

We passed 1 i.e. the first block. The Al

succesfully blocked our move i.e. used 4 as move.
Now we need to block the Al or it will win, so we
need pass 6

Player 1 what do you play ? 6

Move #5: player 1 plays 6 :

0 . .
X X 0
0 . .

Move #6: player 2 plays 2 :

0 X .
X X 0
0 . .

Player 1 what do you play ? 8

We passed 6 i.e. the middle-right. The Al aimed

for the vertical combination and passed 2. So we
need to block it again or Ai will win. So let's
pass 8 as our next move i.e. mid-bottom
 • F*------------------------------------------
222 Al APPLICATION

Player 1 what do you play ? 8

Move #7: player 1 plays 8 :

0 X .
X X 0
0 0 .

Move #8: player 2 plays 9 :

0 X .
X X 0
0 0 X

Player 1 what do you play ? p

We played 8 and now there is no way left for

anyone to win. The Al plays 9 and we are again
asked to pass our our next and last move. Well 3
is the only option left, so be it

Player 1 what do you play ? 3

Move #9: player 1 plays 3 :

OXO
X X 0
0 0 X

Out[3]: [(< main .TicTacToe_game at 0x20bc93b7d90>, 7),

(< main .TicTacToe_game at 0x20bc93e3e80>, 5),
(< main .TicTacToe_game at 0x20bc93e83d0>, 1),
(< main .TicTacToe_game at 0x20bc93e3c70>> 4),
(< main .TicTacToe_game at 0x20bc93e36a0>3 6),
(< main .TicTacToe_game at 0x20bc93eb760>, 2),
(< main .TicTacToe_game at 0x20bc93e94c0>, 8),
(< main .TicTacToe_game at 0x20bc93ebb50>} 9),
(< main .TicTacToe_game at 0x20bc93b7b20>, 3),
< main .TicTacToe_game at 0x20bc93eb4f0>]

So 9 turns are played and the game is finished

with a tie. We also get the inputs or moves
printed. As you saw the Al has successfully
blocked our attempt to win and forced us to
 • F*------------------------------------------
223 Al APPLICATION
•

block it's winning attempts. If we didn't have

noticed, it may have won. The main purpose of this
lesson was to understand the power of Al. I don't
the thought came to your mind or not but the Al
was playing like a real human with us even though
we didin't trained it with anything
24 APPLICATION 2
• Seorch
engine

o------
 24°(---------------
(---- Al APPLICATION 2
-------------------------- y

Problem: You have a website where

you are selling books(say 7). For
the website you have to create a
search engine where the visitor
will a search term and one of the
book details will be shown. You
have the following dataset of the
title of the book., description,
keywords(tags) and price of the book
bit.ly/BookMeta

So we have to create a search engine. Our first

task is to decide which method to use? For this
problem we will recieve the user input in normal
language, so we need to use NLP and then we need
to classify the search result into one book so we
need use a classifier too
Then head over to your jupyter notebook and
import the dataset along with the other essential
dependencies

import pandas
from sktearn.tree import \
DecisionT reeCLassifier
from sktearn.feature_extraction.text import \
CountVectorizer, TfidfTransformer
 r*----------------------------------------------------

226 Al APPLICATION

and also preview our dataset

In [1]: import pandas
from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction.text import \
CountVectorizer, TfidfTransformer
dt = pandas.read_csv('books_metadata.csv')
dt

Out[l]:
Title

0 It's Python

1 Data Science with Python

2 Machine learning with Python

3 Artificial Intelligence with Python

4 Data analysis with Python

5 Data visualization with Python

6 OOP with Python

Description

A beginners guidebook to programming with Pyth...

Data Science with Python is a book created to ...

Do you know all the automatic recommendations ...

Learn the idea behind how to create intelligen...

Learn how to perform different types of analyt...

Looking at raw data is boring, present it in t...

Object Oriented Programming with Python. Learn...

Keywords Price

python for beginners, python programming $3.00

data science, python $3.09

machine learning, python, regression, classifi... $3.19

ai, python, artificial intelligence $3.50

data analysis, pandas, data processing, python $3.00

data visualization, python, matplotlib $3.00

object oreiented programming, python $3.00

 • F*------------------------------------------
227 Al APPLICATION
• -J

As you can see the dataset has 5 columns Title,

Description, Keywords and Price. We have to create
a search engine with keywords of the books. So how
do a search engine works? Let's consider keywords
of the 2nd book i.e. data science & python. The
search terms matching the keywords should be
displayed. A search engine takes th keywords and
creates different and unique combinations with the
keywords like we have data science & python so the
combinations will be made like:

data science python

python science
data python

where these are keywords for Data Science book.

Doing so the search results can be broadened like
it is likely to un-happening that someone will
always enter search terms exactly as the keywords
so the search term can be

data science with python

python for science
data in python

where the we have keywords matching with the terms

so the search engine can understand that the user
wants to see the book for Data Science
Similary we have to create another dataset which
will extract all the words from the keywords for
each book and then form unique combinations with
them and then we can apply CountVectorizer and
TfidfTransformer's to ready them for our
classifier to understand and then we will train
our classifier that will be a decision tree
classifier and then we can pass the search terms
 • F*------------------------------------------
228 Al APPLICATION
-J
•

to it. So let's see how can we do so. First of

let's perform the keyword extraction and forming
combinations for the first book i.e. It's Python.
We will use tokenization to extract individual
words so import the wond_tokenize() function

In [2]: from nltk import word_tokenize

Kys = dt[’Keywords’ ].values

bl_kys = word_tokenize(Kys[0])
bl_kys.remove(', ’)
bl_kys

Out[2]: ['python'3 'for', 'beginners'} 'python', 'programming']

First of all we got the Keywords values from the

dataset and stored it in Kys. Then we tokenized
the keywords for the first book(i.e. indexed 0)
and also removed comma's in the keywords and you
can see the tokenized words printed
We have extracted keywords next step is to create
unique combinations. To do so we will use
constraint; if you don't remember it you should
go back to the Al Algorithms lesson and come back
here

In [3]: import constraint

# Creating Combinations
com = constraint.Problem()
com.addVariables(["a","b"], bl_kys)
com.addConstraint(constraint.AllDifferentConstraint())

We also imported the constraint module to create

an object of Problem. We need the combinations so
we added two variables namely a and b to the
problem and passed the extracted keywords for the
first book i.e. bl_kys. Next to form different
and unique combination we added a constraint using
addConstraint0 and then passed the
AULDifferentConstraint() as constraint
 • F*------------------------------------------
229 Al APPLICATION
•

For each pair of a and b in the solutions we will

recieve the combinations. So let's print them
using the for loop from the getSotutions()

In [4]: wrds = []
for ele in com.getSolutions():
wrds.append(f"{ele['a']} "+f "{ele [ ’ b ' ]}")

wrds

0ut[4]: ['programming python',

'programming beginners'
'programming for',
'programming python',
'python programming',
'python beginners',
'python for',
'beginners python',
'beginners programming',
'beginners python',
'beginners for',
'for beginners',
'for python',
'for programming',
'for python',
'python for',
'python beginners',
'python programming']

Before creating the for loop we created an empty

wrds list to append values it later. Then we
called the getSolutions() method which returns a
list of dictionaries with the defined
variables(i.e. a and b) as keys and the solutions
as values. So we appended them in the wrds list in
each iteration. Note that we used formatted
strings i.e. strings prefixed with f" ” where
variables can be passed inside curly braces { }
which is converted into string. Also note how we
used the variable names(a and b) as indexes to get
the values from the dictionary in each iteration
and then concatenated them
 • F*------------------------------------------
230 Al APPLICATION
-J
•

We have to repeat the same for all the books i.e.

keyword extraction and forming combinations with
them. We need to seperate those combinations
seperately for each book so we will store them in
a dictionary

In [5]: Titles = dt['Title'].values

Kys = dt['Keywords'].values
keywrds = {'Bl': [], 'B2': [], 'B3': [], 'B4': [],
'B5': [], 'B6': [], 'B7': []}
kys = []
wrds = []
for i in range(7):
kys.clear()
wrds.clear()
kys = word_tokenize(Kys[i])
while (kys.count(',')):
kys.remove(',')
com = constraint.Problem()
com.addVariables(["a","b"L kys)
com.addConstraint(
constraint.AllDifferentConstraint())
for ele in com.getSolutions():
wrds.append(f"{ele['a']} "+ f"{ele['b']}")

for k in wrds:
keywrds[f"B{i+1}"].append(k)

We created a keywrds dictionary to store

extracted keyword and their combinations in lists.
Note that we created keys from Bl to B7 and their
values as empty list which we will append in the
for loop. We also created kys and wrds empty
lists to store extracted keywords and
combinations. In the for loop we will get the
index of the books. We started by ciearning the
kys and wrds list which will be helpful in the
next iteration. Then we extract the keywords of
the book indexed i i.e. number from 0-6 got in
each iteration and using while loop we removed
commaJs in the keywords. The loop will be
terminated when the count of comma in the keywords
 • F*------------------------------------------
231 Al APPLICATION
—

will be reduced to 0 i.e. the condition(

kys.count . So the keywords are extracted
and we can can create the Probtem object, add
variables, their range as the keywords and the
AttDiffenentConstnaint () as constraint. Then
using a for loop we appended the combinations to
the wrds list as we did before. And atlas we
appended those combinations from the wrds to the
keywrds dictionaries key values. We can inspect
the dictionary to check the values

In [6]: keywrds

0ut[6]: {'Bl': ['programming python',

'programming beginners',
'programming for',
'programming python',
'python programming',
'python beginners',
'python for',
'beginners python',
'beginners programming',
'beginners python',
'beginners for',
'for beginners',
'for python',
'for programming',
'for python',
'python for',
'python beginners',
'python programming'],
'B2': ['python science',

'B7': ['python programming',

'python oreiented',
'python object',
'programming python',
'programming oreiented',
'programming object',
'oreiented programming',
'oreiented python',
'oreiented object',
'object oreiented',
'object programming',
'object python']}
 9/ s
232 Al APPLICATION
- J—

So we have created a dictionary of all the unique

combinations of the keywords. Now we need to
change these data i.e. dictionary into input and
output datasets. For that we need to create
another 2-D list with 1-D list with elements as
the keywords combination and the book label i.e.
Bl, B2, B7. We can do using for loops too

In [7]: Dt = []
for bk in list(keywrds.keys()):
for ks in keywrds[bk]:
Dt.append([ks,bk])

We created an outer loop and an inner loop. On

the outer loop we will recieve the keys i.e. Bl,
B2, etc. and in the inner loop we will recieve the
values of the recieved key from the keywrds
dictionary. We wil use both of these values to
create our dataset. We can create a pandas
dataframe to view the dataset

In [8]: import numpy

Dt = numpy.array(Dt)
KYS = pandas.DataFrame({'Keywords':Dt[:,0],
’Book’:Dt[:,l]})
KYS

Out[8]:
Keywords Book

0 programming python B1

1 programming beginners B1

2 programming for B1

3 programming python B1

4 python programming B1

158 object programming B7

159 object python B7

160 rows x 2 columns

 • F*------------------------------------------
233 Al APPLICATION
-J
•

So let's divide the dataset into input and

output. And also we need to apply the
CountVectorizer and TfidfTransformer on the
input dataset i.e. keywords

In [9]: inp = Dt[:,0]

Y = Dt[:,l]

enc = CountVectorizer()
enc.fit(inp)
e_inp = enc.transform(inp)

tf = TfidfTransformer()
tf.fit(e_inp)
X = tf.transform(e_inp)

Now we can create our classifier CModet and

train it with the input and output

In [10]: CModel = DecisionTreeClassifier()

CModel.fit(X,y)

Out[10]: DecisionTreeClassifier()

The base stone of our search engine is complete

i.e. the classifier. Now we have to create a
funcition to get the search terms which we will
pass through the input() function. Then we need to
apply the CountVectorizer and TfidfTransformer
on the search terms and pass that to our
classifier to predict the book the user wants. The
classifier will pass an output from B1-B7 so we
need to use that and display something like:

<Title>

description ...
... >

<Price>
 9/ s
234 Al APPLICATION

In [11]: def search_book():

query = input('Search: ')
enc_q = enc.transform([query])
tf_q = tf.transform(enc_q)
bk = CModel.predict(tf_q)
i = {'Bl': 0, ’B2': 1, *B3': 2, 'B4*: 3,
'B5': 4, 'B6': 5, 'B7': 6}
title = dt.values[i[bk[0]],0]
desc = dt.values[i[bk[0]],l]
price = dt.values[i[bk[0]],3]
result = f"""
Tittle:
{title}

Description:
{desc}

Price:{price}"""
print(result)

As you can see we created the search_book()

function. We transformed the input and asked the
classifier to pass the book number. The prediction
will be in the form B1-B7 so we created an i
dictionary to map the predicted values to book
index(in the dt dataset). Then we extracted the
title, description and price of the book and used
an formatted multiline string to display the
search result
Our searach engine is ready! So let's call the
function and pass a relevant search term

In [*]: search_book()

Search: | book for pythor?

So we called our search_book() function which has

prompted us to input the search term. Then let's
pass a value that wasn't present in the keywords.
We will only the python keyword and we want to see
the Python Programming book here
 • F*------------------------------------------
235 Al APPLICATION
• -J

We passed 'book foe python' where python is the

only keyword and it has it's odds too because it
is used in all the books. So can let's hit Enter
and see what our search engine shows!

In [12]: search_book()

Search: book for python

Tittle:
It's Python

Description:
A beginners guidebook to programming with Python. We
f statements, creating loops and how to iterate over

Price:$3.00

Our search engine has showed us what we wanted to

see i.e. the Python Programming book. You can see
the Tittle, description(i.e. clipped so please
execute yourselves to see it properly) and the
price our book. Let's call our function again and
pass something more weird like 'guide to data
cleaning'. You can go a few pages back and see
that the keywords barely have these words. So the
book it should show us is the Data Analysis book.
Let's test our search engine once again

In [*]: search_book()

Search: |guide to data cleaning

So we called our search_book() once again and we

are prompted to input the search term again. And
as mentioned earlier this search term has more
odds that the previous one because the only
keyword used here is data and cleaning which is
paired with some unseen words like guide to. But
let's see what our engine shows us!
 9/ s
236 Al APPLICATION

In [13]: search_book()

Search: guide to data cleaning

Tittle:
Data analysis with Python

Description:
Learn how to perform different types of analytics on
ncoding, Scaling., Standard deviations, interquartile

Price:$3.00

Whoa! Our search engine has nailed it once again.

I'm really surprised by the results. I thought it
will show Data Science book but it specifically
shows us the Data Analysis book
So go pass some search terms yourslef and test
the search engine!
 Previously we created a search
engine for some books. But search
engine is incomplete until voice
search isn't available. So let's
add the voice search feature to
our previously created search
engine and to spice things up the
search results will be automatically
converted into speech

We already know how to create a voice recognizer,

so we need to just pass the voice to captured
through microphone and pass the recongnised text
as search term. But we are also adding an feature
i.e. narration of the search result. To do so we
need to install the Google TTS(Text to Speech)
API, so install it through anconda prompt

~ Anaconda Prompt (anaconda!)

(base) C:\Users\Rahul>pip install gTTS

And we also need the PyAudio package, so install

that too

~ Anaconda Prompt (anaconda!)

(base) C:\Users\Rahul>pip install PyAudio

If you get errors installing the PyAudio package

consider downloading the following:
https://visualstudio.microsoft.com/visual-cpp
-build-tools
 9/ s
239 Al APPLICATION
- J—

After installations move the cell where we

defined the search_book() function and import the
speech_necognizen, gTTS from gtts and IPython

In [29]: def search_book(Type='text'):

import speech_recognition
from gtts import gTTS
import IPython

We will use speech_recognizer to get the

voice(from microphone) and then recognize the
search term, we will use gTTS from gtts to
convert the search result to speech and we will
use IPython to play the audio i.e. speech
As you can see we created the Type parameter,
which we will use to get the method of the search
i.e. either text or voice search. So letJs get the
search terms from the user

In [29]: def search_book(Type='text'):

import speech_recognition
from gtts import gTTS
import IPython

if Type == 'text':
query = input('Search: ')
elif Type == 'voice':
vois = speech_recognition.Recognizer()
with speech_recognition.Microphone() as a:
vois.adjust_for_ambient_noise(a)
print("Your Query:")
aud = vois.listen(source)
query = vois.recognize_google(aud)

We used if statements to either get the input as

text or speech. It is same for the text but we
added the voice search option where we created a
Recognizer object and got the speech from the
user using MicnophoneC) and recognized it
 • F*------------------------------------------
240 Al APPLICATION
—

Now we have the search terms so let's apply the

CountVectorizer and TfidfTransformer to it and
then pass it to the classifier to predict

In [29]: def search_book(Type='text'):

import speech_recognition
from gtts import gTTS
import IPython

if Type == 'text':
query = input('Search: ')
elif Type == 'voice':
vois = speech_recognition.Recognizer()
with speech_recognition.Microphone() as a:
vois.adjust_for_ambient_noise(a)
print("Your Query:")
aud = vois.listen(source)
query = vois.recognize_google(aud)

enc_q = enc.transform([query])
tf_q = tf.transform(enc_q)
bk = CModel.predict(tf_q)

And as we did before we will create search result

with the title, description and price of the
predicted book by using the dictionary to get
index and then extracting the book attributes from
the dataset
i = {'Bl': 0, ' B2': 1, " B3': 2, 'B4': 3,
•B5': 4, 'B6': 5, 'B7': 6}
title = dt.values[i[bk[0]],0]
desc = dt.values[i[bk[0]],1]
price = dt.values[i[bk[0]]^3]
result = f"""
Tittle:
{title}

Description:
{desc}

Price:{price}
 • F*------------------------------------------
241 Al APPLICATION
-J
•

Now we need to change the result into speech. For

the same we will create a gTTS object and then we
will convert the search result into speech

spkr = gTTS(text = result, lang = 'en')

spkr.save("vois_.mp3")
print(result)
IPython.display.Audio("vois_.mp3",
autoplay=True)

We used passed our search result to the text

parameter and also specified the tang as 'en' for
English. Then we used the save() method to save
the converted speech as a .mp3 file and then we
also printed the result. To play the audio, we use
the IPythonJs disptay.Audio0 function where we
passed our audio file and True to the autoptay
parameter. So our voice search engine is ready! It
is a pity that I canJt include the audio here so
you have to do so yourself. So call the
functionand pass voice to the Type parameter and
say what you want to search for

In [ ]: search_book(Type='voice')
 ■x

26 APPLICATION 4
• Captcho AV

solver

J
 A
Al APPLICATION 4
i__________________________________________ >

Problem: You have to create an Al

to solve captcha's. You'll be
provided with 6 images for 2
times(12 in total) and you have to
tell which images has humans in it
without actually looking at the
images. Here are the images:
bit.ly/caPtcha

http://bit.ly/caPtcha

Scan Me!

This is a problem of computer vision. For this we

need to create a model to detect human in the
images provided and then we need to tell which
images has humans in them without taking a look at
them directly
Then let's move to our jupyter notebook and
create human recognizer and we will test it with
the following:
 244 Al APPLICATION

First of all we need to import the image and

convert it into grayscale image

In [1]: import cv2

image = cv2.imread('test_image.jpg')
img_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Now we need to created a model to detect humans.

We will use the HOGDescriptor0 which has
trained models. We will use the
HOGDescriptor_getDefaultPeopteDetector 0 i.e.
a SVM i.e. support vector machine (SVM is a
machine learning method)

In [2]: HumanRecog = cv2.HOGDescriptor()

HumanRecog.setSVMDetector (
cv2.HOGDescriptor_getDefaultPeopleDetector())

Our models is ready to make predictions so we

will pass the image to detectMuttiScate()
method. If you remember we will recieve the
coordinates of the bounding boxes of detected
objects but here we will recieve one more value
i.e. itJs confidence level. Let's store them into
two variables and then pass the grayscale image,
where we need to provide some parameters that are
all too essential for the detection. The first
parameter is winStride which is the size of
detector like how much the detector should divide
the image into like
 • F*------------------------------------------
245 Al APPLICATION
-J
•

The size of winStride (tuple) which we will be

provided is used to scan over the image, more the
smaller more finer details and vice versa. We will
pass (2,2). Then comes the padding i.e. the
height and width added to the image where we will
pass (10,10) and atlas comes the scale which as
import as winStride and also factors it. We will
pass 1.02 as the scale

In [3]: rects, conf = HumanRecog.detectMultiScale(img_gray,

winStride=(2, 2), padding=(10, 10), scale=1.02)

for (x, y, w, h) in rects:

cv2.rectangle(image, (x, y), (x+w, y+h),
(0, 255, 0), 2)

We also added the bounding boxes(rectangles) for

the detected humans on the image using the for
loop and default arguments. So let's have a look
at the results of the recognition

In [*]: cv2.imshow('HOG detection', image)

cv2.waitKey(0)
 • F*------------------------------------------
246 Al APPLICATION
-J
•

As you can see we have the bounding box overlayed

on the image of the detected figures
But while solving the captcha we aren't meant to
look at the photos so we don't need the bounding
boxes either. For that purpose we will just check
whether there are any values in the rects or not
to knwo which image was humans or not like

In [5]: bool((rects).all())

Out[5]: True

We used the all() method (built-in method) and

passed that to bool(). We have non-zero values
i.e. True. If not humans were to be detected the
rects will store 0 value and for boot() that
means False. But we used all() method because any
coordinate of the bounding box can be zero so when
everything is zero no human is detected i.e.
False
Now we can build our captcha solver that will be
a function

In [6]: def captcha_solvr(*images):

import cv2

humn_img = []

We will recieve 6 images which may or may not

conatin humans. We will recieve our images names
as a list(*arg arbitrary argument) and then we
will use a for loop to iterate over the images
names., open them, detect humans in them anf if
detected we will append that image's number in the
humn_img list else we will move onto the next
image. So let's create a for loop to perform all
the functions mentioned above
 'S
247 Al APPLICATION
----------'•

In [6]: def captcha_solvr(*images):

import cv2

humn_img = []
hog = cv2.HOGDescriptor()
hog.setSVMDetector(
cv2.HOGDescriptor_getDefaultPeopleDetector())
for img in images:
rects, weights = 0,0
Img = cv2.cvtColor(cv2.imread(img),
cv2.COLOR_BGR2GRAY)

(rects, weights) = hog.detectMultiScale(Img,

winStride=(2, 2), padding=(10, 10),
scale=1.02)
try:
if bool((rects).all()):
humn_img.append(
images.index(img) + 1)
except:
continue

return humn_img

As you can see we created the humn_img list in

which we will store the index of the images with
human. Then we created our HOGDescriptor object
and added the people detector SVM. Using a for
loop will iterated over each image name given as
argument. The first step is to change the image
into grayscale. Then we passed the image to the
detector and stored the bounding box coordinates
in rects along with weights. Then created the if
statement to check for values in the rects. It
there are values in rects the index of that image
will be appended in the humn_img list. But if
there are no humans present in the image the
detectMuttiScate0 method will return an empty
tuple and the condition i.e. bool((rects).att())
may raise an error; and we catch that error using
try and except block and continue on next image
 • F*------------------------------------------
248 Al APPLICATION
—

Our captcha solver is ready to solve real

problems! Now open the zipped file with the
captchas and which will have a folder called
captcha. Put the folder into your jupyter
notebook home and then load the csv file with the
names of each file i.e. located in
captcha\Captchal\img_names_l.csv

In [7]: from pandas import read_csv

dt = read_csv(r1captcha\Captchal\img_names_l.csv1)
dt

Out[7]:
Labels Check

0 captcha\Captcha1\Caplmg1.jpg 1

1 captcha\Captcha1\Caplmg2.jpg

2 captcha\Captcha1\Caplmg3.jpg 3

3 captcha\Captcha1\Caplmg4.jpg

4 captcha\Captcha1\Caplmg5.jpg

5 captcha\Captcha1\Caplmg6.jpg 6

You can see we have the loaction for the 6 images

and also which images has humans i.e. the Check
column. You can see 1st, 3rd and 6th image have
humans. Though we haven't seen the images, I would
like you to see them after the prediction made by
our captcha solver. So let's call our
captcha_sotvr() and pass the location of the
images

In [*]: img_names = dt[1 Labels 1].tolist()

captcha_solvr(*img_names)

We extracted the image locations from the dt and

passed the unpacked values(prefixing with *
operator) to our captcha_sotvr (). It will take a
couple of seconds to generate the output
 • F*------------------------------------------
249 Al APPLICATION
-J
•

so don't worry if the output doesn't comes, just

wait a bit!

In [8]: img_names = dt['Labels'].tolist()

captcha_solvr(*img_names)

out[8]: [1, 3, 6]

After 6 seconds the output came and we have 1, 3

and 6 as the images detected with humans i.e.
exactly what we wanted. So we have successfully
built a captcha solver! Now I would like you to
check the images visually and see for yourself.
You'll find another set of images inside the
captcha i.e. Captcha2. So go ahead and solve that
with our function (csv file with image location -
captcha\Captcha2\img_names_2.csv)
 ■x

Al
APPLICATION 5
• Computer vision
with deep learning
from scratch

o
z
Al APPLICATION 5
X. T___________________________________ J

For the last project we will create

based classifier with deep learning,
will create a deep learning model to
classify between dog and cats. It is
a widely applied example of creating
deep learning models for CV based
classification. We will use
Tensorflow’s neural network to
create the model. The dataset is
available here - https://microsoft.com/
en-us/download/details.aspx?id=54765
r—c Cats and Dogs Images']----- .

https://www.microsoft.com/en-us/download/details.aspx?id=54765

k?
But the only downside of downloading from the
official site is that it is not organized into
train-test sets and also contains some errors.
If you donJt want to do the splliting manually you
can download it from here http://bit.Ly/Dog_Cats,
though it consists of 25,000 images

http://bit.ly/Dog_Cats

--------- 1 ziP I
 252 Al APPLICATION
--- '•
If you download the dataset from the official
source, you will recieve a zip file. After
unzipping it you will see 3 things, license .docx
file, ReadMeFile.txt and Petimages folder with all
the images of the Cats and Dogs in seperate
folders
• You need to drop the Petimages folder into your
jupyter notebook home
• The dataset contains 25000 images with half of
the images of dogs and the other half of cats.
After that you need to remove the 666.jpg from the
Cats folder and 11702.jpg from the Dogs folder.
They will cause error because they are corrupted
files
• Then you need to create another folder like
Testimages and create Cat_test and Dog_test in it

El « Local Disk (G) ♦ Users ♦ Rahul ♦ Testimages ♦

* Quick access

□ Desktop

i Downloads

* Cat_test Dog_test
0 Pictures *
D Captcha2

□ Cat

D Music

• Then take out some images of Dogs and Cats from

the main folder and then move them to the created
test folders
After performing all of the above oyu are ready
to jump into jupyter notebook. If you don't want
to do the above just download the dataset from
second option and put it here after unzipping it
 • F**------------------------------------------
253 Al APPLICATION
-J
•

Let's import all the dependencies we will need in

creating the deep learning model

In [1]: import numpy

from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from tensorflow.keras.preprocessing.image import \
ImageDataGenerator
from keras.preprocessing import image as kimg
from matplotlib import image, pyplot

We will need all of the above. Before moving any

further let's understand what we are going to
create and how with the above:
• First of all we have dataset of the images of
dogs and cats (i.e. two classes) divided into
testing(5000) and training(20000) sets
• Then we will create a baseline model of
Sequential class. As it sounds, it will provide
the base(template with functions of a model) for
our model
• Now we need to defined the neural networks of
the deep learning model. As we are using images
instead of numeric values we need to pay special
attention. So how do we create a network to work
with images? Well we already learned that Deep
learning models made with
CNN (Convolutional Neural
Networks) are powerful
enough to work with
large-scale images as data.
But we need to know how do
they work with images.
Let's take this image for
example:
 • F*------------------------------------------
254 Al APPLICATION
•

The CNN takes the 2D structure of the image into

account. It extracts the
features specefic to the image
and learn from them. As we
already know how do they
decide that this image is
from the dog class is
unexplainable but what happens
can be elaborated a little.
We know how basically a
neural network works, it takes the input, passes
it through series of hidden layers with the help
of neurons connecting them. Neurons of CNN are
3-Dimensional that takes the width, height and
depth of the input. It applies a lot of filters on
the image to extract features from it like edges,
corners

Edges Corners

After extracting the features the images passes

from the series of hidden layers connected with
neurons. The neurons of CNN are connected to a
small patch of the output of the previous layer.
These layers consists of convolution layers,
activation functions layers, pooling layers, and
the output layer. We need to create all those for
our model
 • F*------------------------------------------
255 Al APPLICATION
•

• Dataset with images

of different cats and
dogs divided into
training and testing
sets

• The first layer of our

model i.e. the input layer
takes the image in it's raw
format i.e. as it is and
moves to the series of
hidden layers

• Next is the
convolutional layer.
All the computations
of the features and
parameters in the
image are
calculated
here. The
neurons takes
the output of
the layer
i.e.
features
features, etc.
(from
ahead
pixeLs)
 • F*------------------------------------------
256 Al APPLICATION
-J
•

• Next comes the

activation function layer.
It applies the activation
function to the previous
layer's output i.e. it
adds non-linearity to
the network so
it can
generalize
well with
any function
used in the
further
process.
Remember we used
relu i.e.
rectified linear
activation
function before

S' • After the activation

function comes the
pooling layer. Polling
layer can be said to
scale the image down and
keep the essential
features and
parameters.
It works
independently on
each one of them
and resizes them
spatially. It
uses MAX
function for the
purpose
 9/ s
257 Al APPLICATION
- J—

• The final layer is the Output

layer which is supplied with
rich connection of neurons
from the previous layers
output because the
final output i.e BP
the
classification
is done here.
Dogs will
be classified
into one and
the cats will
be into another

I think that you have understood what are the

different layers in the deep learning model which
we are going to create and what are their
functions. Even though, except for the input and
the output layer; all the layers are considered
hidden. And to your surprise the
classification is done at
the last layer i.e. the
output layer. This
means that the deep
learning models
spend more time on
learning, feature
extraction, finding
patters-relations,
etc. more and less
on testing
So let's build our
 • F*------------------------------------------
258 Al APPLICATION
-J
•

model and the first step is to create the base

line which we will create using Sequential class

In [2]: CModel = Sequential()

CModel.add(Conv2D(32, (3, 3),
input_shape = (64, 64, 3),
activation = 'relu'))
CModel.add(MaxPooling2D(pool_size = (2, 2)))
CModel.add(Flatten())
CModel.add(Dense(units = 128,
activation = 'relu'))
CModel.add(Dense(units = 1,
activation = 'sigmoid'))

We create our baseline CModel. Then we added the

convolution layer using the Conv2D() function
where the first argument is the number of output
produced by that layer i.e. 32 here(it can be
referred as to features), next is the size of the
kernel where we passed the width and height (3, 3)
of it(it can be referred as how dense the 2-D
convolution window will be), next we passed the
input_shape which consists of the batch size and
channels. And atlas the activation method i.e.
relu. Then we added the pooling layer with the
MaxPooling2D () where the poot_size is (2, 2).
After pooling we need to convert the pooled images
into a continuous vector; for the same we use
Flatten (). Then we added the output layers with
128 hidden units (rich connection of neurons) and
then we intialize the output layer with another
layer with sigmoid as activation function i.e. a
logistic function (classification). So our model
is ready with all the layers in it!
Now we need to compile our model using the
compile() method. Here we need pass the
optimizer that will be the adam optimizer
algorithm, the loss function will be binary cross
entropy i.e. loss of each vector output
 z
259 Al APPLICATION
--- '•
(classification that will be 0 or l{binary})

CModel.compile(optimizer = 'adam',
loss = 'binary_crossentropy',
metrics = ['accuracy'])

we also specified accuracy as the metrics. Now

we need to generate the training and testing sets.
But you may ask that we have already created them
into seperate folders; that is right but we
haven't imported the images yet. Also we are
dealing with thousands of images here and we
cannot pass them in one go. For that purpose we
have already specified the batch size in the
input_shape of the convolution layer. So we need
to use ImageDataGenerator() to create batches
In [4]: X_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal-flip = True)
y_datagen = ImageDataGenerator(rescale = 1./255)

We have created two generators for the train and

test seperately i.e. X_datagen and y_datagen
respectively. You may think that X means input and
y means output but that's not the case here. Their
covention are not train and test here so keep that
in mind. We can also specifed some preprocessing
on the images in the generators i.e. the rescale
where we are telling to rescale the images,
shear_range is shear value

and the zoom_range that is the zoom value and the

 • F*------------------------------------------
260 Al APPLICATION
—

Now we can load our images and pass them to the

generator to create batches

In [5]: X = X_datagen.flow_from_directory(
"Petimages",
target_size = (64, 64),
batch_size = 32, class_mode = 'binary')

y = y_datagen.flow_from_directory(
'Testimages', target_size = (64, 64),
batch_size = 32, class_mode = 'binary')

Found 20000 images belonging to 2 classes.

Found 5000 images belonging to 2 classes.

As mentioned earlier X and y denotes train and

test not input and output. We used the
fLow_from_directory() method of the created
generators. We passed the parent directories i.e.
training set folder and the testing folder along
with the target and batch size. Our ctass_mode is
binary i.e. 0 or l(cat or dog). Doing so images
will be loaded in batches of specified batch_size
in each iteration to save memory. If we didinJt
passed the images in batches it will require a lot
of memory. You can see that X has 20000 images of
2 classes(subfolders act as classes) and y has
5000 images in total of 2 classes too
Now comes the major part i.e. training the deep
learning model. You may not think this of big deal
but it is; a lot! We already know the size of our
dataset and the data isn-’t numeric data it
consists of images. Training the dataset will
require a lot of computing and hardware
accelearation. The time for training the model
solely depends on your hardware specs especially
your CPU. IJm running at 3.6 GHz with a Ryzen
3rd-gen Hexacore and 16 gigs of memory. So how
much do you think it may take? 2 or 3 minutes? No,
my friend it may take even hours to complete
 • F*--------------------------------------------------------------------------------------

261 Al APPLICATION
• -J

We called the fit() method and passed the

training and testing sets, specified the steps
i.e. length of the respective datasets, epochs
and verbose as 1 i.e. we want to see the training
process. We are storing the training process
outputs (losses) in the TrainP varaible

In [ ]: TrainP = CModel.fit(X, steps_per_epoch = len(X),

validation_data = y, validation_steps=len(y)3
epochs = 25, verbose = 1)

Before executing the code, I want to warn you

again that the training process will take a lot of
time so please read this through and then execute
the code if you are doing things side by side

In [*]: TrainP = CModel.fit(X, steps_per_epoch = len(X),

validation_data = y, validation_steps=len(y),
epochs = 25, verbose = 1)

Epoch 1/25
625/625 [==============================] - 67s
0.6976
Epoch 2/25
625/625 [==============================] - 67s
0.7508
Epoch 3/25
625/625 [==============================] - 67s
0.7560
Epoch 4/25
61/625 [ = > ] - ETA:
Epoch 1/25
after a
625/625 [============================== ] - 67s 107ms
white
0.6976
Epoch 2/25
625/625 [============================== ] - 67s 107ms
0.7508
Epoch 3/25
625/625 [============================== ] - 67s 107ms
0.7560
Epoch 4/25
218/625 [=========> ] - ETA: 37s
 9/ \
262 Al APPLICATION
- J—

As you can see the training processing can be

monitered with the progress bar. Each epoch is
taking an approximate time of 67 seconds. You can
see the hardware consumption captured at different
times during the training process

S Performance £ # X
CPU 34% q / 0/ 100
GPU 10% *

VRAM 12% I *59GHz

3

RAM 37%
FPS —

60 SECONDS 0

Performance H X
CPU 39% QQ0/ 100
GPU 9% *

VRAM 12% 3 59GHz

RAM 36%
FPS -

60 SECONDS 0

I captured them in between the training process

where the percentage ranged from 31% - 46%
The whole process took around 25-26 minutes in my
system. I will include the summary of the whole
training process for as follows:
 263
r*------------
Al APPLICATION

In [18]: TrainP = CModel.fit(X, steps_per_epoch = len(X),

validation_data = y, validation_steps=len(y),
epochs = 25, verbose = 1)

Epoch 1/25
625/625 [==============================] - 67s 107ms/step - loss: 0.5553 - accu
racy: 0.7175 - val_loss: 0.5793 - val_accuracy: 0.6976
Epoch 2/25
625/625 [==============================] - 67s 107ms/step - loss: 0.5372 - accu
racy: 0.7281 - val_loss: 0.5152 - val_accuracy: 0.7508
Epoch 3/25
625/625 [==============================] - 67s 107ms/step - loss: 0.5263 - accu
racy: 0.7387 - val_loss: 0.5028 - val_accuracy: 0.7560
Epoch 4/25
625/625 [==============================] - 67s 107ms/step - loss: 0.5176 - accu
racy: 0.7395 - val_loss: 0.5108 - val_accuracy: 0.7410
Epoch 5/25
625/625 [==============================] - 67s 106ms/step - loss: 0.5091 - accu
racy: 0.7453 - val_loss: 0.4910 - val_accuracy: 0.7628
Epoch 6/25
625/625 [==============================] - 67s 107ms/step - loss: 0.5049 - accu
racy: 0.7466 - val_loss: 0.5041 - val_accuracy: 0.7572
Epoch 7/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4925 - accu
racy: 0.7548 - val_loss: 0.4846 - val_accuracy: 0.7628
Epoch 8/25
625/625 [==============================] - 67s 108ms/step - loss: 0.4889 - accu
racy: 0.7611 - val_loss: 0.4869 - val_accuracy: 0.7676
Epoch 9/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4860 - accu
racy: 0.7606 - val_loss: 0.5165 - val_accuracy: 0.7506
Epoch 10/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4798 - accu
Epoch 11/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4794 - accu
racy: 0.7636 - val_loss: 0.4827 - val_accuracy: 0.7696
Epoch 12/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4709 - accu
racy: 0.7701 - val_loss: 0.4784 - val_accuracy: 0.7686
Epoch 13/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4677 - accu
racy: 0.7713 - val_loss: 0.5009 - val_accuracy: 0.7630
Epoch 14/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4648 - accu
racy: 0.7733 - val_loss: 0.4746 - val_accuracy: 0.7684
Epoch 15/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4576 - accu
racy: 0.7768 - val_loss: 0.4922 - val_accuracy: 0.7710
Epoch 16/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4528 - accu
Epoch 17/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4477 - accu
racy: 0.7861 - val_loss: 0.4746 - val_accuracy: 0.7680
Epoch 18/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4454 - accu
racy: 0.7846 - val_loss: 0.5438 - val_accuracy: 0.7480
Epoch 19/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4422 - accu
racy: 0.7877 - val_loss: 0.5549 - val_accuracy: 0.7518
Epoch 20/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4372 - accu
racy: 0.7895 - val_loss: 0.4832 - val_accuracy: 0.7684
 • F*--------------------------------------------------------------------------------------

264 Al APPLICATION

Epoch 21/25
625/625 [==============================] - 68s 109ms/step - loss: 0.4358 - accu
racy: 0.7901 - val_loss: 0.6326 - val_accuracy: 0.7062
Epoch 22/25
625/625 [==============================] - 69s lllms/step - loss: 0.4342 - accu
racy: 0.7943 - val_loss: 0.4914 - val_accuracy: 0.7788
Epoch 23/25
625/625 [==============================] - 70s 112ms/step - loss: 0.4290 - accu
racy: 0.7976 - val_loss: 0.4847 - val_accuracy: 0.7762
Epoch 24/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4284 - accu
racy: 0.7960 - val_loss: 0.5695 - val_accuracy: 0.7428
Epoch 25/25
625/625 [==============================] - 67s 107ms/step - loss: 0.4225 - accu
racy: 0.7998 - val_loss: 0.5150 - val_accuracy: 0.7774

It may be a lot to ask you to analyze the whole

output but please give it a try. You'll see that
at each epoch how much time it took to take each
step, what was the loss, accuracy and total time
to complete that batch. And moreover you'll
understand the progress of the model
But if you're still not convinced, we can plot
the data and visualize them. Remember we stored
the training progress output in TrainP so let's
plot using those values

In [19]: # Loss PLot

pyplot.subplot(211)
pyplot.title('Cross Entropy Loss')
pyplot.plot(TrainP.history['loss'],
color='blue', label='train')
pyplot.plot(TrainP.history['val_loss'],
color='orange', label='test')
pyplot.show()
# Accuracy PLot
pyplot.subplot(212)
pyplot.title('Classification Accuracy')
pyplot.plot(TrainP.history['accuracy'],
color='blue', label='train')
pyplot.plot(TrainP.history['val_accuracy'],
color='orange', label='test')
pyplot.show()

We plotted the loss i.e. binary cross entropy

values and the accuracy values which we extracted
from the history attribute of TrainP
 r*------------
265 Al APPLICATION

You can see the first plot is the Cross Entropy

Loss and the second plot is the Classification
accuracy. The x-axis i.e. 0-25 represents the
number of epochs in both of the plots and the
y-axis is the cross entropy loss and accuracy for
both of the plots respectively
So let's analyze the first plot i.e. Cross
Entropy Loss. The losses(blue line {training}) are
reducing with each epoch uniformly but as for the
loss values(orange line {testing}) are
non-uniform. They were decreasing fast during the
first few epochs but at the end it shot-up and
became inconsistent; the same can be said for the
classification accuracy plot. The accuracy has
increased to 0.80 uniformly but accuracy(loss)
values are inconsistent
 • F*--------------------------------------------------------------------------------------

266 Al APPLICATION
• -J

So our deep learning computer vision based

classification model made with convolutional
neural networks is ready to make predictions! So
let's use the following test image of a dog and
pass it to our CModet

In [22]: img = kimg.load_img('Testimage.jpeg',

target_size = (64, 64))
img = kimg.img_to_array(img)
img = numpy.expand_dims(img> axis = 0)

pred = CModel.predict(img)

if pred[0][0] == 1:
pyplot.imshow(
image.imread('TestImage.jpeg'))
pyplot.axis(’off')
pyplot.title('Dog')

else:
pyplot.imshow(
image.imread('TestImage.jpeg'))
pyplot.axis(’off')
pyplot.title('Cat')

Dog

Before analyzing the above cell., I would like you

to go back to the first cell and look at the
dependencies that aren't used yet, this will help
you to understand what belongs to where
 267 Al APPLICATION
--- '•
The first step is to load our image that we did
with the Load_imag() function (from keras -
image aliased as kimg) where we specified the
target size also. Then we need to change the image
in the format of array to be read by the CModet
so we used the img_to_array() function. We also
need to expand the dimensions with axis zero.
Finally we passed the formatted image to the
CModei for classification
The classifier will return either 0 or 1 (0
represents the first class{first subfolder i.e.
Cat in this case} and 1 represents the second
class i.e. Dog)

◄ C] « Local Disk (C:) ♦ Users ♦ Rahul ♦ Petimages ♦ v O fl Search Petimages

# Quick access

Q Desl *
Downloads ✓
[Q Docum *
O Pictures * Cat Dog

rn Captcha^

□ Cat

rn d
n Music

We need to map them accordingly. Then we can plot

our image (imported with imreadO function) with
pyptot's imshowO and add the prediction as itJs
title. Note that we removed the axis using
axis('off') function
You can go ahead and snap some photos of your
pets (if you have) or use another photo to test
our CModei!
 INTELLIGENCE?

• Human or artificial
intelligence
• Datasets

o
 A

28 INTELLIGENCE?
i__________________________________________ )

I would like you to look back at the

date where you started learning with
this book and look how much new
knowledge you have gained until then.
I don't why you wanted to learn
about artificial intelligence maybe
for studies or just out of curosity
but what to do from now on? You
have learn't about different types
of Al systems, algorithms, machine
learning models, etc. but what you have concluded
till now? Al is profoundly used for automation of
tasks or replacing that of humans. I'm not talking
about and Science fiction where machines will become
more smarter than humans and we will become lazy but
just give it a thought. Programming, algorithms,
etc. everything is in it's place but how do even a
computer tell which image has what features? It
doesn't have any eye or vision like us. You may
argue that just in the previous Al application we
learned how the CNN algorithm works, but keep that
in mind that they are considered hidden. We don't
know what logic they used to arrive at a specefic
decision. Even though humans are the one's who
created these but there isn't a proper explanation
for that. Let's consider an example where a train at
a certain speed leaves a station and keeps the speed
uniform. Using mathematical formula's we can easily
calculate when will the train reach the nest
station. Similarly most of the Al is made up
mathematical functions, probabilities, etc. but how
they use it is up to them. Think about the day where
you first enrolled in school. You may not remember
but we were blank as empty pages but our teachers
started to teach us all sort of things like let's
take simple addition
 • ----------
270 INTEL
•

you were taught that a pair or more number can be

added i.e. called addition like 2 plus 3 equals 5.
It may have been hard at that time or from the
perspective of a kid. But know the formula's
you're dealing with are nothing to compare with.
But machines never learned those things, they jump
straight to the problems with different
mathematical approachs
A computer vision based classification model was
created with rich data of images with tumors which
took around a several hours or so. The model was
given some real life problems to solve and a human
doctor aged 60 was given the same problems. The
doctor solved the cases with 70%-75% of accuracy
whereas the classifier solved the problems with
92% of accuracy. The doctor with tens of years of
experience lose to the model created hours ago.
What do you say? You may think that the model
wasn't not the one actually solving the problems
it was trained by humans afterall, but my friend
the doctor wasn't a doctor from the begining he
was also taught by humans and learned as time
passed by. The doctor has full freedom to learn
all the same things as the model. They both has
their own intelligences, artificial and human. So
even if you can't admit it but the artificial
intelligence is far more better than the human
intelligence in this case
But don't conclude things! These models will not
perform always you want. There are many cases
where well trained models perform false prediction
We learned about face recognition in this book,
right? Many cases are seen where the face
recognizers have done false predictions which led
to a lot hassles. Recently all the phones are
equipped with face recongnition locks, but also
many were breaked by others because of false
 • ----------
271 INTEL
•

predictions. This rises to security problems and

these methods were left by many users but a lot of
people still use it because of the leniency. You
cannot tell the face reognizers that this is lake
becuase he has a mole here or his face's shape is
like these, they extract features on their own
which is mostly correct but it itself doesn't know
that it needs to recognize face or so. Eventh ough
we can tell that computer vision enables computers
to view the worlds as us but that pseudo-truth.
Now it's upto you to decide what to do. You may
not view this as a problem more or less even you
may think it's really helpful but

We aren't fast,, precise and accurate,

but the machines are.
machines aren't smart,
but we are.

But there is no saying that it machines cannot

become smart. Intelligence whether it is human or
artificial it has and never will see it's limits
but one will be left behind for sure
 DATASETS

All the datasets used in this book is available

in the following link (Google Drive Link):
http://bit.Ly/_DataSets
/------------------------------------------------ \
http://bit.ly/_DataSets

datasets y

I would like to hear your thoughts about the book

and what do you think about it. I'm not a regular
so will really feel good if you wrote a review or
tell me the flaws which I'll make sure to work on