Chapter 1

Introduction to Machine Learning

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R, PESITM, Shivamogga
Need For Machine Learning
• Business organizations use huge amount of data for their daily
activities
• Earlier this data was not utilized at full potential
– Two reasons
• Data being scattered at different archives and not able to integrate
• Lack of software tools to unearth useful information

• Popularity of machine learning

– High volume of available data to manage :FB, X (twitter), YouTube.
Gets doubled/year
– Cost of storage has reduced: cheap h/w, easy to capture, process,
store, distribute and transmit
– Availability of complex algorithms

Sunilkumar H R, PESITM, Shivamogga

The knowledge pyramid
• Data : All facts are data. Text, audio,
image, video etc
• Information: processed data. Include
patterns, associations/relationships
among data.
– Eg. Sales data can be analyzed to extract
information like which is the fast moving/selling
product
• Knowledge : condensed information
– Historical patterns and future trends of the above
sales example is called knowledge
• Intelligence: applied knowledge for action
/an actionable form of knowledge
• Wisdom: that represents maturity of
mind. So far, exhibited by humans only !!!

Sunilkumar H R, PESITM, Shivamogga

Machine learning explained

• Machine learning is important sub branch of AI

• Arthur Samuel. Pioneers of AI, “ machine learning is the field of study that gives
the computers the ability to learn without being explicitly programmed ”
– System should learn by itself without explicit programming
• Conventional programming uses predefined rules or logic to solve a problem,
while machine learning uses data to construct a model or logic for the problem
• Conventional programming relies on programming logic, while machine learning
depends on real-life data-driven approaches that involve learning and modeling.
• Conventional programming is a manual and fixed process, where the programmer
designs the solution explicitly, while machine learning is an automatic and flexible
process, where the algorithm learns from data to generate a solution.

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R, PESITM, Shivamogga
• As humans take decisions based on an experience, computers make
models based on extracted patterns in the input data and then use these
data data-filled models for prediction and to take decisions.

Sunilkumar H R, PESITM, Shivamogga

 What is model?

• A model can any one of the following

– Mathematical equation
– Relational diagrams like trees/graphs
– Logical if/else rules
– Groupings called clusters
• A model can be a formula, procedure or representation that can generate
data decisions

Pattern Model
Local and only applicable to Global and fits the entire
certain attributes dataset

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R, PESITM, Shivamogga
Sunilkumar H R, PESITM, Shivamogga
Machine learning in relation to other fields
• ML uses the concepts of AI, Data Science and statistics primarily

Sunilkumar H R, PESITM, Shivamogga

ML and AI, data science, data mining, data analytics

• Data science is an umbrella that

encompasses many fields
– Big data
– Data mining
– Data analytics
– Pattern recognition

Sunilkumar H R, PESITM, Shivamogga

Big data

• Data science concerns about collection of data

• Big data is a field of data science
– Volume: huge amount of data is generated by big giants like FB, Twitter, You Tube
– Variety: data is available in different form; images, videos etc
– Velocity: it deals with the speed at which data is generated and processed
• Applications
– Language translation
– Image recognition

Sunilkumar H R, PESITM, Shivamogga

Data Mining

• Same as mining earth for precious metals

• Unearthing of data to recognize particular patterns hidden in the data
• This pattern is used for prediction using ML
• Tools like : Rapid Miner, Oracle data mining, IBM SPSS Modeler, Python, Knime,
Apache etc

Sunilkumar H R, PESITM, Shivamogga

Data Analytics

• Another branch of data science used to extract useful knowledge from

crude data
• Predictive data analytics is used for making predictions

Sunilkumar H R, PESITM, Shivamogga

Pattern recognition
• It is an engineering field
• It uses machine learning algorithms to extract the features for pattern analysis and
pattern classification

Sunilkumar H R, PESITM, Shivamogga

 Types of Machine learning

• Learning: like adaptation, occurs as the result of interaction of the

program with its environment

Sunilkumar H R, PESITM, Shivamogga

 Labeled and Unlabelled data

• As we know that data is raw fact!! Usually data is in the form of a table.
Some time it can be referred as a data point, sample or an example
• If table, each row represents a data point. Features are attributes or
characteristics of an object.
• Normally, the columns are attributes
• Out of all attributes, one attribute is important and is called label
• Label is the feature that we aim to predict
• That’s what two types of data
– Labeled data
– Unlabelled data

Sunilkumar H R, PESITM, Shivamogga

 Labeled data
• Example: let us take dataset called “Iris flower dataset” or “Fishers
dataset”. The data set has 50 samples of Iris- with four attributes, length
and width of sepals and petals. The target variable is called class. There
are three classes- Iris setosa, Iris virginica and Iris versicolor

Sunilkumar H R, PESITM, Shivamogga

 Unlabelled data
• Data that is not associated with labels
• They are classified with characteristics they poses

Sunilkumar H R, PESITM, Shivamogga

 Supervised learning

• It uses labeled data set. As name tells, it uses a supervisor or a teacher

• A supervisor provides labeled data so that the model is constructed and
generates test data
• Here learning takes place in two stages (Analogy)
– During the first stage, the teacher communicates the information to the student that the
student is supposed to master. The student receives the information and understands it.
During this stage the teacher has no knowledge whether the information is grasped by
the student
– In second stage of learning, the teacher asks the student a set of questions to find out
how much information has been grasped by the student. Based on these questions the
student is tested, and the teacher informs the student about his assessment.
THIS IS SUPERVISED LEARNING
• Supervised learning has two models
• Classification
• Regression

Sunilkumar H R, PESITM, Shivamogga

 Classification

Sunilkumar H R, PESITM, Shivamogga

 Classification (Contd..)
• The input attributes of the classification algorithms are called independent
variables
• The target attribute is called label or dependent variable
• The relationship between the input and target variable is represented in
the form of a structure  classification model
• The focus of classification is to predict the label that is in a discrete form
• From the figure, the classification algorithm takes a set of labeled data
images (dogs, cats) to construct a model that can later be used to classify
an unknown test image data
• The classification has two stages (learning)
– Training :learning algorithm takes labeled data set and starts learning. After the training
set, samples are processed and the model is generated
– Testing: the constructed model is tested with test or unknown sample and assign a label
This is the classification process

Sunilkumar H R, PESITM, Shivamogga

• The classification model can also be classified as
– Generative model
– Discriminative models
• Generative models: they deal with the process of data generation and its
distribution eg: probabilistic models
• Discriminative models: don’t care about generation of data, simply they
concentrate on classifying the given data

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R, PESITM, Shivamogga
 Regression Models

• Regression models, unlike classification algorithms, predict continuous

variables like price. (it is a number) eg: linear regression

Sunilkumar H R, PESITM, Shivamogga

 Unsupervised learning
• Self instruction, there are no supervisor or teacher component
• Concept of trial and error
• Here the program is supplied with objects but labels are not defined
• The algorithm itself observes the examples and recognizes patterns based
on the principals of grouping  similar objects form same group
• Eg: cluster analysis and dimensional reduction

Sunilkumar H R, PESITM, Shivamogga

 Cluster analysis
• It aims to group the objects into disjoint clusters or groups. Objects are
clustered based on the attributes
• Eg: finding region of interest in an image, detection of abnormal growth in
a medical image, determining clusters of signatures in a gene database

Sunilkumar H R, PESITM, Shivamogga

 Key algorithms

Sunilkumar H R, PESITM, Shivamogga

Semi supervised learning

Sunilkumar H R, PESITM, Shivamogga

Reinforcement Learning

Sunilkumar H R, PESITM, Shivamogga

 Challenges of Machine learning

• ILL-POSED PROBLEMS –PROBLEMS WHOSE SPECIFICATIONS ARE NOT

CLEAR : example
• HUGE DATA: data should be large and free from missing/incorrect data
• HUGE COMPUTATION POWER: GPU/TPU , Algorithms are complex & time
complexity has increased (TPU, AI accelerated ASIC by Google for NN)
• COMPLEXITY OF ALGORITHMS :
• BIAS-VARIANCE : error of the model. Over fitting and under fitting
– Over fitting: a model fits training data correctly but fails for test data, it lacks
generalization
– Under fitting : a model fails for training data, but has good generalization

Sunilkumar H R, PESITM, Shivamogga

 Machine learning process (CRISP-DM)
• It represents data mining solutions for business organization – CRISP-DM
• Machine learning is like data mining

Sunilkumar H R, PESITM, Shivamogga

Machine learning Applications

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R, PESITM, Shivamogga
• Summery

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R, PESITM, Shivamogga
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 Basics of Learning Theory

Predicting future is not a magic, it’s artificial intelligence –

Dave Waters

Sunilkumar H R, PESITM
 Preamble:

• Learning is a process by which one can acquire knowledge and construct

new ideas or concepts based on the experiences
• Machine learning is an intelligent way of learning general concept from
training examples without writing a program
• There are many machine learning algorithms through which computers
can intelligently learn from past data or experiences, identify patterns and
make predictions when new data is fed

Sunilkumar H R, PESITM
 Introduction to learning and its types
• Learning: the process of acquiring knowledge and expertise through study,
experience, or being taught
• To make machines learn, we need to simulate the strategies of human
learning in machines
• Will the computers learn?
• What sort of tasks can computers learn?
• This depends on the nature of problems that the computers can solve.
• Two kinds of problems
– Well posed
– Ill posed
• Computers can learn only well posed problems as they have well defined
specifications and have the following components
– Class of learning tasks (T)
– A measure of performance (P)
– A source of experience (E)

Sunilkumar H R, PESITM
• The standard definition of learning proposed by Tom Mitchel is that a program can
learn from E for the task T, and P improves with experience E.

Sunilkumar H R, PESITM
Learning Model=Hypothesis Set + Learning Algorithm

Sunilkumar H R, PESITM
 Classical and Adaptive Learning Systems
• A classical ML system has components such as Input, Process and Output
• The input values are taken from the environment directly. These values
are processed and a hypothesis is generated as output model. This model
used for making predictions. The predicted values are consumed by the
environment.
• Adaptive learning systems interact with the input for getting labeled data
as direct inputs are not available. It follows reinforcement learning (As
discussed in chapter 1)

Sunilkumar H R, PESITM
 Learning Types
• There are different types of learning
1. Learn by memorization
2. Learn by example
3. Learn by being taught
4. Learn by critical thinking
5. Self learning
6. Learning to solve problems
7. Learning by generalizing explanantions

Sunilkumar H R, PESITM
1. Learn by memorization or learn by repetition : Called as rote learning,
memorizing without understanding logic or concept
2. Learn by example / learn by experience: Previous knowledge acquired at
some time used to come conclusions
– The learner learns by inferring a general rule from the set of
observations
3. Learn by being taught : An expert or teacher will make him learn –
passive learning
– Interactively learn through queries – active learning
4. Learning by critical thinking : Also called deductive thinking
– Deduces new facts or conclusion from related known facts and
information
5. Self learning – reinforcement learning
6. Learning to solve problems : is a type of cognitive learning where learning
happens in the mind and is possible by devising a methodology to
achieve a goal
7. Learning by generalizing explanations : also called as explanation based
learning , that exploits domain knowledge from experts to improve the
accuracy of learned concepts by supervised
Sunilkumar H R, PESITM learning
Introduction to CONCEPT LEARNING

• Concept learning is a learning strategy of acquiring abstract knowledge or

inferring a general concept or deriving a category from the given training
samples. It’s a process of abstraction and generalization from the data
• Concept learning helps to classify an object that has a set of common,
relevant features. Thus it helps a learner compare and contrast categories
based on the similarity and association of positive and negative instances
in the training data to classify an object
• The learner tries to simplify by observing the common features from the
training samples and then apply this simplified model to future samples
• That’s what this task is also known as learning by experience

Sunilkumar H R, PESITM
• Each concept or category obtained by learning is a boolean valued
function which takes a true or false value
• For example, humans can identify different kind of animals based on
common relevant features (specific set of features)
• THIESE SPECIAL FEATURES THAT DISTINGUISH ONE ANIMAL FROM
ANOTHER CAN BE CALLED AS CONCEPT
• This way of learning categories for object and to recognize new instances
is called as concept learning ‘ it is formally defined as inferring a Boolean
valued function by processing training instances’
• Concept learning requires three things
– Input : training data set
– Output: target concept or target function f
– Test : new instance to test the learned model
• Another definition: “given a set of hypotheses, the learner
searches through the hypothesis space to identify the best
hypothesis that matches the target concept”
Sunilkumar H R, PESITM
SL Horns Tail Tusks Paws Fur Color Hooves Size Elephant
No
1 No Short Yes No No Black No Big Yes
2 Yes Short No No No Brown Yes Medium No
3 No Short Yes No No Black No Medium Yes
4 No Long No Yes Yes White No Medium No
5 No Short Yes Yes Yes Black No Big Yes

Independent attributes: Horns, tail, tusks, paws, fur, color, hooves, size
The dependent attribute: Elephant
The target concept is to identify the animal to be an Elephant.

Sunilkumar H R, PESITM
 Representation of hypothesis

• A hypothesis h approximates a target function ‘f’ to represent the

relationship between the independent attributes and the dependent
attribute of the training instances
• The hypothesis is the predicted approximate model that best maps the
inputs to outputs
• Each hypothesis is represented as a conjunction of attribute conditions in
the antecedent part
• For example (Tail=short) ^(color=black)…..
• The set of hypothesis in the search space is called as ‘Hypotheses’ and
represented as H
• In the antecedent of an attribute condition of a hypothesis, each attribute
can take value as either ‘?’ or ‘ ø ‘
• ‘?’ denotes that the attribute can take any value e.g. color=?
• ‘ ø ‘denotes that the attribute can not take any value

Sunilkumar H R, PESITM
• The most general hypothesis can allow any value for each of the attribute.
It can be written as <?,?,?,?,?,?,?>
• The most specific hypothesis will not allow any value for each of the
attribute. It can be represented as < ø ,ø, ø, ø, ø, ø, ø, ø >
• Hypothesis space: is the set of all possible hypotheses that approximates
the target function f
• From this set of hypotheses in the hypothesis space , a machine learning
algorithm would determine the best possible hypothesis that would best
describe the target function or best fit the outputs.
• The subset of hypothesis space that is consistent with all observed training
instances is called as Version space. Version space represents the only
hypothesis that are used for the classification

Sunilkumar H R, PESITM
Sunilkumar H R, PESITM
 Chapter 4

Similarity –based learning

“Any-one who stops learning is old, whether at twenty
or twenty ” – Henry Ford
• Similarity based learning is a supervised learning techniques that
predicts the class label of a test instance by gauging the similarity
of this test instance with training instances
• This technique can be used to solve both classification as well as
regression problems
• It is also called as instance based learning
• Instance-based learning makes prediction by computing distances
or similarities between test instance and specific set of training
instances local to the test instance in an incremental process
• It considers only the nearest instance or instances to predict the
class of unseen instances
• Application: image processing, text classification, pattern
recognition, bio informatics, data mining, information retrieval,
Sunilkumar H R PESITM
natural language processing etc

Sunilkumar H R, PESITM, Shivamogga

 Introduction to similarity or instance based learning
• Similarity based learning is also called as instance based
learning/just in time learning since it does not build an abstract
model of the training instances and performs lazy learning when
classifying a new instance
• This learning mechanism simply stores all data and uses it only
when it needs to classify an unseen instance
• Advantage of this method is, processing only occurs whenever
there is a request to classify a new instance. This methodology is
particularly useful when the whole data set is not available in the
beginning but collected in incremental manner
• The drawback of this learning is that it requires a large memory to
store the data since a global model is not constructed initially with
the training data
Sunilkumar
• Popular H Rmetrics
distance PESITMuse: Hamming distance, Euclidean
distance, Manhattan dist, Minkowski dist, Cosine Dist, Mahalaois
Dist, Pearson’s correlation, Jaccard Coefficient, Tanimoto
Coefficient

Sunilkumar H R, PESITM, Shivamogga

Differences between Instance and Model based Learning

Instance based learning Model Based learning

Lazy learners Eager Learners
Processing of training instances is Processing of training instances is done
done only during testing phase during training phase
No model is built with the training Generalizes a model with the training
instances before it receives a test instances before it receives a test
instance instances
Predicts the class of the test Predicts the class of the test instance
instance directly from the training from the model built
data
Slow in testing phase Fast in testing phase
LearnsSunilkumar H R PESITM
by making many local Learns by creating global
approximation approximation
Ex: K-NN and its variants Ex: DCNN its variants

Sunilkumar H R, PESITM, Shivamogga

 Nearest Neighbor Learning

• A natural approach to similarity –based classification is k-Nearest

neighbor (k-NN), which is a non parametric method used for both
classification and regression problems
• It is a simple and powerful non-parametric algorithm that predicts
the category of the test instance according to the ‘k’ training
samples which are closer to the test instance and classifies it to
that category which has the largest probability

Sunilkumar H R PESITM

Sunilkumar H R, PESITM, Shivamogga

• The algorithm relies on the assumption that similar objects are
close to each other in the feature space
• K-NN performs instance based learning which just stores the
training data instances and learning instances case by case.
• The model is also ‘memory based’ as it uses training data at time
when predictions need to be made.
• It is lazy learning algorithm as there is no prediction model is built
earlier
• The classification happens only after getting the test instance
• The algorithm classifies a new instance by determining the ‘k’
most similar instances (i.e. k nearest neighbors) and summarizing
the output of those ‘k’ instances
• If the target variable is discrete then it is a classification problem,
so it selects a most common class value among the ’k’ instances by
a majority voteH R PESITM
Sunilkumar
• However, if the target variable is continuous then it is a regression
problem and hence the mean output variable of the ‘k’ instances is
the output of the test instance.

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R PESITM

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R PESITM

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R PESITM

Sunilkumar H R, PESITM, Shivamogga

 How to choose the value of k for KNN Algorithm?
• The value of k is very crucial in the KNN algorithm to define the
number of neighbors in the algorithm. The value of k in the k-
nearest neighbors (k-NN) algorithm should be chosen based on the
input data. If the input data has more outliers or noise, a higher
value of k would be better. It is recommended to choose an odd
value for k to avoid ties in classification. Cross-validation methods
can help in selecting the best k value for the given dataset.
• k-NN classifier performance is strictly affected by three factors:
No. of nearest Neighbors (k), distance metrics and decision rule
• If k is small then it may result in over fitting/less stable or big,
then, it may include many irrelevant points from other classes
• It suits best forHlower
Sunilkumar dimensional data
R PESITM

Sunilkumar H R, PESITM, Shivamogga

 Weighted K-Nearest Neighbor algorithm
• Here k closest neighbors to the test instance are assigned a higher
weight in the decision as compared neighbors that are far away
from the test instance. The idea is that the weights are inversely
proportional to distances

Sunilkumar H R PESITM

Sunilkumar H R, PESITM, Shivamogga

Sunilkumar H R PESITM

Sunilkumar H R, PESITM, Shivamogga