Introduction

Machine Learning Course

P. Viswanath
IIITS
 Overview
• History of AI
– past, present, future.
• Techniques
– Supervised learning techniques
– Feature extraction
–
• Some Challenges
• The term artificial intelligence was first coined
in 1956, at the Dartmouth conference.
• Artificial Intelligence is still a growing active
field of science and technology.
– Has the potential to affect our lives
 AI?
• Homo sapiens (human beings) are able to
control (and exploit) other species and nature
because of their thinking capability.
 Turing Test

---------------------------------------------------------------------------
Human or Machine?
The Chinese Room Argument
Information Processing?
Topics in AI
 What we already achieved in AI?
• Board games – Chess, Checkers, Go, etc
• Solving Puzzles – Sudoku, etc
• Route finding in a map
• Image/speech enhancement
– Creating high resolution images, noise
suppression, …
•
 Machine Learning?
• ML is a subset of AI.
• We focus on mathematical/algorithmic
aspects of learning which can be programmed
over a machine.
• We look into various learning paradigms.
• Prerequisites of the course are
– Probability theory
– Linear algebra
– Calculus
 Evaluation/Examinations

• 20 Marks (16.6%) for Quizzes (best 2 out of 3 taken)

• 25 Marks (20.8%) for Assignments (3 assignments)
• 15 Marks (12.5%) for Mid1
• 25 Marks (20.8%) for Mid2
• 35 Marks (29.16%) for Endsem.
• Total – 120 Marks (100%).
 Text books/ References

1. “Pattern Classification” by R. O. Duda, P. E. Hart and D. G. Stork.

2. “An Introduction to Statistical Learning” by Gareth James, Daniela Witten,
Trevor Hastie and Robert Tibshirani.
3. “Pattern Recognition and Machine Learning” by Christopher M. Bishop.
4. “Introduction to Machine Learning” by Ethem Alpaydin.
5. “Pattern Recognition: An Algorithmic Approach” by M. Narasimha Murty,
V. Susheela Devi.
6. “Machine learning” by Tom Mitchell.
 Programming Assignments
• We follow C and or Python.
• No other languages are allowed.
• Plagiarism verification is done.
– Simply changing variable names may not work for
you?!
 Examinations/Quizzes
• Small numeric problems are asked.
• You need to employ the techniques discussed
in the class.
• Since, these are small numeric problems, just
by trial and error you can get the answer.
– But, this is not respected.
– You must follow the method described in the
class.
LET US BEGIN …
 Deductive Vs Inductive Learning
• Deductive
– Rules of the game are (hard coded) given ahead.
– Eg: An algorithm to do multiplication of numbers
is given. Given any two numbers you can apply
this and get the answer.
• Inductive
– We are given with examples (not the concept). We
need to learn the mapping from i/p to o/p.
• Supervised learning problems in AI comes under this
 Learning strategies
• Supervised
– Classification, Regression, …
• Unsupervised
– Clustering, density estimation, …
• Reinforced
– A robot navigating through obstacles, …
• Learn the good features (attributes)
– Feature extraction
Unsupervised “Semi” supervised Fully supervised
 What is a classification problem?
• Let there are two classes of objects.
– Class 1: Set of dog pictures
– Class 2: Set of cat pictures
• Problem is –
– Given a picture, you should say whether it is cat or
dog.
– For a human being it is easy…, but for a machine it
is a non-trivial problem.
 Training (Learning phase)

We have shown a set of dog pictures and a set of cat

pictures to a child.
 Testing phase

DOG

This picture as it is
may not be in the Child has done more than
training set just remembering
What is learning (pattern recognition)?
• Child has learnt what is it that is common
among dogs … and, what is it that is common
among cats… also, what are the distinguishing
features/attributes.
• Child has learnt the pattern (regularity) behind
all dogs and the pattern behind all cats.
• Child then recognized a test image as having a
particular pattern that is unique to dogs.
 Basic concepts
Object
 x1  Feature vector X  
  - A vector of observations (measurements).
x
 2   X
X    - X is a point in feature space  .
 
 xd 

Class to which X belongs is y Y

-Needs to be estimated, based on training set.

Task
- To design a classifer (decision rule) f :   Y
which decides about the class label based on X.
 An example
 is a set of persons
Object
Feature vector
h - A vector of observations (height, weight).
 
Person w

Class to which X belongs is y  overweight , normal 

-Needs to be estimated, based on training set.

Task
- To design a classifer (decision rule) f :   Y
- given height and weight of a person, classify him/her.
 Feature extraction
Task: to extract features which are good for classification.
Good features: • Objects from the same class have similar feature values.
• Objects from different classes have different values.

“Good” features “Bad” features

 Feature Space

Normal
persons

weight(w) Overweight
persons

height(h)

Training set is shown in the feature space

 Learning Steps
• Feature extraction: This is an important step.
Good features are needed.
– This is a lower level step. Normally done by
techniques like image processing, speech processing,
video processing, etc.
• Training set: Set of feature vectors along with
their class labels.
– An expert can see a few examples and give labels to
them based on his experience.
• Build the classifier by using the training set.
 Classification Problem
• Given a training set, build the classifier.

• One has to evaluate, how good is the built

classifier.
– Of course, it has to agree with the training set
• Is this 100% true?
– But, it should do more than this.
– The behavior of the classifier when it is asked to
classify some thing which is not in the training set
determines the quality.
 An easy, but bad classifier
• Remember the training set.
• See whether the given feature vector to be
classified is available in the training set.
• If yes, then return the label of that training
example.
• Else return a random class label.

• This is called Rote learning

 Classifiers
• There are many classification methods.
– Baye’s classifier, Naïve Bayes classifier
– HMM (graphical model)
– Artificial Neural Networks
– Decision Trees
– SVMs
– ….
 Generative vs. Discriminative Classifiers
Generative Models Discriminative Models
• Learn the source. May be • Learn to directly predict the
distribution is learnt. labels from the data
• Often, makes use of • Often, assume a simple
conditional independence boundary (e.g., linear)
and priors • Examples
• Examples – Logistic regression
– Naïve Bayes classifier – SVM
– Bayesian network – Boosted decision trees

• Can be used to generate • Doesn’t have the capacity to

new examples ! generate new examples.

Slide credit: D. Hoiem

k-Nearest Neighbor Classifier

The pattern to be classified

If k = 1 then the class assigned is

If k = 3 then the class assigned is
 Linear Classifier
Classifier:
If f(x1,x2) < 0 assign Class 1;
x2
If f(x1,x2) > 0 assign Class 2;
Class 2

f(x1,x2) = w1x1+w2x2+b = 0
Class 1

x1
 Perceptron
• Perceptron is the name given to the linear
classifier.
• If there exists a Perceptron that correctly
classifies all training examples, then we say
that the training set is linearly separable.
• In 1960s Rosenblatt gave an algorithm for
Perceptron learning for linearly separable
data.
 Perceptron
• For linearly separable data many classifiers are
possible.
All being doing equally good
on training set, which one is
Class 2
good on the unseen test set?

Class 1
 Maximizing the Margin SVM
Var1 IDEA : Select the
separating
hyperplane that
maximizes the
margin!

Margin
Width

Margin
Width
Var2
Artificial Neural Networks
 Generative Models
• Bayes
– Naïve Bayes
• Graphical models
– Belief networks
 Remember…
• No classifier is inherently better than any
other: you need to make assumptions to
generalize

• Two components of the error

– Bias: due to over-simplifications
– Variance: due to inability to perfectly estimate
parameters from limited data

Slide
Slide
credit:
credit:
D. D.
Hoiem
Hoiem
 Generalization

Training set (labels known) Test set (labels unknown)

• How well does a learned model generalize from

the data it was trained on to a new test set?
 Generalization
• Components of generalization error
– Bias: how much the average model over all training sets differ from the
true model?
• Error due to inaccurate assumptions/simplifications made by the
model
– Variance: how much models estimated from different training sets
differ from each other
• Underfitting: model is too “simple” to represent all the
relevant class characteristics
– High bias and low variance
– High training error and high test error
• Overfitting: model is too “complex” and fits irrelevant
characteristics (noise) in the data
– Low bias and high variance
– Low training error and high test error

Slide credit: L. Lazebnik

 Clustering Strategies
• K-means
– Iteratively re-assign points to the nearest cluster
center
• Agglomerative clustering
– Start with each point as its own cluster and iteratively
merge the closest clusters
• Mean-shift clustering
– Estimate modes of pdf
• Spectral clustering
– Split the nodes in a graph based on assigned links with
similarity weights
 Feature Extraction
• Principal Component Analysis
• Fisher Discriminant Analysis
•
 Representing Face Images: Eigenfaces
Q: How do we pick the set of basis faces?

A: We take a set of real training faces

Then we find (learn) a set of basis faces which best represent the differences
between them

That is, apply PCA and choose top Eigen vectors (Eigen faces)

We can then store each face as a set of weights for those basis faces
 Eigenfaces: the idea
• Think of a face as being a weighted combination of some “component” or
“basis” faces

• These basis faces are called eigenfaces

-8029 2900 1751 1445 4238 6193

…
 Eigenfaces: representing faces
• These basis faces can be differently weighted to represent any face

• So we can use different vectors of weights to represent different faces

-8029 -1183 2900 -2088 1751 -4336 1445 -669 4238 -4221 6193 10549
SOME CHALLENGES …
• King – man + woman = Queen
• Face – emotion + surprise = Surprised face

• Interpretable models.
THANK YOU