Machine Learning Notes - Lec 01 - Basics of Machine Learning

Machine Learning
Lecture 01
Basics of Machine Learning
Dr. Rao Muhammad Adeel Nawab

How to Work
. ‫م‬
. ‫م‬
. ‫م‬ ‫ﷲ‬
! ‫ دت‬%$ & ' ‫*)ی‬+ ,- ‫ ﷲ‬: ‫آ‬
. / 0 ‫ د‬10 2 ' 34$ ‫اور‬ 2
Power of Dua
$? @A ?$ @A > )<= ; 891 ‫ز‬

8H I‫ر‬$ J ‫ ٔں‬F‫ اور ڈ‬B CD
8H I$ B Q$ O P‫ اور د‬KLM
Dr. Rao Muhammad Adeel Nawab 3
Dua – Take Help from Allah before starting any task

About ME
PhD - University of Sheffield, UK
Assistant Professor, COMSATS University Islamabad (CUI), Lahore Campus
(2012– to-date)
Leading NLP Research Group
Publications
More than 25 research publications include:
7 impact factor journal papers
19 international conference/workshop papers
20+ MPhil students supervised
110+ Citations
H-Index 7

About ME: Research Profile on Google Scholar

Course Details
Instructor
EMAIL: [email protected]
OFFICE: Room 04 @ Faculty Block
URL: https://lahore.comsats.edu.pk/Employees/74
Google Scholar Citations:

https://scholar.google.com/citations?user=fUn6DXYAAAAJ&hl=en

Course Details
Classes
Lectures 1 & 2
Tuesday (17:30 - 20:30), D-106 (D-Block)

Course Details
Google Class Room Code: fyq22oq
Join class room using your cuilahore email id
Assessment
Coursework: 25%
Midterm Examination: 25%
Final Examination: 50%
Office Hours
Email requests for appointment

Course Focus
Mainly get EXCELLENCE in two things
1. Become a great human being
2. Become a great Machine Learning Engineer
To become a great human being

Get sincere with yourself
When you get sincere with yourself your ‫ ﺧﻠﻭﺕ‬and ‫ ﺟﻠﻭﺕ‬is the
same
Aims
To introduce some of the main topics in machine learning,
especially automated concept learning or classification
from examples, but also learning association rules and
clustering
To develop skills in designing and building serious artificial
intelligence programs

Course Learning Outcomes (CLOs)
By the end of this course the students should
understand some of the main approaches/algorithms that are
used for representing concepts and learning them
automatically
be able to analyze datasets and transform them into
representations that will permit machine learning algorithms
learn concepts/discover patterns in them
be able to evaluate competing machine learning algorithms
over the same datasets

Pre-requisites
Good programming skills

Course Content – Key Topics
Concept Learning and General-to-Specific Ordering
Decision Tree Learning
Artificial Neural Network
Deep Learning
Evaluating Hypothesis
Bayesian Learning
Instance Based Learning
Clustering
Course Project Presentations
References
Primary Texts
T. Mitchell. Machine Learning. WCB/McGraw-Hill, Boston,
1997.
http://www.cs.cmu.edu/~tom/mlbook.html
I. H. Witten and E. Frank. Data Mining: Practical Machine
Learning Tools and Techniques with Java Implementations,
3rd Edition, Morgan Kaufmann, San Francisco, 2011.

References
Also useful
S. Russell and P. Norvig. Artificial Intelligence: A Modern
Approach (2nd ed), Pearson Education, 2003.
N. J. Nilsson. Introduction to Machine Learning, Draft of a
proposed textbook, 1997. Available
at:http://robotics.stanford.edu/people/nilsson/mlbook.ht
ml.

Journals and Conferences
Machine Learning (journal)
Journal of Machine Learning Research (journal)
Neural Computation (journal)
Journal of Intelligent Systems (journal)
International Conference on Machine Learning (ICML)
(conference)
Neural Information Processing Systems (NIPS)
(conference)

Machine Learning Toolkits
Python
scikit-learn (a.k.a. sk-learn)
http://scikit-learn.org/stable/
Documentation
http://scikit-learn.org/stable/documentation.html
Java
WEKA
https://www.cs.waikato.ac.nz/ml/weka/

Lecture Outline
What is Machine Learning
Varieties of Learning
Learning Input-Output Functions – General Settings
Constituents of Machine Learning
An Illustrative Example – Gender Identification
Reading:
Chapter 1 of Mitchell
Chapter 1 of Witten & Frank


Ultimate Goal!
To develop such a machine

which behaves like human.

Machine Learning
The study of how to
design computer
programs whose
performance at
some task improves
through experience.

Machine Learning
A computer program is said to be learn
from experience E
with respect to some class of tasks T and
performance measure P
if its performance at tasks in T as measured by P improves
with experience E

Machine Learning
Traditional Programming
Data
Computer Output
Program
Machine Learning
Data
Computer Program
Output
Types Of Learning
Deductive
Inductive DEDUCTIVE INDUCTIVE
Information Theory
Pattern Hypothesis
Tentative Hypothesis Observation
Theory Confirmation
Deductive Learning
Deductive learning works on existing facts and knowledge
and deduces new knowledge from the old.
Example: Throwing a ball in air
I know Newton's Laws. So I conclude that if I let a ball go, it
will certainly fall downwards.
Deductive learning does not generate "new" knowledge at all.
It makes the reasoning system more efficient.

Inductive Learning
Inductive learning takes examples and generalizes rather
than starting with existing knowledge.
Example: Throwing a ball in air
By experience, every time I have let a ball go, it falls
downwards. Therefore, I conclude that the next time I let a ball
go, it will also come down.
There is scope of error in inductive reasoning.
Several successful systems developed using this approach.

Disciplines Contributing To ML
Statistics
Artificial Intelligence
Biology
Cognitive Science
Information Theory
Philosophy
Control Theory
Computational Complexity
Why Study Machine Learning
Technological or Engineering motivation
To build computer systems that can improve their performance
at tasks with experience (data).
Massive growth in on-line data
Cognitive science motivation

To understand better how humans learn by modelling the
learning process.

Why Study Machine Learning?
Computer science motivation
To understand better properties of various algorithms for
function approximation.
How the data must be represented?
How much data they require?
How accurate they can be?
How to choose optimal data for training?

Some Areas Of Applications
Data mining
Medicine
Business
Agriculture
Computer games
Software applications
Personalized / self customizing program

Real World Examples of Machine Learning
Siri and Cortana Recommendation
Engine
Google Search

Real World Examples of Machine Learning
Spam Detection
Face Recognition
Sentiment Analysis
Gender
Identification

When Would We Use Machine Learning
When patterns exist in our data
Even if we don’t know what they are
When we cannot pin down the functional relationships
mathematically
Else we would just code up the algorithm
When we have lots of data
Labeled training sets harder to come by
Data is of high-dimension
High dimension “features”
For example, sensor data

Why Machine Learning is Hard
You See Your ML Algorithm Sees

Why Machine Learning is Hard
What is a “2”?

ML Summary – Learn from Data
Data = Model + Error

ML Summary – Learn from Data
Data = Model + Error
Data can Be: Build a Predictive Model Should be minimum

Structured,
Unstructured, Semi-
structured
Goals
Predict a model from data with minimum error
Predictive model would be efficient if error would be minimum
To minimize error, Data should be:
Unique/High quality Data
Large amount of data
Phases in Machine Learning
Analyze a training set of examples of correct behavior.
1. Training Store some representation of the newly learnt knowledge
(often some form of rules).
Rules are checked and, if necessary, additional training is

2. Validation given. Sometimes additional test data are used, but
instead of using a human to validate the rules, some other
automatic knowledge based component may be used.
The rules are used in responding to some new (unseen)

3. Application situations.
Learning Input-Ouput Functions –
General Settings

What is to be learned?
Function
Program.
Finite state machine.
Grammar.
Problem solving system.

Learning Input-Output Functions
Most of ML revolves around Learning Input-Output
Functions (or Concept Learning)
Concept Learning is a major subclass of inductive learning.
This takes examples of a concept and tries to build a general
description of the concept.
Very often, the examples are described using attribute-value
pairs.
Two very common inductive learning algorithms are
Version Spaces and ID3.

Learning Input-Output Functions: General Settings
Typically we are trying to learn a function f.
f is often referred to as the target function.
f takes a vector-valued input a n-tuple x = (x1, x2, . . .xn).
f itself may be vector-valued yielding a k-tuple as output.
Often f produces a single output value
E.g., k = 1 (in such cases the output is not thought of as a vector).

Job of the learner is to output a hypothesis h which is its
guess or approximation of the target function f.
h is assumed a priori to be drawn from a class of functions
H.
Note that f may or may not be in H and this may / may
not be known.

x1 Target function f y1
x2 y2
. .
X= Y or
. .
. .
xn yn
Training Example = {x1, …. xm} +f(xi)
for each xi ϵ TE
Learner h, where h ≈f
Hypothesis
Space h1
h2
hi hk

Representation of Learning Problem
Representation of Hypothesis
Will be discussed in next lecture
Representation of Data
Representation of Input
Attribute-value pair
Representation of Output (or Concept to be Learned)
Attribute-value pair
Data Types of Values
Categorical/Ordinal – Gender: Male/Female
Numeric – Age: 10, 15, 17, 21, 45 …..
Input
Single Value or
Vector Valued
Output
Mostly Single Valued
Instance or Example or Data Point or Observation
Input + Output
Instance is a “Vector of attribute values”
Learn from input to predict output

Input – Single Instance # Input Output
Output - Single 1 1 1
2 2 4
3 3 9
4 4 16
1 1
2 4
Input Output Here the concept to learn
3 9
4 16
is x2.

Input – multiple values Instance # Input Output
vector of “input attribute values” 1 [1,2,3] 1
2 [2,3,4] 5
Output - Single 3 [3,4,5] 11
4 [4,5,6] 19
[1, 2, 3] 1
[2, 3, 4] 5
Input
[3, 4, 5] Output Here the concept to
11
[4, 5, 6]
learn is: [a,b,c] -> a*c – b
19

Input – vector of “input attribute values” Output
Output – Single
Set of
Input
Vectors
What is
output
here??

Summary of Machine Learning Task
Machine Learning = Representation + Evaluation + Optimization
Representation Evaluation of Tweaking of

of Data and Learned Model Parameters
Hypothesis

Constituents of Machine Learning

Constituents of Machine Learning Task
1. Goal: Supervised/ Unsupervised/ Semi-supervised
2. Features/Attributes Selection: Selection of most
distinguishing features.
E.g., word positions, morphology, age, gender, etc.
3. Features Extraction: Values of features e.g., male, female, 62
4. Training Data: Subset of data used to train model (learning)
5. Test data: Subset of data to evaluate trained model
6. Evaluation measures: Accuracy, precision, recall, F-measure
7. Test of significance
Goal - Varieties of Learning
Supervised Learning (or Classification)
Unsupervised Learning (or Clustering)
Semi-supervised Learning


The learner selects h based on a set E of
training examples.
Each training example consists of:
A n-tuple x drawn from the set of input vectors
over which f is defined.
If for each training example x the value of f, i.e.
f(x), is also supplied then the learning setting is
called a supervised learning setting.

Set of
Output
Input
Vectors

Given a collection of records (training set)
Each record contains a set of attributes, one of the
attributes is the class.
Find a model for class attribute as a function of
the values of other attributes.
Goal: previously unseen records should be
assigned a class as accurately as possible.
A test set is used to determine the accuracy of the
model.
Usually, the given data set is divided into training
and test sets, with training set used to build the
model and test set used to validate it.
Supervised Learning (or Classification) - Applications
Direct Marketing
Reduce cost of mailing by targeting a set of consumers likely to
buy a new cell-phone product.
Fraud Detection
Predict fraudulent cases in credit card transactions.
Customer Attrition/Churn
To predict whether a customer is likely to be lost to a
competitor.


If the training examples do not have
associated output values, then the learning
setting is called unsupervised learning.
Useful for splitting datasets into partitions,
which is known as clustering.

Output
Set of
Input
Vector

Given a set of data points, each having a set of
attributes, and a similarity measure among
them, find clusters such that
Data points in one cluster are more similar to one
another.
Data points in separate clusters are less similar to
one another.
Similarity Measures:
Euclidean Distance if attributes are continuous.
Other Problem-specific Measures.

Unsupervised Learning (or Clustering) - Application
Market Segmentation:
Subdivide a market into distinct subsets of customers where
any subset may conceivably be selected as a market target to
be reached with a distinct marketing mix.
Document Clustering:
To find groups of documents that are similar to each other
based on the important terms appearing in them.

Semi-Supervised Learning
Semi-supervised learning combines both labeled and
unlabeled examples to generate an appropriate function
or classifier.

Semi-Supervised Learning
Set of Output
Input
Vector

Semi-supervised Learning: Application
Speech Analysis:
To analyze the speech.
Web Content Classification:
to classify any given webpage into one of several categories
(like "Educational", " Shopping", "Forum", etc.).

An Illustrative Example – Gender
Identification

An Illustrative Example
Learning Problem – Gender Identification

Human Learning vs Machine Learning
Human Difference of Representation Computer

Goal?
Supervised
Collect labelled/tagged/annotated data
High quality and large amount of data is needed
Unsupervised
Collect un-labelled/un-tagged/un-annotated data
Semi-supervised
Collect combination of labeled and un-labelled data

Gender Identification – Data Preparation
Input – Examples from which concept will be learned
Human
Can be represented using multiple attributes/features
Output – Concept to be Learned
Gender
Can be represented using a single attribute

Input Vectors
Input vectors go by a variety of names in the ML literature:
input vector, pattern vector, feature vector, sample, example,
instance.
Components xi of input vectors variously called:
features, attributes, input variables, components.
Values of components generally of two major sorts:
Numeric also called ordinal and dividing into real-valued and
discrete-valued numbers.
Nominal also called categorical, enumerated or (confusingly)
discrete.

Output
If a learner produces a hypothesis h that outputs a real-
number.
It is called function estimator.
Its output is called an output value or estimate.
If a learner produces a hypothesis h that outputs a categorical
value.
It is called classifier, recognizer or categorizer.
Its output is called a label, class, category or decision.
Outputs may also be vector-valued with the components being
numeric or nominal (or both).
Gender Identification – Data Preparation
Representation of Input
Attribute Value pair
What Attributes/Features?
Problem Dependent
Data Dependent
Write the names of features
What Values for Input Attributes/Features?
Depends upon the type of information it carries
Representation of Output
Attribute Value pair

Representation of Input and Output
Attributes/Features Object / Instance/ Example
Attribute Data Type Possible Values
Height (cm) Numeric Range: 160- 185
Weight (lb) Numeric Range: 110 - 220
Features/Attributes
Hair Length Categorical Bald/ Short/
Medium/ Long
Beard Categorical True / False
Scarf Categorical True / False
Gender Categorical Male / Female

Gender Identification – Data Collection
Attributes/Features have been selected to represent Inputs and
Output(s)
Next Step - Identify reliable (or good) sources of data for feature
extraction
Feature Extraction
“Values of attributes” can be obtained through
Observations – No. of hands, No. Eyes
Measurements – Height, Length of Hair
automatic or manual

Representation of Data as Attribute-Value Pairs

Input Vectors and Output
Label /
Instance =
Output of
Input+Output
Classifier
Set of Input
Vectors

Choosing a Model
Depends upon Goal - Discussed earlier
Supervised
Classification: Binary- or Multi-class
Regression
Unsupervised
Selecting Machine Learning Algorithm
Representation of Hypothesis
Number of features chosen to train model
Choosing Number of Parameters:
Training time, number of iterations, error tolerance,
etc.

Split Data for Learning and Evaluation
Split data into Training Set and Test Set
Random Sampling
2/3 of data is used for Training and 1/3 for Testing
Learning or Training
is carried out using Training Set
Evaluation or Validation
is carried out using Test Set

Split Data for Learning and Evaluation
Training
set
Whole
Data
Testing
set

Learning
Input to Learner
Training Data
Set of Hypothesis
Output of Learner
Hypothesis – which best fits the training data
Note that learned function (or hypothesis) is an approximation of the
Target Function to be learned
Learning
Training Example = {x1, …. xm} +f(xi) for each xi ϵ TE
Learner
Model
h1
Hypothesis Space
h2 h, where h ≈f
hi hk h best fits the
training data
Evaluation
Important to know how well our learner is doing, i.e. how
good the hypothesis it produces is?
Standard approach is to test hypothesis (or Trained Model) on
a set of inputs and function values (or output) not used in
training, referred to as the test set.
Common evaluation measures used are:
Mean-squared error and
Accuracy = proportion of instances correctly classified

Evaluation
Apply Learned/Trained Model on Test Data
How good you have learned?
Evaluate your Trained Model on some Test data (unseen
examples)

Evaluation
Trained
Model
Prediction
Actual Predicted Accuracy

179.1 185 Long Yes No Male Male 1
160.5 130 Short No No Female Male 0
177.8 160 Bald No No Male Female 0
161.1 100 Medium No No Female Female 1
Accuracy = Correctly Classified / Total instances = 2/ 4 = 0.50 or 50%

Tuning of Parameters
Depends on the Machine Learning algorithm

ML Assumption
If the trained model performs well on large test

data, it will perform well on real-time data.

Prediction or Application Phase
Training
set
Learned
Model
Unseen Prediction
Example Extract Features as in Training set
How To Become a Great Human
Being

Balanced Life is Ideal Life
Get Excellence in five things
1. Health
2. Spirituality
3. Work
4. Friend
5. Family
A Journey from BIGNNER to EXCELLENCE
You must have a combination of five things with different
variations. However, aggregate will be same.
Excellence
1. Health
I can run (or brisk walk) 5 kilometers in one go
I take 7-9 hours sleep per night (TIP: Go to bed at 10pm)
I take 3 meals of balanced diet daily
2. Spirituality
ٔ ٔ T
. W ! I$ V ‫ م‬U . S ‫ ا‬J . R ; P‫د‬

Excellence
1. Work
Become an authority in your field
For example - Dr. Abdul Qadeer Khan Sb is an authority in research
2. Friend
Have a DADDU YAR in life to drain out on daily basis
3. Family
1. Take Duas of Parents and elders by doing their ‫ ﺧﺩﻣﺕ‬and ‫ﺍﺩﺏ‬
2. Your wife/husband should be your best friend
3. Be humble and kind to kids, subordinates and poor people
To Achieve Excellence - Have a Learning and
Progressive Attitude
. \ % R 1 $ ‫ د‬% R 1 $ Z[ XY > 891 ‫ ز‬-
. \ dR1 $ ‫ چ‬c 8a ‫`_ ا‬$ \ % R1 $ _ ^‫ آد‬-

To Achieve Excellence - Have a Learning and
Progressive Attitude
\ n p ‫ی اور‬oJ U _n 8m $ ، \ ‫ چ‬c ‫ ری‬efg U djk hi‫ ری ا‬efg

(rs ‫ رو^ ر‬tu1v‫ل ا‬xI$ )
3„$ … 0
. †v . ƒ h ‫ آپ‬. • ‫ آد^ ا‬r€ \ r• ‫ @~ ری‬$ ‫|ی‬$} 2 _n 8{ ‫رو ح‬
(rs ‫ رو^ ر‬tu1v‫ل ا‬xI$ )

Have a BIG Goal in Life :
. ‫|ے آپ د‬$} . ‫ا‬، \ ‫ چ‬c 8{ ‫|ی آپ‬$} d ‡ˆ$

Goal 01:
\ % ‡Š o F A > ‫> ! رس‬
Goal 02: ٔ Žv
ٔ 8 H • ? ‫)ی‬ *• 1•$ ‘ ‫ل‬ ‫رہ‬1% • 2 ‫آج‬
89 ‫ ۓ‬I$ 8H •|}• ‫ ب‬% A 8{ Machine Learning
Have a BIG Goal in Life :
Goal must be bigger than your life

My Goal in Life
Get Excellence in following five things
To establish such educational institutes in the whole world
whose students have excellence in five things
1. Health
2. Spirituality
3. Work
4. Friend
5. Family

Power of Visualization
Everything that exists now, was an imagination of
someone
Examples – phone, bulb, aero plane etc.
A positive thought is thousand times stronger than a
negative thought
Keep visualizing that you will succeed and you will
succeed one day

Secret of Achieving BIG Goals
Two things will change your life and give you energy to
achieve BIG goals
1. Books you read
2. From which people you take inspiration and with what
type of people you spend most of your time
For example - Who is your GURU

Machine Learning Notes - Lec 01 - Basics of Machine Learning

Uploaded by

Copyright:

Available Formats

Machine Learning Notes - Lec 01 - Basics of Machine Learning

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Machine Learning Notes - Lec 01 - Basics of Machine Learning

Uploaded by

Copyright:

Available Formats

What are the main topics that will be covered in the course?

What are the main topics that will be covered in the course?

What are the aims of the course?

What are the aims of the course?

Machine Learning

Dr. Rao Muhammad Adeel Nawab

$? @A ?$ @A > )<= ; 891 ‫ز‬

Dr. Rao Muhammad Adeel Nawab 4

Dr. Rao Muhammad Adeel Nawab 5

Dr. Rao Muhammad Adeel Nawab 6

Google Scholar Citations:

Dr. Rao Muhammad Adeel Nawab 7

Tuesday (17:30 - 20:30), D-106 (D-Block)

Dr. Rao Muhammad Adeel Nawab 8

Dr. Rao Muhammad Adeel Nawab 9

To become a great human being

Dr. Rao Muhammad Adeel Nawab 11

Dr. Rao Muhammad Adeel Nawab 12

Dr. Rao Muhammad Adeel Nawab 13

Dr. Rao Muhammad Adeel Nawab 15

Dr. Rao Muhammad Adeel Nawab 16

Dr. Rao Muhammad Adeel Nawab 17

Dr. Rao Muhammad Adeel Nawab 18

Dr. Rao Muhammad Adeel Nawab 19

Dr. Rao Muhammad Adeel Nawab 20

To develop such a machine

Dr. Rao Muhammad Adeel Nawab 21

Dr. Rao Muhammad Adeel Nawab 22

Dr. Rao Muhammad Adeel Nawab 23

Tentative Hypothesis Observation

Dr. Rao Muhammad Adeel Nawab 26

Dr. Rao Muhammad Adeel Nawab 27

Cognitive science motivation

Dr. Rao Muhammad Adeel Nawab 29

Dr. Rao Muhammad Adeel Nawab 30

Dr. Rao Muhammad Adeel Nawab 31

Dr. Rao Muhammad Adeel Nawab 32

Dr. Rao Muhammad Adeel Nawab 33

Dr. Rao Muhammad Adeel Nawab 34

Dr. Rao Muhammad Adeel Nawab 35

Dr. Rao Muhammad Adeel Nawab 36

Data = Model + Error

Dr. Rao Muhammad Adeel Nawab 37

Data can Be: Build a Predictive Model Should be minimum

Rules are checked and, if necessary, additional training is

The rules are used in responding to some new (unseen)

Dr. Rao Muhammad Adeel Nawab 40

Dr. Rao Muhammad Adeel Nawab 41

Dr. Rao Muhammad Adeel Nawab 42

Dr. Rao Muhammad Adeel Nawab 43

Dr. Rao Muhammad Adeel Nawab 44

Dr. Rao Muhammad Adeel Nawab 45

Learn from input to predict output

Dr. Rao Muhammad Adeel Nawab 48

Dr. Rao Muhammad Adeel Nawab 49

Dr. Rao Muhammad Adeel Nawab 50

Dr. Rao Muhammad Adeel Nawab 51

Machine Learning = Representation + Evaluation + Optimization

Representation Evaluation of Tweaking of

Dr. Rao Muhammad Adeel Nawab 52