ENG6500: Intro to Machine Learning

“ML: Overview (Part 1)”

Prof. S. Areibi
School of Engineering
University of Guelph
Learning
 What is Learning?
• “The acquisition of knowledge or skills through experience,
study, and the capability to generalize”

• “Learning is the process of acquiring new understanding,

knowledge, behavior, skills, attitudes, and preferences”
– Wikipedia.

• “Learning denotes changes in a system that ... enable a

system to do the same task … more efficiently the next
time.” - Herbert Simon

• “Learning is constructing or modifying representations of

what is being experienced.” - Ryszard Michalski

How do we learn? 3
 How do we Learn?
• By telling us exactly what to do
Writing steps to follow (i.e., a flowchart or program)

• By showing us multiple examples

We are “forced” to learn (this becomes our knowledge)

• By observation (real-world experience, trial and errors)

In case examples are not enough, not clear and/or there
is no expert.

• By being rewarded or punished to reinforce concepts.

What are the different techniques for learning?

4
 Learning Techniques
 Memorization
o Accumulation of individual facts
o Limited by:
• Time to observe and preserve facts
• Memory Capacity to store facts

 Generalization:
o Deduce new facts from old facts
o Essentially a predictive activity
o Assumes that the past predicts the future.
o Limited by:
• Accuracy of deduction process

Learning and memory are closely related concepts. Learning is the acquisition
of skill or knowledge, while memory is the expression of what you've acquired.
5
AI: Machine/Deep
Learning
Artificial Intelligence
• Artificial Intelligence
means getting a
computer to mimic
human behavior in
some way (Speech
recognition, Image
Classification, …) .. Rule
Based
• Machine learning is a
subset of AI and pretty
old field. It incorporates
algorithms that have
been around and new
algorithms … Data
Driven
7
 Tom Mitchell: ML
Definition
• Tom Mitchell (1988) well posed Learning Problem:
• A computer program is said to learn from
Experience (E) with respect to some Task (T) and
some Performance measure (P), if its performance
on T, as measured by P, improves with experience E.

A well-posed learning problem <P, T, E>

Improve performance P
at task T
with experience E

9
Defining the Learning Task
1. Categorizing Email Spam:
• T: Categorize email messages as spam or legitimate
• P: Percentage of email messages correctly classified
• E: Database of emails, some with human-given labels
2. Recognizing Words:
• T: Recognizing hand-written words
• P: Percentage of words correctly classified
• E: Database of human-labeled images of handwritten words
3. Checkers:
• T: Playing Checkers
• P: Percentage of games won against an arbitrary opponent
• E: Playing practice games against itself
4. Driving:
• T: Driving four-lane highways using vision sensors
• P: Average distance traveled before a human-judged error
• E: A sequence of images and steering commands recorded while observing a
human 10
 Related Fields
data
mining control theory

statistics
decision theory
information theory machine
learning
cognitive science
databases
psychological models
evolutionary neuroscience
models

Machine learning is primarily concerned with the

accuracy and effectiveness of the computer system.

11
Machine Learning:
Applications
 Machine Learning
Applications
• Retail: Market basket analysis, Customer relationship
management (CRM)
• Finance: Credit scoring, fraud detection
• Manufacturing: Control, robotics, troubleshooting
• Medicine: Medical diagnosis, correct image classification
• Telecommunications: Spam filters, intrusion detection
• Web mining: Search engines
• Autonomous Driving: Vehicles, Trucks, Drones, …
• Electronic Design Automation: Synthesis, Placement, …
• Chip Fabrication, … improving yield
• IOT (Internet of Things) … Wireless Sensor Networks
13
 Web Page Search
It is Everywhere

Web Page Search

14
 Email Anti-Spam
It is Everywhere

Email Anti-Spam

15
Character Recognition
It is Everywhere

Character Recognition

16
 Voice Recognition
It is Everywhere

Voice Recognition

Google Voice Recognition Now Supports 119 Languages

17
 Face Recognition
It is Everywhere

Face Recognition and tagging

18
From Deep Blue to
AlphaGo
1998 IBM Deep Blue

Now

2016

Alpha Go developed by DeepMind Technologies

which was later acquired by Google
19
Machine Learning
Machine Learning is…
Machine learning is a field of computer science that gives
computers the ability to learn without being explicitly programmed

Machine learning, is a branch of Artificial Intelligence

(AI), concerns the construction and study of systems
that can learn from data.

• Automatically detect patterns

in data
• Use the uncovered patterns to
predict future data or other
outcomes of interest
21
 ML (PREDICTION)

Machine learning is about predicting the future

based on the past.
-- Hal Daume III

past future
c t
n d i
a r e
l e pr
Training model/ Testing model/
Data predictor Data predictor

22
 Machine Learning
Machine learning is a field of computer science that gives
computers the ability to learn without being explicitly
programmed

? Machine Learning
Labeled Data algorithm

Training
Prediction

Learned
Labeled Data Prediction
model

Unseen Data
Hyper Parameter Tuning

Methods that can learn from and make predictions on data

23
 Machine Learning
• Machine Learning is a diverse field covering a wide territory and
has impacted many verticals.
• It is able to tackle tasks in language and image processing,
anomaly detection, credit scoring sentiment analysis, forecasting
alongside dozens of other downstream tasks.
• A proficient developer, in this line of work; has to be able to
draw, & borrow, from many adjacent fields such as
mathematics, statistics, programming, and most importantly
common sense.
• It turns out that developing and training a model only takes a
small fraction of the project duration.
• The bulk of the time and resources are spent on data
acquisition, preparation, hyperparameter tuning, optimization,
using the correct evaluation criteria, and model deployment.
24
 Types of ML Classes
Supervised: Learning with a labeled training set
Example: classify email to be either Spam or Ham

Unsupervised: Discover patterns in unlabeled data

Example: cluster similar documents based on text

Reinforcement learning: learn to act based on feedback/reward

Example: learn to play Go, or Chess, reward: win or lose

class B

Clustering
class A

Regression
Classification

Anomaly Detection
Sequence labeling
…
25
Data
 Data Objects
 Data sets are made up of data objects.
 A data object represents an entity.
 Rows represent:
o Medical records: patients, treatments
o University records: students, professors, courses
 Also called samples , examples, instances, data
points, objects, tuples.
 Data objects are described by attributes.
 Data: Features
Object #1
o Rows -> data objects;
Object #2
o Columns ->attributes. Object …
Object #n
27
 Attributes (Features)
 Whenever you perform classification, the first
step is to understand the problem and identify
potential features and label.
 Features are those “characteristics” which
affect the results obtained by the ML model.
 Features are also known as “Attributes” which
help the model during
classification/regression/clustering.
 For example, in the case of a loan distribution,
bank manager's identify customer’s
occupation, income, age, location, previous
loan history, transaction history, and credit
score. 28
 Records Features
& Features
Label
Weather data example: Play or not to Play?

ID code Outloo Temperature Humidity Windy Play

k
a Sunny Hot High False No
Record
b Sunny Hot High True No
c Overcast Hot High False Yes
Training Data

d Rainy Mild High False Yes

e Rainy Cool Normal False Yes
f Rainy Cool Normal True No
g Overcast Cool Normal True Yes
h Sunny Mild High False No
i Sunny Cool Normal False Yes
j Rainy Mild Normal False Yes
k Sunny Mild Normal True Yes
Testing Data

l Overcast Mild High True Yes

m Overcast Hot Normal False Yes
n Rainy Mild High True No
29
Data Preprocessing
Why Data Preprocessing?
• Data in the real world is dirty
– incomplete: lacking attribute values, lacking certain attributes of
interest, or containing only aggregate data
– noisy: containing errors or outliers
– inconsistent: containing discrepancies in codes or names
• No quality data, no quality mining results!
– Quality decisions must be based on quality data
– Data warehouse needs consistent integration of quality data

Cleaning, preprocessing, preparing

Data is an important task to be able to
Develop effective ML frameworks
31
 Data Reduction Strategies
 Data reduction: Obtain a reduced representation of the data
set that is much smaller in volume but yet produces the same
(or almost the same) analytical results
 Why data reduction? —
 A database/data warehouse may store terabytes of data.

Complex data analysis may take a very long time to run on

the complete data set.
 Data reduction strategies
 Dimensionality reduction, e.g., remove unimportant

attributes

Wavelet transforms

Principal Components Analysis (PCA)

Feature subset selection, feature creation

32
Why/When Machine
Learning?
Machine Learning vs. Traditional Programming

Machine learning teaches computers to do what comes naturally

to humans: learn from given examples and experience.

Answer: Prediction

Label

34
 Where ML works best?
• Complex problems involving a large amount of data with
lots of variables, but no existing formula or equation that
describes the system.
• Some common scenarios where machine learning applies
1. Systems are too complex for handwritten rules
a) Face and speech recognition
b) FPGA routability prediction.
c) ASIC congestion estimation.
2. Rules of a task are constantly changing
a) Fraud detection
b) Email Spam
3. Nature of the data itself keeps changing
a) Automated trading, predicting shopping trends,
b) Mutations in viruses
35
Why are things working
today?

• More compute power

Better

• More data

Accuracy
• Better algorithms
/models

• Applications and Industry

Amount of Training Data

36
Machine Learning Methods
 Machine Learning
Approaches

Machine
Learning

Semi- Reinforcement
Supervised Unsupervised
supervised Learning
. .

Machine learning provides systems with the ability to learn

without being explicitly programmed.

38
Machine Learning
Approaches

39
Regression
 Regression y

Y1
 Regression analysis: A collective
name for techniques for the modeling
Y1’
and analysis of numerical data y=x+1
consisting of values of a dependent
variable (also called response
X1 x
variable or measurement) and of
one or more independent variables
(aka. explanatory variables or
 Used for prediction
(including forecasting of
predictors)
time-series data),
 The parameters are estimated so as inference, hypothesis
to give a "best fit" of the data testing, and modeling of
 Most commonly the best fit is causal relationships
evaluated by using the least squares
method, but other criteria have also 41
 Regression Example
The best fit line is determined by varying the values of w
and w0.

Price of a used car

y = wx+w0
x : car attributes
(e.g. mileage)
y : price

Other Applications?

The values of w and w0 get selected in such a way to produce the

minimum predictor error. 42
rmining the slope, intercept of the line is an Optimization Problem
42
 Stock Market

Predict the price of a stock given historical data

43
Weather Prediction

Temperature

Predict the temperature given historical data

44
 Estimating FPGA
Congestion
• Congestion occurs when the
demand for routing resources
exceeds the supply in some
region of a design

3
2 connections
channels available

Excessive congestion causes subsequent routing stage to fail

45
 Congestion
Training Extract
congestion map
with TCL script label

Xilinx Detailed
Router
372 FPGA Placement
Ultrascale placer file results
Benchmarks Congestion
features Filtering
• NCPR
• WLPA
• etc Training

Testing
Congestion
New features
Circui
FPGA Placement • NCPR
file results Prediction Model
t placer • WLPA
• etc

Deployment Predicted
Congestion of New
Circuit

D. Maarouf, A. Al-hyari, Z. Abouwaimer, T. Martin, A. Gunter, G. Grewal, S. Areibi, A. Vannelli

“A Machine Learning Based Congestion Estimation for Modern FPGAs”,
International Conference on Field Programmable Logic & Applications (FPL 2018), Ireland, pp. 427-434
46
Classification
Supervised Learning: Classification
• The classification has two phases, a training
(learning) phase, and the testing (evaluation) phase.
• In the training phase, classifier trains its model on a
given dataset.
• The model is developed during the training phase.
o Development of the model means that several
parameters are adjusted to predict a value
(weights in ANN).
• In the evaluation phase, it tests the classifier
performance.Training
• Performance is evaluated on the basis of various
parameters such as accuracy, error, precision,
Which metric and
should be used?
recall.

Testing 48
Classification Example 1
Day Outlook Temperature Humidity Wind Play Tennis
1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
3 Overcast Hot High Weak Yes
4 Rain Mild High Weak Yes
5 Rain Cool Normal Weak Yes
6 Rain Cool Normal Strong No
7 Overcast Cool Normal Strong Yes
8 Sunny Mild High Weak No
9 Sunny Cool Normal Weak Yes
10 Rain Mild Normal Weak Yes
11 Sunny Mild Normal Strong Yes
12 Overcast Mild High Strong Yes
13 Overcast Hot Normal Weak Yes
14 Rain Mild High Strong No

Outlook
Label
Sunny Overcast Rain

Humidity Yes Wind Decision Tree Algorithm

High Normal Strong Weak
No Yes No Yes

49
Classification Example 2
Machine Learning Algorithm
Should be able to create this
Partition or border to distinguish
Differentiate Between low risk and high risk
between low-risk
and high-risk
customers from
their income and
savings

Income and
Savings are the
features within
the data used to
train the ML Model

50
Classification Example 3
Classify a car to be
either a family car or
not!
Each car has two

X2: Engine Power

features to be used
as input to the
classifier: +
+ +
+ +
 Price

 Engine power + +
These would be the +
labels for each car
X1: Price
1 if x is family class: Positive Example
r=
0 if x is not a family class: Negative Example

51
Unsupervised Learning
Unsupervised Learning
• Draws inferences from datasets that don’t have labeled
responses associated with the input data.

• Clustering is the most common unsupervised learning technique.

It puts data into different groups based on shared characteristics
in the data.

• Clustering is used to find hidden groupings in applications such as:

 Gene Sequence Analysis,
 Market Research, and
 Electronic Design Automation
 Object Recognition among many others.

53
Unsupervised learning
Create Clusters (groups) of similar features

Unsupervised learning: given data, i.e. examples, but no labels

We seek to categorize a new object to a specific group

54
Clustering Data: Group similar things

55
Article Clustering

Apply ML to classify documents (Clustering)

56
Reinforcement Learning
 RL is Learning from
Interaction
 Reinforcement learning
 An agent interacts with the environment:
o Makes sensor observations
o Must select action
o Receives rewards
• positive for “good” states
• negative for “bad” states

Applications:
• Robotics
Towel Folding • Playing chess
http://youtu.be/gy5g33S0Gzo
58
Summary
 Summary
o Over the past decade, artificial intelligence (AI) has become a
popular subject both within and outside of the scientific community.
o AI is a field focused on automating intellectual tasks normally
performed by humans, and ML and DL are specific methods of
achieving this goal.
o Machine Learning is a diverse field covering a wide territory (stats,
neuro science, …) and has impacted many verticals.
o It is able to tackle tasks in language and image processing, anomaly
detection, credit scoring sentiment analysis, forecasting alongside
dozens of other downstream tasks.
o Machine learning tasks are typically classified into three broad
categories, depending on the nature of the learning "signal" or
"feedback" available to a learning system. These are: (a) Supervised,
(b) Unsupervised, (c) Reinforcement Learning
60
Resources
ML Intro: Misc. Resources
o YouTube
• https://www.youtube.com/watch?v=KNAWp2S3w94
• https://www.youtube.com/watch?v=ukzFI9rgwfU
• https://www.youtube.com/watch?v=XvZsiWj6VuU
• https://www.youtube.com/watch?v=h0e2HAPTGF4
• https://www.youtube.com/watch?v=Gv9_4yMHFhI
• https://www.youtube.com/watch?v=IpGxLWOIZy4
• https://www.youtube.com/watch?v=f_uwKZIAeM0

o Courses and mini courses:

• https://www.learndatasci.com/best-machine-learning-courses/
• https://www.coursera.org/learn/machine-learning
• https://www.edx.org/learn/machine-learning
• https://developers.google.com/machine-learning/crash-course/
• https://www.udemy.com/topic/machine-learning/
• https://www.youtube.com/watch?v=9f-GarcDY58
• https://www.youtube.com/watch?v=79ry4-FdrdM
• https://www.tensorflow.org/resources/learn-ml?gclid=CjwKCAjw6dmSBhBkEiwA_W-EoOrWHh10
oPeIdMF2LXISTAMxeZ7dWIqcdbp-Uka9Jw2FdoSS7Xh8TxoCnUsQAvD_BwE
62
ML Intro: Misc. Resources
o Documents:
• https://www.simplilearn.com/data-mining-vs-machine-learning-article#:~:text=Data%20mining%20is%2
0designed%20to,total%20of%20the%20gathered%20data
.
• https://www.seas.upenn.edu/~cis519/fall2017/lectures/01_introduction.pdf

o Tutorials:
• https://www.guru99.com/machine-learning-tutorial.html
• https://www.toptal.com/machine-learning/machine-learning-theory-an-introductory-primer
• https://www.geeksforgeeks.org/getting-started-machine-learning/?ref=leftbar-rightbar
• https://intellipaat.com/blog/supervised-learning-vs-unsupervised-learning-vs-reinforcement-learning/
• https://www.geeksforgeeks.org/what-is-reinforcement
-learning/#:~:text=Reinforcement%20learning%20differs%20from%20supervised,to%20perform%20the
%20given%20task
.

o Books:
• A Course on Machine Learning  http://ciml.info/
• Deep Learning  http://www.deeplearningbook.org

63