3/11/2025

“Intelligence is the art of good guesswork.”

-H.B. Barlow, The Oxford Companion to the Mind

CSE655
Deep Learning
Introduction
Spring 2025
Introduction to Machine Learning Most of the following slides are adapted E. Alpaydin
© 2016-2025 Yakup Genc & Y. Sinan Akgul

1 2

Why Machine Learning? What is Machine Learning

• Most of the real-world problems are: • … a computer program that can learn from experience
– NP-Hard with respect to some class of tasks and performance
measure …” – Mitchell 1997
• scene matching, big data problems …
• “ … ML is the science of getting computers to learn,
– Ill-defined without explicitly programmed…” –Ng
• 3D reconstruction from a single image, missing data
• ML is more than just memorizing facts:
– The right answer is subjective – learning the underlying structure of the problem or data
• Image segmentation, language, …
• Also known as:
– Hard to model – Regression
• scene classification, customer behavior
– Pattern recognition
• Machine Learning tries to use statistical reasoning to find approximate – Data mining
solutions for tackling the above difficulties – Automated programming

Joshi et al Dönnes
March 2025 Deep Learning 3 March 2025 Deep Learning 4

3 4
1
 3/11/2025

Black Box Model of ML Supervised Machine Learning

Test Data
# of experiences observed outcome observed input
Training
Data Magic Black Box
Model
(Learning Machine)

arg max 𝑔𝑜𝑜𝑑𝑛𝑒𝑠𝑠( 𝑦 − 𝑓(𝑥 ))

Result ∈
Apple

Training data: Sample images of the class of objects to be learned hypothesis/program space performance measure selected hypothesis
Model: The model can identify any koala images...
Dönnes
March 2025 Deep Learning 5 March 2025 Deep Learning 6

5 6

Learning Algorithms Why “Learn”?

• Supervised Learning • Machine learning is programming computers to optimize a performance

– Generative/Discriminative models criterion using example data or past experience.
– ANN/Boosting/Decision Tree/NNA/Random Forests • There is no need to “learn” to calculate payroll
• Unsupervised Learning • Learning is used when:
– Clustering – Human expertise does not exist (navigating on Mars),
– Humans are unable to explain their expertise (speech recognition)
– K-Means/Dirichlet/Gaussian Processes/EM
– Solution changes in time (routing on a computer network)
• Semi-Supervised Learning – Solution needs to be adapted to particular cases (user biometrics)
– Constrained Clustering/Distance Metric Learning/Manifold based
Learning/Compressed Sensing/Active Learning

Joshi et al
March 2025 Deep Learning 7 March 2025 Deep Learning 8

7 8
2
 3/11/2025

When We Talk About “Learning” … Artificial Intelligence

• Learning general models from a data of particular examples

• Data is cheap and abundant (data warehouses, data marts);
knowledge/model is expensive and scarce.
• Example in retail: Customer transactions to consumer behavior:
People who bought “Da Vinci Code” also bought “The Five People You Meet in
Heaven” (www.amazon.com)
• Build a model that is a good and useful approximation to the data

Parikshit Ram, GA Tech

March 2025 Deep Learning 9 March 2025 Deep Learning 10

9 10

Machine Learning – Another View [Book Review] Superintelligence by Bostrom

• Any intellect that greatly exceeds the cognitive performance of humans

in virtually all domains of interest….
• How to…
– Artificial Intelligence
– Whole Brain Emulation – uploading
– Biological Cognition
– Brain-computer Interfaces
– Networks and Organizations

March 2025 Deep Learning 11 March 2025 Deep Learning 12

11 12
3
 3/11/2025

Learning Associations Machine Learning Tasks

• Basket analysis: • Classification

– with and without missing inputs
P (Y | X ) probability that somebody who buys X also buys Y where X and
• Regression
Y are products/services.
• Transcription – e.g., OCR
• Translation – e.g., NLP
Example: P ( ayran | pide ) = 0.7 • Anomaly detection
• Synthesis and sampling
• Imputation of missing data
• Denoising
• Density function estimation

March 2025 Deep Learning 13 March 2025 Deep Learning 14

13 14

Classification Classification: Applications

• Aka “Pattern Recognition”

• Example: Credit
• Face recognition: Pose, lighting, occlusion (glasses, beard), make-up,
scoring
hair style
• Differentiating
• Character recognition: Different handwriting styles
between low-risk and
high-risk customers • Speech recognition: Temporal dependency.
from their income and – Use of a dictionary or the syntax of the language.
savings – Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic
for speech
• Medical diagnosis: From symptoms to illnesses
Discriminant: IF income > θ1 AND savings > θ2 • ...
THEN low-risk ELSE high-risk
March 2025 Deep Learning 15 March 2025 Deep Learning 16

15 16
4
 3/11/2025

Face Recognition Regression

Training examples of a person

• Example: Price of a
used car
y = wx+w0
• x : car attributes
y : price
Test images
y = g (x | θ)
g ( ) model,
θ parameters
AT&T Laboratories, Cambridge UK
http://www.uk.research.att.com/facedatabase.html
March 2025 Deep Learning 17 March 2025 Deep Learning 18

17 18

Regression Applications Supervised Learning: Uses

• Navigating a car: Angle of the steering wheel (CMU • Prediction of future cases: Use the rule to predict the output for future
NavLab) inputs
• Kinematics of a robot arm • Knowledge extraction: The rule is easy to understand
• Compression: The rule is simpler than the data it explains
(x,y) α1= g1(x,y)
α2= g2(x,y) • Outlier detection: Exceptions that are not covered by the rule, e.g.,
α2 fraud

α1

Response surface design

March 2025 Deep Learning 19 March 2025 Deep Learning 20

19 20
5
 3/11/2025

Unsupervised Learning Reinforcement Learning

• Learning “what normally happens” • Learning a policy: A sequence of outputs

• No output • No supervised output but delayed reward
• Clustering: Grouping similar instances • Credit assignment problem
• Example applications • Game playing
– Customer segmentation in CRM • Robot in a maze
– Image compression: Color quantization • Multiple agents, partial observability, ...
– Bioinformatics: Learning motifs

March 2025 Deep Learning 21 March 2025 Deep Learning 22

21 22

All Related… Resources: Datasets

• UCI Repository: http://archive.ics.uci.edu/ml/

• MNIST Database: ttp://yann.lecun.com/exdb/mnist/
• UCI KDD Archive: http://kdd.ics.uci.edu/summary.data.application.html
• A list … http://www.dmoz.org/Computers/Artificial_Intelligence/Machine_Learning/Datasets/
– Statlib: http://lib.stat.cmu.edu/
– Delve: http://www.cs.utoronto.ca/~delve/

March 2025 Deep Learning 23 March 2025 Deep Learning 24

23 24
6
 3/11/2025

Input Encoding

• Prediction / two class classification

– Label positive data as +1 and all others as –1
– The input vector xi represents the input data as a vector of
features
Assessing and Comparing • E.g., xi = (4.2,6.7,12,….,0.5)
Test Data
xi

Classification Algorithms Positive

Data Magic Black
Box (Learning Model
Negative Machine)
Data

Prediction +1, 𝑦 > 𝑒

+1/-1 or
−1, 𝑦 ≤ 𝑒
March 2025 Deep Learning 26

25 26

Performance Measurements Precision and Recall

Confusion matrix /
contingency table
actual class (observation)

Predictions TP
Test Data tp fp
(true positive) (false positive)
FP predicted class Correct result Unexpected result
+1 (expectation) fn tn
Model (false negative) (true negative)
TN Missing result Correct absence of result
-1

Precision = Accuracy =
= +1 FN
= -1 Recall =

March 2025 Deep Learning 27 March 2025 Deep Learning 28

27 28
7
 3/11/2025

Precision and Recall ROC (Receiver Operator Characteristic) Curve

Confusion matrix /
contingency table
actual class (observation)

tp fp 1
(true positive) (false positive) 𝐶
predicted class Correct result Unexpected result 𝐶 𝐶
(expectation) fn tn
(false negative) (true negative) 𝐶
Missing result Correct absence of result true positive rate
𝐶

True positive rate = = positives correctly classified

total positive observations
(recall, hit rate)
0 1
false positive rate
negatives incorrectly classified
False positive rate = = (false alarm rate)
total negative obvervations
Discrete Classifiers 𝐶 , … , 𝐶
March 2025 Deep Learning 29 March 2025 Deep Learning 30

29 30

ROC (Receiver Operator Characteristic) Curve Calculating ROC Curves

Perfect classification… Unconditionally issuing
positive classifications • Binary classifiers:
– E.g. decision trees
1
– Discrete output
– Therefore each classifier would be a point (using the training dataset) on the ROC
Random classifier
space
true positive rate • Classifiers with a score ~ probabilistic classifier
No good!
– Calibrated  probability
– Un-calibrated  score/rank
0 1 – Response is obtained using a threshold
false positive rate
– Each threshold produces a different point in ROC space
No tp as well as no fp
– So vary the threshold from -∞ and+∞  could be very inefficient

March 2025 Deep Learning 31 March 2025 Deep Learning 32

31 32
8
 3/11/2025

Problems with Accuracy Percent Reduction in Error

• Assumes equal cost for both kinds of errors • 80% accuracy = 20% error
– cost(b-type-error) = cost (c-type-error) • suppose learning increases accuracy from 80% to 90%
• Is 99% accuracy good? • error reduced from 20% to 10%
– can be excellent, good, mediocre, poor, terrible • 50% reduction in error
– depends on problem
• 99.90% to 99.99% = 90% reduction in error
• is 10% accuracy bad?
• 50% to 75% = 50% reduction in error
– information retrieval
• can be applied to many other measures
• BaseRate = accuracy of predicting predominant class (on most problems
obtaining BaseRate accuracy is easy)

March 2025 Deep Learning 33 March 2025 Deep Learning 34

33 34

Noise and Model Complexity

Use the simpler one because

• Simpler to use
(lower computational
complexity)
• Easier to train (lower

Learning Theories •
space complexity)
Easier to explain
(more interpretable)
• Generalizes better (lower
variance - Occam’s razor)

March 2025 Deep Learning 35 March 2025 Deep Learning

36

35 36
9
 3/11/2025

Model Selection & Generalization Bias and Variance Tradeoff (Simple Problem)

• Learning is an ill-posed problem; data is not sufficient to find a unique

solution
• The need for inductive bias, assumptions about H
• Generalization: How well a model performs on new data
• Overfitting: H more complex than C or f
• Underfitting: H less complex than C or f

March 2025 Deep Learning 37 March 2025 Deep Learning 38

37 38

Bias and Variance

Bias Problem
– The hypothesis space made available by a particular classification method does
not include sufficient hypotheses

Variance Problem
– The hypothesis space made available is too large for the training data, and the
selected hypothesis may not be accurate on unseen data

March 2025 Deep Learning 40

39 40
10
 3/11/2025

Bias vs. Variance Bias and Variance

• Imagine that the training set was drawn at random from a whole set of
training sets
• The squared loss can be decomposed into a “bias” term and a
“variance” term:
– Bias = systematic error in the model’s estimates
– Variance = noise in the estimates cause by sampling noise in the training set

March 2025 Deep Learning 41 March 2025 Deep Learning 42

41 42

11