UET

Since 2004

ĐẠI HỌC CÔNG NGHỆ, ĐHQGHN

VNU-University of Engineering and Technology

INT3405 - Machine Learning

Lecture 2: General Concepts for ML

Hanoi, 09/2024
 Recap: Traditional Programming vs Machine Learning

https://images.techhive.com/images/article/2017/05/traditional-programming-vs-machine-learning-100723299-large.jpg

FIT-CS INT3405 - Machine Learning 2

 Recap: Machine Learning vs. Deep Learning

Source: https://www.linkedin.com/pulse/lets-understand-difference-between-machine-learning-vs-gauri-bapat

FIT-CS INT3405 - Machine Learning 3

 Recap: Machine Learning vs. Deep Learning vs AI

Source: https://www.ibm.com/blogs/systems/ai-machine-learning-and-deep-learning-whats-the-difference/

FIT-CS INT3405 - Machine Learning 4

 Recap: Types of Machine Learning

Source:
https://medium.com/marketing-and-entrepr
eneurship/10-companies-using-machine-le
arning-in-cool-ways-887c25f913c3

FIT-CS INT3405 - Machine Learning 5

 Outline
● Statistics - Probability
● Typical Data Distribution
● Typical Measurements
○ Entropy, Cross Entropy
○ Mutual Information
○ Kullback-Leibler Divergence
● Learning Theory

FIT-CS INT3405 - Machine Learning 6

 What animal is it?

A cat?
or a tiger?

FIT-CS INT3405 - Machine Learning 7

 Game: Toss a coin

FIT-CS INT3405 - Machine Learning 8

Random Variable (RV)
• Random variable (RV) is a variable whose possible values are
numerical outcomes of a random phenomena.
Random Variable

Discrete RV Continuous RV
 Discrete Random Variable (DRV)
• Let toss the coin twice, and X is the number of
the head occurs.
How many values of X ?

• Let toss the dice twice, and X is the sum of the

dice
How many values of X?

FIT-CS INT3405 - Machine Learning 10

 DRV - Probability Density Function (1)
• A discrete random variable X whose values in x1, x2, …, xn.
• Probability Density Function:
• Denotation: pi = f(xi) = P(X=xi)
• Conditions:
x1 x2 Xn-1 xn
•

• f(x1) f(xn-1) f(xn)

f(x2)

FIT-CS INT3405 - Machine Learning 11

 DRV – Probability Density Function (2)
Example: Toss two coins, X is denoted as the occurrences of head.
PDF
S S x P(x)
0 1/4 = .25
S H 1 2/4 = .50

2 1/4 = .25
H S

Probability
.50

H H .25

There are 4 possible outcomes 0 1 2 x

FIT-CS INT3405 - Machine Learning 12

 Continuous Random Variable (CRV)

Time to
Weight & Height complete a task

FIT-CS INT3405 - Machine Learning 13

 CRV – Probability Density Function (1)
• f(x) is the probability density function of a given continuous random
variable X, if:

FIT-CS INT3405 - Machine Learning 14

 Cumulative Probability Distribution (1)
• Consider a random variable X, its cumulative probability function
F(x) is defined as follows:

• The probability of X in (a,b] as:

FIT-CS INT3405 - Machine Learning 15

 Cumulative Probability Distribution (2)

4) Probability density function f(x) = F’(x) as long as the derivative exits

FIT-CS INT3405 - Machine Learning 16

 Cumulative Probability Distribution (3)
• Discrete Random Variable:

• Continuous Random Variable:

FIT-CS INT3405 - Machine Learning 17

 Expected Value of Random Variable
• Discrete Random Variable:

• Continuous Random Variable:

FIT-CS INT3405 - Machine Learning 18

 Expected Value - Properties
1) E[C] = C, C: constant
2) E[CX] = C.E[X]
3) E[X + Y]=E[X] + E[Y]
4) E[XY] = E[X].E[Y] if X and Y are independent
5) Given a function h(x), we have
if X is discrete

if X is continuous

FIT-CS INT3405 - Machine Learning 19

 Variance of Random Variable

FIT-CS INT3405 - Machine Learning 20

 Variance - Properties
1) Var[C] = 0, C: constant
2) Var[CX] = C2 Var[X]
Var[X + C]=Var[X]
3) Var[X + Y] = Var[X] + Var[Y] if X and Y are independent.

FIT-CS INT3405 - Machine Learning 21

 Three Rules of Probability

(Bayesian Rule)

FIT-CS INT3405 - Machine Learning 22

 Joint Probability

FIT-CS INT3405 - Machine Learning 23

 Marginal Probability
• Discrete random variable:

• Continuous random variable:

FIT-CS INT3405 - Machine Learning 24

 Conditional Probability
• Bayesian Rule:

• Conditional Probability:

FIT-CS INT3405 - Machine Learning 25

 Posterior-, Prior Probability & Likelihood

Posterior Prior
Likelihood
probability probability

FIT-CS INT3405 - Machine Learning 26

 Independence & Dependence
• If x, y are independent, so:

• Bayesian Rule:

FIT-CS INT3405 - Machine Learning 27

 Outline
● Statistics - Probability
● Typical Data Distribution
● Typical Measurements
○ Entropy, Cross Entropy
○ Mutual Information
○ Kullback-Leibler Divergence
● Learning Theory

FIT-CS INT3405 - Machine Learning 28

 Typical Data Distributions

FIT-CS INT3405 - Machine Learning 29

 Uniform Distribution

P(X) = 1 / total number of possible outcomes

FIT-CS INT3405 - Machine Learning 30

 Uniform Distribution – Example

The probability of doomsday being Monday is 1/7

The probability that a numeric letter appearing in

500k VNĐ note is 1/10

FIT-CS INT3405 - Machine Learning 31

 Bernoulli Distribution

FIT-CS INT3405 - Machine Learning 32

 Bernoulli Distribution – Example

The probability of male/female baby

FIT-CS INT3405 - Machine Learning 33

 Binary Distribution

FIT-CS INT3405 - Machine Learning 34

 Binary Distribution – Example
X P(X)
0 0.000977
1 0.009766
2 0.043945
3 0.117188
4 0.205078
5 0.246094
6 0.205078
7 0.117188
8 0.043945
The probability getting X times of head in 10
9 0.009766
times tossing a coin
10 0.000977

FIT-CS INT3405 - Machine Learning 35

 Categorical Distribution

It is general form of Bernoulli

distribution with more than 2 possible
outcomes

FIT-CS INT3405 - Machine Learning 36

 Categorical Distribution - Example

FIT-CS INT3405 - Machine Learning 37

 Univariate Normal Distribution

Normal distribution = Gaussian distribution

FIT-CS INT3405 - Machine Learning 38

 Univariate Normal Distribution - Example

Score Distribution of a University Entrance Exam IQ Distribution

FIT-CS INT3405 - Machine Learning 39

 Multivariate Normal Distribution

FIT-CS INT3405 - Machine Learning 40

 Exponential Distribution

FIT-CS INT3405 - Machine Learning 41

 Exponential Distribution - Example

Time distribution of call duration over 15

minutes in a Telecom company

FIT-CS INT3405 - Machine Learning 42

 Outline
● Statistics - Probability
● Typical Data Distribution
● Typical Measurements
○ Entropy, Cross Entropy
○ Mutual Information
○ Kullback-Leibler Divergence
● Learning Theory

FIT-CS INT3405 - Machine Learning 43

 Typical Measurement - Entropy
• Measure the ‘uncertainty’ or ‘surprise’ in data

• Discrete random variable:

• Continuous random variable:

FIT-CS INT3405 - Machine Learning 44

 Typical Measurement – Mutual Information
• Measure the mutual dependence between two random variables.

FIT-CS INT3405 - Machine Learning 45

 Typical Measurement – Cross Entropy
• Measure the difference between two distributions

• Discrete random variable:

• Continuous random variable:

FIT-CS INT3405 - Machine Learning 46

 Typical Measurement – KL Divergence
• Measure the difference between two distributions

• Discrete random variable:

• Continuous random variable:

FIT-CS INT3405 - Machine Learning 47

 Outline
● Statistics - Probability
● Typical Data Distribution
● Typical Measurements
○ Entropy, Cross Entropy
○ Mutual Information
○ Kullback-Leibler Divergence
● Learning Theory

FIT-CS INT3405 - Machine Learning 48

 Inductive Learning

FIT-CS INT3405 - Machine Learning 49

 The Statistical Learning Framework (1)

FIT-CS INT3405 - Machine Learning 50

 The Statistical Learning Framework (2)

FIT-CS INT3405 - Machine Learning 51

 The Statistical Learning Framework (3)

FIT-CS INT3405 - Machine Learning 52

 The Statistical Learning Framework (4)

FIT-CS INT3405 - Machine Learning 53

 The Statistical Learning Framework (5)

FIT-CS INT3405 - Machine Learning 54

 The Statistical Learning Framework (6)

FIT-CS INT3405 - Machine Learning 55

 Generalization- & Empirical Error

FIT-CS INT3405 - Machine Learning 56

 Train/Test/Validation

FIT-CS INT3405 - Machine Learning 57

 Cross Validation

FIT-CS INT3405 - Machine Learning 58

 Supervised Learning Workflow

FIT-CS INT3405 - Machine Learning 59

 Unsupervised Learning Workflow

FIT-CS INT3405 - Machine Learning 60

 Summary
● Statistics - Probability
● Typical Data Distribution
● Typical Measurements
○ Entropy, Cross Entropy
○ Mutual Information
○ Kullback-Leibler Divergence
● Learning Theory

FIT-CS INT3405 - Machine Learning 61

 Reading
To understand this lecture better: Chapter 2, 3, 4 of PML.
For next lectures: 3.1 to 3.2, ESL, 11.2 PML

FIT-CS INT3405 - Machine Learning 62

UET
Since 2004

ĐẠI HỌC CÔNG NGHỆ, ĐHQGHN

VNU-University of Engineering and Technology

Q&A