Scribd
Upload
10 views

Intro

This document provides an overview of machine learning concepts including definitions of machine learning, types of machine learning systems (supervised learning, unsupervised learning, reinforcement learning), examples of machine learning algorithms (linear regression, logistic regression, decision trees, ensemble methods), and considerations for developing machine learning models (overfitting, regularization, testing and validation). The document uses examples and diagrams to explain machine learning techniques at a high level.

Uploaded by

Tran Kim Toai
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
10 views

Intro

This document provides an overview of machine learning concepts including definitions of machine learning, types of machine learning systems (supervised learning, unsupervised learning, reinforcement learning), examples of machine learning algorithms (linear regression, logistic regression, decision trees, ensemble methods), and considerations for developing machine learning models (overfitting, regularization, testing and validation). The document uses examples and diagrams to explain machine learning techniques at a high level.

Uploaded by

Tran Kim Toai
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 38

Introduction: the Machine Learning

Landscape

Xudong Liu
Assistant Professor
School of Computing
University of North Florida

Aug. 19, 2019

Thanks to Vineet Bansal for the majority of the slides


What is ML?
“A field of study that gives computers the
ability to learn without being explicitly
programmed.” - Arthur Samuel, 1959

“A computer program is said to learn from


experience E with respect to some task T and
some performance measure P, if its performance
on T, as measured by P, improves with
experience E.” – Tom Mitchell, 1997

ML is a prominent sub-field in AI, “the new


electricity.” - Andrew Ng
(https://www.youtube.com/watch?v=21EiKfQYZXc)
What is ML?

“A field of study that gives computers the ability to learn without being
explicitly programmed.”

A machine-learning system is trained rather than explicitly programmed.


Types of ML Systems

Supervised Learning - Training Data contains desired solutions, or labels

Features
Types of ML Systems

Supervised Learning - Training Data contains desired solutions, or labels

1. K-Nearest Neighbors
2. Linear Regression
3. Logistic Regression
4. Support Vector Machines
5. Decision Trees and Random Forests
6. Neural Networks
Types of ML Systems

Unsupervised Learning - Training Data is unlabeled


Types of ML Systems

Unsupervised Learning - Training Data is unlabeled

1. K-Means (Clustering)
2. Principal Component Analysis (Dimensionality Reduction)
3. Deep Neural Networks
Types of ML Systems

Reinforcement Learning - Training Data does not


contain target output, but instead contains some
possible output together with a measure of how good
that output is.

<input>, <correct output>

<input>, <some output>, <grade for this output>


Classification vs Regression
ML Landscape
Unsupervised Learning - Clustering

Clustering

• Color “clusters” of points in a homogenous cloud of data.

• Use Cases

• Behavioral Segmentation in Marketing


• Useful as a pre-processing step before applying other classification
algorithms.
• Cluster ID could be added as feature for each data point.
Unsupervised Learning - Clustering
k-means Algorithm

• Guess some cluster centers


• Repeat until converged
• E-Step: assign points to the nearest cluster center
• M-Step: set the cluster centers to the mean
Unsupervised Learning - Clustering
Choosing k
ML Landscape
Linear Regression
Linear Regression

1 %11 %21 … %&1 Define a Hypothesis


1 %12 %22 … %&2 hϴ(X) = !^ = X ϴ
X=
⋮ ⋮ … … ⋮
1 %'
1
%'
2 … %&' Define a Cost Function (a measure of how bad we’re doing)
n
!1 1
MSE ", ℎϴ = Σ[!^$ —!$]2
!2 i= 1
y= …

!' Repeat until convergence:


• Calculate Cost Function on chosen ϴ
• Calculate slope of Cost Function
ϴ0 !^1 • Tweak ϴ so as to move downhill (reduce Cost Function value)
ϴ1 !^2
ϴ = ϴ2 ^=
! …
ϴ is now optimized for our training data.
⋮ ^'
!
ϴ&
Logistic Regression

Used to estimate the probability that an instance belongs to a particular class.


Logistic Regression
Logistic Regression

No closed-form solution, but we can use Gradient Descent!


ML Landscape
Overfitting and Underfitting
Bias-Variance Tradeoff
Regularization

How to ensure that we’re not overfitting to our training data?

Impose a small penalty on model complexity.

l1 penalty (Lasso Regression) l2 penalty (Ridge Regression)


Testing and Validation
K-fold Cross Validation
Decision Tree
Basic Idea

• Construct a tree to ask a series of questions from your data.


Decision Tree
Let’s see how it works on a real dataset.
• Boundaries always are axis-parallel
Decision Tree
Decision Trees can be used for regression!

• Minimize MSE instead of impurity.


Decision Tree
Advantages

• White Box – easily interpretable

Disadvantages

• Prone to overfitting
• Regularize by setting maximum depth
• Comes up only with orthogonal boundaries
• Sensitive to training set rotation – Use PCA!
ML Landscape
Ensemble Methods
Basic Idea

• Two Decision Trees by themselves may overfit. But combining their predictions may be a good idea!
Ensemble Methods
Basic Idea

• Two Decision Trees by themselves may overfit. But combining their predictions may be a good idea!
Bagging
Bagging = Bootstrap Aggregation

• Use the same training algorithm for every predictor, but train them on different random subsets of
the training set.

Random Forest is an Ensemble of Decision Trees, generally trained via the bagging method.
Boosting
Basic Idea

• Train several weak learners sequentially, each trying to correct the errors made by its predecessor.

Adaptive Boosting (ADABoost)


• Give more relative weight to the misclassified instances.
Boosting
Gradient Boosting

• Try to fit a new predictor to the residual errors


made by the previous predictor.

Best Performance

• Random Forests and Gradient Boosting


Methods (implemented in the xgBoost library)
have been winning most competitions on
Kaggle recently on structured data.

• Deep Learning (especially Convolutional


Networks) is the clear winner for unstructured
data problems (perception/speech/vision etc.)
ML Landscape
ML Landscape
ML Landscape

You might also like