01_ml-overview_notes
01_ml-overview_notes
One of the main motivations why we develop (computer) programs to automate various
kinds of processes. Originally developed as a subfield of Artificial Intelligence (AI), one of
the goals behind machine learning was to replace the need for developing computer
programs “manually.” Considering that programs are being developed to automate
processes, we can think of machine learning as the process of “automating automation.” In
other words, machine learning lets computers “create” programs (often, the intent for
developing these programs is making predictions) themselves. In other words, machine
learning is the process of turning data into programs.
It is said that the term machine learning was first coined by Arthur Lee Samuel, a pioneer in
the AI field, in 1959 1. One quote that almost every introductory machine learning resource
cites is the following, which summarizes the concept behind machine learning nicely and
concisely:
Machine learning is the field of study that gives computers the ability to learn
without being explicitly programmed. 2 — Arthur L. Samuel, AI pioneer, 1959
Now, before we introduce machine learning more formally, here is what some other people
said about the field:
The field of machine learning is concerned with the question of how to construct
computer programs that automatically improve with experience.
— Tom Mitchell, Professor Machine Learning at Carnegie Mellon University and author of the
popular “Machine Learning” textbook Machine learning is the hot new thing.
— John L. Hennessy, President of Stanford (2000–2016)
1
Arthur L Samuel. “Some studies in machine learning using the game of checkers”. In: IBM Journal of
research and development 3.3 (1959), pp. 210–229.
2
This is not a direct quote but a paraphrased version of Samuel’s sentence ”Programming computers to
learn from experience should eventually eliminate the need for much of this detailed programming effort.”
Page 2
A bit more concrete is Tom Mitchell’s description from his Machine Learning book3:
To illustrate this quote with an example, consider the problem of recognizing handwritten
digits:
After the field of machine learning was “founded” more than a half a century ago, we can
now find applications of machine learning in almost every aspect of hour life. Popular
applications of machine learning include the following:
While we go over some of these applications in class, it is a good exercise to think about
how machine learning could be applied in these problem areas or tasks listed above:
The three broad categories of machine learning are summarized in the following figure:
Supervised learing, unsupervised learning, and reinforcement learning. Note that in this
class, we will primarily focus on supervised learning, which is the “most developed” branch
of machine learning. While we will also cover various unsupervised learning algorithms,
reinforcement learning will be out of the scope of this class.
No labels/targets No feedback
Find hidden structure in data
Unsupervised Learning
Figure 3: Categories of Machine Learning (Source: Raschka & Mirjalili: Python Machine Learning,
2nd Ed.)
Supervised Learning
Supervised learning is the subcategory of machine learning that focuses on learning a classifi-
cation or regression model, that is, learning from labeled training data (i.e., inputs that also
contain the desired outputs or targets; basically, “examples” of what we want to predict).
x2
x1
Figure 4: Illustration of a binary classification problem (plus, minus) and two feature variable (x1
and x2). (Source: Raschka & Mirjalili: Python Machine Learning, 2nd Ed.).
Page 5
x
Figure 5: Illustration of a linear regression model with one feature variable (x1) and the target
variable y. The dashed-line indicates the functional form of the linear regression model. (Source:
Raschka & Mirjalili: Python Machine Learning, 2nd Ed.).
Unsupervised learning
x2
x1
Figure 6: Illustration of clustering, where the dashed lines indicate potential group membership
assignments of unlabeled data points. (Source: Raschka & Mirjalili: Python Machine Learning,
2nd Ed.).
Page 6
Reinforcement learning
Reinforcement is the process of learning from rewards while performing a series of actions.
In reinforcement learning, we do not tell the learner or agent, for example, a (ro)bot, which
action to take but merely assign a reward to each action and/or the overall outcome. Instead
of having “correct/false” label for each step, the learner must discover or learn a behavior
that maximizes the reward for a series of actions. In that sense, it is not a supervised setting
and somewhat related to unsupervised learning; however, reinforcement learning really is
its own category of machine learning. Reinforcement learning will not be covered further in
this class.
Typical applications of reinforcement learning involve playing games (chess, Go, Atari video
games) and some form of robots, e.g., drones, warehouse robots, and more recently self-
driving cars.
Environment
Reward
State
Action
Agent
Figure 7: Illustration of reinforcement learning (Source: Raschka & Mirjalili: Python Machine
Learning, 2nd Ed.).
Semi-supervised learning
Losely speaking, semi-supervised learning can be described as a mix between supervised and
unsupervised learning. In semi-supervised learning tasks, some training examples contain
outputs, but some do not. We then use the labeled training subset to label the unlabeled
portion of the training set, which we then also utilize for model training.
Unless noted otherwise (later in future lectures), we will focus on supervised learning and
classification, the most prevalent form of machine learning, from now on.
In supervised learning, we are given a labeled training dataset from which a machine learn-
ing algorithm can learn a model that can predict labels of unlabeled data points. These
unlabeled data points could be either test data points (for which we actually have labels
but we withold them for testing purposes) or unlabeled data we will collect in future. For
example, given a corpus of spam and non-spam email, a supervised learning task would be
to learn a model that predicts to which class (spam or non-spam) new emails belong.
More formally, we define h as the “hypothesis,” a function that we use to approximate some
unknown function
f (x) = y, (1)
where x is a vector of input features associated with a training example or dataset instance
(for example, the pixel values of an image) and y is the outcome we want to predict (e.g.,
what class of object we see in an image). In other words, h(x) is a function that predicts y.
Page 7
h : X → Y, (2)
where X = Rm and Y = {1, ..., k} with class labels k. in the special case of binary
classifi- cation, we have Y = {0, 1} (or Y = {−1, 1}).
And in regression, the task is to learn a function
h : Rm → R. (3)
Labels
Training Data
Labels
Training Dataset
Learning Algorithm
Final Model New Data
Labels
Labels
A crucial assumption we make in supervised learning is that the training examples have the
same distribution as the test (future) examples. In real-world applications, this assumption
is oftentimes violated, which is one of the common challenges in the field.
Please note that the notation varies wildly across the literature, since machine learning is
a field that is popular across so many disciplines. For example, in the context of
statistical learning theory, we can think of the dataset D = {< x[1], y[1] >, < x[2], y[2]
> ... ,<
x[n], y[n] >} as a sample from the population of all possible input-target pairs, and x[i]
and y[i] are instances of two random variables X ∈ X and Y ∈ Y and that the training pairs
are drawn from a joint distribution P (X, Y ) = P (X)P (Y |X).
Given an error term ϵ we can then formulate the following relationship:
Y = f (X) + ϵ. (5)
The goal in statistical learning is then to estimate f , which can then be used to predict Y :
fˆ(X) = Yˆ . (6)
In previous section, we defined refered to the ith pair in a labeled training set D as <
x[i], y[i] >.
We will adopt the convention to use italics for scalars, boldface characters for vectors, and
uppercase boldface fonts for matrices.
• x: A scalar denoting a single training example with 1 feature (e.g., the height of a
person)
x1
x2
x= . (7)
.
xm
(Note that most programming languages, incl. Python, start indexing at 0!)
n×m
• X: Design matrix, X ∈ R , which stores n training examples, where m is the
number of features.
T
1
x2 T
X= . (8)
.
T
x.
n
Page 9
Note that in order to distinguish the feature index and the training example index, we will
use a square-bracket superscript notation to refer to the ith training example and a regular
subscript notation to refer to the jth feature:
[1] [1]
··· x[1]m
1 x2
x [2] [2]
x 1 x2 ··· xm[2]
..
.
. .
X= x[n]
2
[n]
. (9)
. ··· xm
x[n]
1
The corresponding targets are represented in a column vector y, y ∈ Rn :
[1]
y
y[2]
y=
. . (10)
y[n]
Hypothesis space
In the previous section on supervised learning, we defined the hypothesis h(x) to predict
a target y. Machine learning algorithms sample from a hypothesis space that is usually
smaller than the entire space of all possible hypothesesH – an exhaustive search covering
all h ∈ H would be computationally infeasible sinceHgrows exponentially with the size
(dimensionality) of the training set. This is illustrated in the following paragraph.
Assume we are given a dataset with 4 features and 3 class labels, the class label (y ∈
Setosa,
{ Versicolor, Virginica ). Also, assume all features are binary. Given 4 features with
binary
} values (True, False), we have 24 = 16 different feature combinations (see table
below). Now, of the 16 rules, we have three classes to consider (Setosa, Versicolor,
Virginica). Hence, we have 316 = 43, 046, 721 potential combinations of 16 rules that we can
evaluate (this is the size hypothesis space, |H| = 43, 046, 721)!
Table 1: Example of decision rules for the Iris flower data dataset.
sepal length <5 sepal width <5 petal length <5 petal width <5 Class
cm cm cm cm Label
True True True True Setosa
True True True False Versicolor
True True False True Setosa
... ... ... ...
Now, imagine the features are not binary but real-valued. The hypothesis space will become
so big that it would be impossible to evaluate it exhaustively. Hence, we use machine
learning to reduce the search space within the hypothesis space. (A neural network with a
single hidden layer, a finite number of neurons, and non-linear activation functions such as
sigmoid units, was proved to be a universal function approximator5. However, the concept
of universal function approximation6 does not imply practicality, usefulness, or adequate
performance in practical problems.
5
George Cybenko. “Approximations by superpositions of a sigmoidal function”. In: Mathematics of
Control, Signals and Systems 2 (1989), pp. 183–192.
6
Bal´azs Csan´ad Cs´aji. “Approximation with artificial neural networks”. In: Faculty of Sciences,
Etvs Lornd University, Hungary 24 (2001), p. 48.
Page 10
Hypothesis space
a particular learning
algorithm category
has access to
Hypothesis space
a particular learning
algorithm can sample
Particular hypothesis
(i.e., a model/classifier)
Typically, the number of training examples required is proportional to the flexibility of the
learning algorithms. I.e., we need more training for models with
(As a rule of thumb, the more parameter to fit and/or hyperparameters to tune, the larger
the set of hypotheses to choose from).
Below are some classes of algorithms that were are going to discuss in this class:
To aid our conceptual understsanding, each of the algorithms can be categorized into
various categories.
• eager vs lazy;
• batch vs online;
• parametric vs nonparametric;
• discriminative vs generative.
These concepts or categorizations will become more clear once we discussed a few of the
different algorithms. However, below are brief descriptions of the various categorizations
listed above.
Eager vs lazy learners. Eager learners are algorithms that process training data imme-
diately whereas lazy learners defer the processing step until the prediction. In fact, lazy
learners do not have an explicit training step other than storing the training data. A pop-
ular example of a lazy learner is the Nearest Neighbor algorithm, which we will discuss in
the next lecture.
Batch vs online learning. Batch learning refers to the fact that the model is learned
on the entire set of training examples. Online learners, in contrast, learn from one training
example at the time. It is not uncommon, in practical applications, to learn a model via
batch learning and then update it later using online learning.
Parametric vs nonparametric models. Parametric models are “fixed” models, where
we assume a certain functional form for f (x) = y. For example, linear regression can be
considered as a parametric model with h(x) = w1x1 + ... + wmxm + b. Nonparametric
models are more “flexible” and do not have a pre-specfied number of parameters. In fact,
the number of parameters grows typically with the size of the training set. For example,
a decision tree would be an example of a nonparametric model, where each decision node
(e.g., a binary “True/False” assertion) can be regarded as a parameter.
Discriminative vs generative. Generative models (classically) describe methods that
model the joint distribution P (X, Y ) = P (Y )P | (X Y ) = P (X)P (Y X) for training
pairs |
< x[i], y[i] > 7. Discriminative models are taking a more “direct” approach for modeling
P (Y | X) directly. While generative models provide typically more insights and allow sam-
pling from the joint distribution, discriminative models are typically easier to compute and
produce more accurate predictions. Helpful for understanding discriminative models is the
following analogy: discriminative modeling is like trying to extract information from text in
a foreign language without learning that language.
Another useful way to think about different machine learning algorithms is Pedro Domingons
categorization of machine learning algorithms into five tribes, which he defined in his book
“The Master Algorithm”8.
7
However, in recent years, the term ”generative model” has also been used to describe models that learn
an approximation of X and sample training examples ∼ x X. Examples of such models are Generative
Adversarial Networks and Variational Autoencoders; deep learning models that are not covered in this class.
8
Pedro Domingos. The master algorithm: How the quest for the ultimate learning machine will remake
our world. Basic Books, 2015.
Page 12
As it was already indicated in the figure of Pedro Domingo’s 5 Tribes of Machine Learning,
there are several different components of machine learning algorithms.
Representation. The first component is the “representation,” i.e., which hypotheses we
can represent given a certain algorithm class.
Optimization. The second component is the optimization metric that we use to fit the
model.
Evaluation. The evaluation component is the step where we evaluate the performance of
the model after model fitting.
To extend this list slightly, these are the following 5 steps that we want to think about when
approaching a machine learning application:
Note that optimization objectives are usually not the same in practice. For example, the
optimization objective of the logistic regression algorithm is to minimize the negative log-
likelihood (or binary cross-entropy), whereas the evaluation metric could be the
classification accuracy or misclassification error.
Also, while the list above indicates a linear workflow, in practice, we often jump back to
previous steps and e.g., collect more data, try out different algorithms, and/or tune the
“knobs” (i.e., “hyperparameters”) of learning algorithms.
The following two subsections will provide a short overview of the optimization (training)
and evaluation parts.
Training
• Combinatorial search, greedy search (e.g., decision trees over, not within nodes);
• Unconstrained convex optimization (e.g., logistic regression);
• Constrained convex optimization (e.g., SVM);
• Nonconvex optimization, here: using backpropagation, chain rule, reverse autodiff.
(e.g., neural networks).
course)
There exists a number of different algorithms for each optimization task category (for exam-
ple, gradient descent or conjugate gradient, and quasi-Newton methods to optimize convex
optimization problems). Also, the objective functions that we optimize can take different
forms. Below are some examples:
• Minimize a mean squared error cost (or loss) function (CART, decision tree regression,
linear regression, adaptive linear neurons, . . . )
• Maximize log-likelihood or minimize cross-entropy loss (or cost) function
• Minimize hinge loss (support vector machine)
9
Vahid Mirjalili et al. “Semi-Adversarial Networks: Convolutional Autoencoders for Imparting Privacy
to Face Images”. In: 2018 International Conference on Biometrics (ICB). IEEE. 2018.
Page 14
Evaluation
Intuition
There are several different evaluation metric to assess the performance of a model, and the
most common ones will be discussed in future lectures. Unless noted otherwise, though, we
will focus on the classification accuracy (ACC) or misclassification error (ERR = 1 −
ACC).
The classification accuracy of an algorithm is usually evaluated empirically by counting the
fraction of correctly classified instances considering all instances that the model attempted
to classify. For instance, if we have a test dataset of 100 instances and a model classified
7,000 out of 10,000 instances correctly, then we say that the model has a 70% accuracy
on that dataset. In practice, we are often interested in the generalization performance of
a model, which is the performance on new, unseen data that has the same distribution as
the training data. The simplest way to estimate the generalization accuracy is to compute
the accuracy on a reasonably sized unseen dataset (e.g., the test dataset that we set aside).
However, there are several different techniques to estimate the generalization performance
which have different strengths and weaknesses. Being such an important topic, we will
devote a seperate lecture to model evaluation.
Prediction Error
To introduce the notion of the prediction or misclassification error more formally, consider
the 0-1 loss function
0 if yˆ =
L(yˆ, y) = y (11)
1 if yˆ /=
y,
where yˆ is the class label predicted by a given hypothesis h(x) = yˆ, and y is the
ground truth (correct class label). Then, the prediction error can be defined as the
expected value
ERRD Σn
= 1 L yˆ[i] , , (13)
test
n
y[i]
i=1
• Accuracy (1-Error)
• ROC AUC
• Precision
• Recall
• (Cross) Entropy
Page 15
• Likelihood
• Squared Error/MSE
• L-norms
• Utility
• Fitness
• ...
There are several different motivations or approaches we take when are studying ML. While
the following bullet points attempt an overall categorization, there are many exceptions:
• Engineers: Focus on developing systems with high predictive performance for real-
world problem solving
Note that machine learning was originally inspired by neuroscience, when the first attempt
of an artificial neuron, the McCulloch-Pitts Neuron10, was modeled after a biological neuron
and letter lead to the popular perceptron algorithm by Frank Rosenblatt11 – more on that
in later lectures.
As mentioned in earlier section, when we are applying machine learning to real-world prob-
lem solving, we need to define the overall objective first and then choose the right tool for
the task. Unfortunately, as a rule of thumb, simpler models are usually associated with
lower accuracy. Hence, when we are choosing modeling approaches or learning algorithms,
we have to think about whether the emphasis is on understanding a phenomenon or mak-
ing accurate predictions. Then, the problem becomes how meaningful the insights into a
modeling procedure are if the predictions are not accurate. (I.e., what is it good for if I
understand the model if the model is wrong).
In this context, a famous quote by our department’s founder:
Back in 2001, Leo Breiman wrote an interesting, highly recommended article called “Sta-
tistical Modeling: The Two Cultures” 12 where he contrasted two different approaches with
respect the the two different goals “information” and “prediction.” He referred to the two
approaches as the “data modeling culture” and the “algorithmic modeling culture.”
A B
y
nature
x The values of the parameters are estimated from C The analysis in this culture considers the inside of
Figure 13: Three screenshots from Breiman’s “Statistical Modeling: The Two Cultures” paper.
(A) the two overall motivations or goals in analyzing data. (B) The so-called “data modeling
culture.” The so-called “algorithmic modeling” culture.
The moral of the story is that whether to choose a “data model” or an “algorithmic model”
really depends on the problem we want to solve, and it’s best to use the appropriate tool
for the task. In this course, we will of course not be restricted to one kind or the other.
We will cover techniques of what would fit into the “data modeling culture” (Bayes optimal
classifiers, Bayesian networks, naive Bayes, logistic regression, etc.) as well as “algorithmic
approaches” (k-nearest neighbors, decision trees, support vector machines, etc.).
Further, Breiman mentions three lessons learned in the statistical modeling and machine
learning communities, which I summarized below:
• Rashomon effect; the the multiplicity of good models. Often we have mutliple good
models that fit the data well. If we have different models that all fit the data well,
which one should we pick?
• Occam’s razor. While we prefer favoring simple models, there is usually a conflict
between accuracy and simplicity to varying degree (in later lectures, we will learn
about techniques for selecting models within a “sweet spot” considering that this is a
trade-off)
• Bellman and the “curse of dimensionality.” Usually, having more data is con-
sidered a good thing (i.e., more information). However, more data can be harmful to
a model and make it more prone to overfitting (fitting the training data too closely
12
Leo Breiman et al. “Statistical modeling: The two cultures (with comments and a rejoinder by the
author)”. In: Statistical science 16.3 (2001), pp. 199–231.
Page 17
and not generalizing well to new data that was not seen during training; fitting noise).
Note that the curse of dimensionality refers to an increasing number of feature vari-
ables given a fixed number of training examples. Some models have smart
workarounds for dealing with large feature sets. E.g., Breiman’s random forest
algorithm, which partitions the feature space to fit individual decision trees that are
then later joined to a decistion tree ensemble (the particular algorithm is called
random forest), but more on that in future lectures.
Also note that there’s a “No Free Lunch” theorem for machine learning 13, meaning that
there’s no single best algorithm that works well across different problem domains.
Data mining focuses on the discovery of patterns in datasets or “gaining knowledge and
insights” from data – often, this involves a heavy focus on computational techniques,
working with databases, etc (nowdays, the term is more or less synonymous to “data
science”). We can then think of machine learning algorithms as tools within a data mining
project. Data mining is not “just” but also emphasis data processing, visualization, and tasks
that are traditionally not categorized as “machine learning” (for example, association rule
mining).
Artificial intelligence (AI) was created as a subfield of computer science focussing on solving
tasks that humans are good at (for example, natural language processing, image
recognition). Or in other words, the goal of AI is to mimick human intelligence.
There are two subtypes of AI: Artificial general intelligence (AGI) and narrow AI. AGI
refers to an intelligence that equals humans in several tasks, i.e., multi-purpose AI. In
contrast, narrow AI is more narrowly focused on solving a particular task that humans are
traditionally good at (e.g., playing a game, or driving a car – I would not go so far and
refer to “image classification” as AI).
In general, AI can be approached in many ways. One approach is to write a computer
program that implements a set of rules devised by domain experts. Now, hand-crafting rules
can be very laborious and time consuming. The field of machine learning then emerged as
a subfield of AI – it was concerned with the development of algorithms so that computers
can automatically learn (predictive) models from data.
Assume we want to develop a program that can recognize handwritten digits from images.
One approach would be to look at all of these images and come up with a set of (nested) if-
this-than-that rules to determine which digit is displayed in a particular image (for instance,
by looking at the relative locations of pixels). Another approach would be to use a machine
learning algorithm, which can fit a predictive model based on a thousands of labeled image
samples that we may have collected in a database.
Now, there is also deep learning, which in turn is a subfield of machine learning, referring
to a particular subset of models that are particularly good at certain tasks such as image
recognition and natural language processing. We will talk more about deep learning toward
the end of this class.
13
David H Wolpert. “The lack of a priori distinctions between learning algorithms”. In: Neural compu-
tation 8.7 (1996), pp. 1341–1390.
Or in short, machine learning (and deep learning) definitely helps to develop “AI,” however,
AI doesn’t necessarily have to be developed using machine learning – although, machine
learning makes “AI” much more convenient.
Page 18
Deep Learning
AI
Figure 14: Relationship between machine learning, deep learning, and artificial intelligence. Note
that there is also overlap between Machine learning and data mining, data science, statistics, etc.
(not shown).
Software
We will talk more about software in upcoming lectures, but at this point, I want to provide
a brief overview of the “Python for scientific computing” landscape.
Figure 15: Scientific Python packages, some of which we will discuss in class. (Image
by Jake VanderPlas; Source: https://speakerdeck.com/jakevdp/the-state-of-the-stack-scipy-2015-
keynote?slide=8). The graphic is structured in terms of “low level” to “high level.” For example,
NumPy is a numerical array library for Python. SciPy is a package with scientific computing func-
tions that extends/depends on NumPy. Scikit-learn is a machine learning library that uses both
NumPy and SciPy.
The Python-topics we will make use of in this course are the following:
Glossary
Machine learning borrows concepts from many other fields and redefines what has been
known in other fields under different names. Below is a small glossary of machine learning-
specfic terms along with some key concepts to help navigate the machine learning literature.
• Predicted output, yˆ: use this to distinguish from targets; here, means output
from the model.
• Loss function: Often used synomymously with cost function; sometimes also called
error function. In some contexts the loss for a single data point, whereas the cost
function refers to the overall (average or summed) loss over the entire dataset.
• Model: In the machine learning field, the terms hypothesis and model are often used
interchangeably. In other sciences, they can have different meanings: A hypothesis
could be the “educated guess” by the scientist, and the model would be the manifes-
tation of this guess to test this hypothesis.
• Learning algorithm: Again, our goal is to find or approximate the target function,
and the learning algorithm is a set of instructions that tries to model the target
function using our training dataset. A learning algorithm comes with a hypothesis
space, the set of possible hypotheses it explores to model the unknown target function
by formulating the final hypothesis.