0% found this document useful (0 votes)

20 views

Machine Learning With Apps in R

University document on intro to machine learning

Uploaded by

hachan

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

20 views

Machine Learning With Apps in R

University document on intro to machine learning

Uploaded by

hachan

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 43

MICHAEL CLARK

CENTER FOR SOCIAL RESEARCH

UNIVERSITY OF NOTRE DAME

AN INTRODUCTION TO MACHINE LEARNING

W I T H A P P L I C AT I O N S I N R

Machine Learning

Contents

Preface

Introduction: Explanation & Prediction

Some Terminology

Tools You Already Have

The Standard Linear Model

Logistic Regression

Expansions of Those Tools

Generalized Linear Models

Generalized Additive Models

The Loss Function

Continuous Outcomes
Squared Error

Absolute Error

10
10

Negative Log-likelihood
R Example

Categorical Outcomes
11

Misclassification

Binomial log-likelihood
12

Exponential
Hinge Loss

Regularization
R Example

Applications in R

Bias-Variance Tradeoff
14

Bias & Variance

The Tradeoff

Diagnosing Bias-Variance Issues & Possible Solutions

Worst Case Scenario

High Variance
16

High Bias

Cross-Validation

Adding Another Validation Set

K-fold Cross-Validation

Leave-one-out Cross-Validation

Bootstrap

Other Stuff

Model Assessment & Selection

Beyond Classification Accuracy: Other Measures of Performance

Process Overview

Data Preparation

Define Data and Data Partitions

Feature Scaling

Feature Engineering
21

Discretization

Model Selection
Model Assessment

Opening the Black Box

The Dataset

R Implementation

Machine Learning

Feature Selection & The Data Partition

k-nearest Neighbors

Strengths & Weaknesses

Neural Nets

Strengths & Weaknesses

Trees & Forests

Strengths & Weaknesses

Support Vector Machines

Strengths & Weaknesses

Other

Unsupervised Learning

Clustering

Latent Variable Models

Graphical Structure
36

Imputation

Ensembles

Bagging

Boosting

Stacking

Feature Selection & Importance

Textual Analysis

39
39

Bayesian Approaches
More Stuff

Summary

Cautionary Notes
Some Guidelines
Conclusion

40
40

Brief Glossary of Common Terms

Applications in R

Preface
The purpose of this document is to provide a conceptual introduction to statistical or machine learning (ML) techniques for those that
might not normally be exposed to such approaches during their typical required statistical training1 . Machine learning2 can be described
as a form of a statistics, often even utilizing well-known and familiar
techniques, that has bit of a different focus than traditional analytical
practice in the social sciences and other disciplines. The key notion is
that flexible, automatic approaches are used to detect patterns within
the data, with a primary focus on making predictions on future data.
If one surveys the number of techniques available in ML without
context, it will surely be overwhelming in terms of the sheer number
of those approaches, as well as the various tweaks and variations of
them. However, the specifics of the techniques are not as important
as more general concepts that would be applicable in most every ML
setting, and indeed, many traditional ones as well. While there will be
examples using the R statistical environment and descriptions of a few
specific approaches, the focus here is more on ideas than application3
and kept at the conceptual level as much as possible. However, some
applied examples of more common techniques will be provided in
detail.
As for prerequisite knowledge, I will assume a basic familiarity with
regression analyses typically presented to those in applied disciplines,
particularly those of the social sciences. Regarding programming, one
should be at least somewhat familiar with using R and Rstudio, and
either of my introductions here and here will be plenty. Note that I
wont do as much explaining of the R code as in those introductions,
and in some cases I will be more concerned with getting to a result
than clearly detailing the path to it. Armed with such introductory
knowledge as can be found in those documents, if there are parts of
R code that are unclear one would have the tools to investigate and
discover for themselves the details, which results in more learning
anyway.

I generally have in mind social science

researchers but hopefully keep present
the material broadly enough for anyone
that may be interested.
2
Also referred to as applied statistical
learning, statistical engineering, data
science or data mining in other contexts.

Indeed, there is evidence that with

large enough samples many techniques
converge to similar performance.

The latest version of this document

is dated September 9, 2013 (original
March 2013).

Machine Learning

Introduction: Explanation &

Prediction
F O R A N Y PA RT I C U L A R A N A LY S I S C O N D U C T E D , emphasis can be
placed on understanding the underlying mechanisms which have specific theoretical underpinnings, versus a focus that dwells more on
performance and, more to the point, future performance. These are not
mutually exclusive goals in the least, and probably most studies contain a little of both in some form or fashion. I will refer to the former
emphasis as that of explanation, and the latter that of prediction.
In studies with a more explanatory focus, traditionally analysis concerns a single data set. For example, one assumes a data generating
distribution for the response, and one evaluates the overall fit of a
single model to the data at hand, e.g. in terms of R-squared, and statistical significance for the various predictors in the model. One assesses
how well the model lines up with the theory that led to the analysis,
and modifies it accordingly, if need be, for future studies to consider.
Some studies may look at predictions for specific, possibly hypothetical
values of the predictors, or examine the particular nature of individual
predictors effects. In many cases, only a single model is considered.
In general though, little attempt is made to explicitly understand how
well the model will do with future data, but we hope to have gained
greater insight as to the underlying mechanisms guiding the response
of interest. Following Breiman (2001), this would be more akin to the
data modeling culture.
For the other type of study focused on prediction, newer techniques
are available that are far more focused on performance, not only for
the current data under examination but for future data the selected
model might be applied to. While still possible, relative predictor importance is less of an issue, and oftentimes there may be no particular
theory to drive the analysis. There may be thousands of input variables, such that no simple summary would likely be possible anyway.
However, many of the techniques applied in such analyses are quite
powerful, and steps are taken to ensure better results for new data.
Again referencing Breiman (2001), this perspective is more of the algorithmic modeling culture.
While the two approaches are not exclusive, I present two extreme
views of the situation:
To paraphrase provocatively, machine learning is statistics minus any
checking of models and assumptions. ~Brian Ripley, 2004
... the focus in the statistical community on data models has:
Led to irrelevant theory and questionable scientific conclusions.

Applications in R

Kept statisticians from using more suitable algorithmic models.

Prevented statisticians from working on exciting new problems. ~Leo
Brieman, 2001

Respective departments of computer science and statistics now overlap more than ever as more relaxed views seem to prevail today, but
there are potential drawbacks to placing too much emphasis on either
approach historically associated with them. Models that just work
have the potential to be dangerous if they are little understood. Situations for which much time is spent sorting out details for an ill-fitting
model suffers the converse problem- some (though often perhaps very
little actually) understanding with little pragmatism. While this paper
will focus on more algorithmic approaches, guidance will be provided
with an eye toward their use in situations where the typical data modeling approach would be applied, thereby hopefully shedding some
light on a path toward obtaining the best of both worlds.

Some Terminology
For those used to statistical concepts such as dependent variables,
clustering, and predictors, etc. you will have to get used to some differences in terminology4 such as targets, unsupervised learning, and
inputs etc. This doesnt take too much, even if it is somewhat annoying
when one is first starting out. I wont be too beholden to either in this
paper, and it should be clear from the context whats being referred to.
Initially I will start off mostly with non-ML terms and note in brackets
its ML version to help the orientation along.

Tools You Already Have

O N E T H I N G T H AT I S I M P O RTA N T T O K E E P I N M I N D A S Y O U B E G I N is
that standard techniques are still available, although we might tweak
them or do more with them. So having a basic background in statistics
is all that is required to get started with machine learning. Again, the
difference between ML and traditional statistical analysis is one more
of focus than method.

The Standard Linear Model

All introductory statistics courses will cover linear regression in great
detail, and it certainly can serve as a starting point here. We can describe it as follows in matrix notation:

See this for a comparison.

Machine Learning

y = N (, 2 )
= X
Where y is a normally distributed vector of responses [target] with
mean and constant variance 2 . X is a typical model matrix, i.e. a
matrix of predictor variables and in which the first column is a vector of 1s for the intercept [bias5 ], and is the vector of coefficients
[weights] corresponding to the intercept and predictors in the model.
What might be given less focus in applied courses however is how
often it wont be the best tool for the job or even applicable in the form
it is presented. Because of this many applied researchers are still hammering screws with it, even as the explosion of statistical techniques
of the past quarter century has rendered obsolete many current introductory statistical texts that are written for disciplines. Even so, the
concepts one gains in learning the standard linear model are generalizable, and even a few modifications of it, while still maintaining the
basic design, can render it still very effective in situations where it is
appropriate.
Typically in fitting [learning] a model we tend to talk about Rsquared and statistical significance of the coefficients for a small
number of predictors. For our purposes, let the focus instead be on
the residual sum of squares6 with an eye towards its reduction and
model comparison. We will not have a situation in which we are only
considering one model fit, and so must find one that reduces the sum
of the squared errors but without unnecessary complexity and overfitting, concepts well return to later. Furthermore, we will be much more
concerned with the model fit on new data [generalization].

Yes, you will see bias refer to an

intercept, and also mean something
entirely different in our discussion of
bias vs. variance.

(y f ( x ))2 where f ( x ) is a function

of the model predictors, and in this
context a linear combination of them
(X).
6

Logistic Regression
Logistic regression is often used where the response is categorical in
nature, usually with binary outcome in which some event occurs or
does not occur [label]. One could still use the standard linear model
here, but you could end up with nonsensical predictions that fall outside the 0-1 range regarding the probability of the event occurring, to
go along with other shortcomings. Furthermore, it is no more effort
nor is any understanding lost in using a logistic regression over the
linear probability model. It is also good to keep logistic regression in
mind as we discuss other classification approaches later on.
Logistic regression is also typically covered in an introduction to
statistics for applied disciplines because of the pervasiveness of binary
responses, or responses that have been made as such7 . Like the standard linear model, just a few modifications can enable one to use it to
provide better performance, particularly with new data. The gist is,

It is generally a bad idea to discretize

continuous variables, especially the
dependent variable. However contextual
issues, e.g. disease diagnosis, might
warrant it.

Applications in R

it is not the case that we have to abandon familiar tools in the move
toward a machine learning perspective.

Expansions of Those Tools

Generalized Linear Models
To begin, logistic regression is a generalized linear model assuming a
binomial distribution for the response and with a logit link function as
follows:
y = Bin(, size = 1)
= g()
= X
This is the same presentation format as seen with the standard linear model presented before, except now we have a link function g(.)
and so are dealing with a transformed response. In the case of the
standard linear model, the distribution assumed is the gaussian and
the link function is the identity link, i.e. no transformation is made.
The link function used will depend on the analysis performed, and
while there is choice in the matter, the distributions used have a typical, or canonical link function8 .
Generalized linear models expand the standard linear model, which
is a special case of generalized linear model, beyond the gaussian
distribution for the response, and allow for better fitting models of
categorical, count, and skewed response variables. We have also have a
counterpart to the residual sum of squares, though well now refer to it
as the deviance.

Generalized Additive Models

Additive models extend the generalized linear model to incorporate
nonlinear relationships of predictors to the response. We might note it
as follows:
y = f amily(, ...)
= g()
= X + f ( X )
So we have the generalized linear model but also smooth functions
f ( X ) of one or more predictors. More detail can be found in Wood
(2006) and I provide an introduction here.
Things do start to get fuzzy with GAMs. It becomes more difficult
to obtain statistical inference for the smoothed terms in the model,
and the nonlinearity does not always lend itself to easy interpretation.

As another example, for the Poisson

distribution, the typical link function
would be the log()

Machine Learning

However really this just means that we have a little more work to get
the desired level of understanding. GAMs can be seen as a segue toward more black box/algorithmic techniques. Compared to some of
those techniques in machine learning, GAMs are notably more interpretable, though perhaps less so than GLMs. Also, part of the estimation process includes regularization and validation in determining the
nature of the smooth function, topics of which we will return later.

The Loss Function

G I V E N A S E T O F P R E D I C T O R VA R I A B L E S X and some response y, we
look for some function f ( X ) to make predictions of y from those input
variables. We also need a function to penalize errors in prediction- a
loss function, L(Y, f ( X )). With chosen loss function, we then find the
model which will minimize loss, generally speaking. We will start with
the familiar and note a couple others that might be used.

Continuous Outcomes
Squared Error
The classic loss function for linear models with continuous response is
the squared error loss function, or the residual sum of squares.
L(Y, f ( X )) = (y f ( X ))2

Absolute Error
For an approach more robust to extreme observations, we might
choose absolute rather than squared error as follows. In this case,
predictions are a conditional median rather than a conditional mean.
L(Y, f ( X )) = |(y f ( X ))|

Negative Log-likelihood
We can also think of our usual likelihood methods learned in a standard applied statistics course as incorporating a loss function that is
the negative log-likelihood pertaining to the model of interest. If we
assume a normal distribution for the response we can note the loss
function as:
L(Y, f ( X )) = n ln + 21 2 (y f ( X ))2

Applications in R

In this case it would converge to the same answer as the squared

error/least squares solution.

R Example
The following provides code that one could use with the optim function in R to find estimates of regression coefficients (beta) that minimize the squared error. X is a design matrix of our predictor variables
with the first column a vector of 1s in order to estimate the intercept. y
is the continuous variable to be modeled9 .

Type ?optim at the console for more

detail.

sqerrloss = function(beta, X, y) {
mu = X %*% beta
sum((y - mu)^2)
}
set.seed(123)
X = cbind(1, rnorm(100), rnorm(100))
y = rowSums(X[, -1] + rnorm(100))
out1 = optim(par = c(0, 0, 0), fn = sqerrloss, X = X, y = y)
out2 = lm(y ~ X[, 2] + X[, 3]) # check with lm
rbind(c(out1$par, out1$value), c(coef(out2), sum(resid(out2)^2)))
##
(Intercept) X[, 2] X[, 3]
## [1,]
0.2702 0.7336 1.048 351.1
## [2,]
0.2701 0.7337 1.048 351.1

Categorical Outcomes
Here well also look at some loss functions useful in classification problems. Note that there is not necessary exclusion in loss functions for
continuous vs. categorical outcomes10 .

Misclassification
Probably the most straightforward is misclassification, or 0-1 loss. If
we note f as the prediction, and for convenience we assume a [-1,1]
response instead of a [0,1] response:
L(Y, f ( X )) = I (y 6= sign( f ))
In the above, I is the indicator function and so we are summing
misclassifications.

Binomial log-likelihood
L(Y, f ( X )) = log(1 + e2y f )
The above is in deviance form, but is equivalent to binomial log
likelihood if y is on the 0-1 scale.

For example, we could use minimize

squared errors in the case of classification also.

Machine Learning

Exponential
Exponential loss is yet another loss function at our disposal.
L(Y, f ( X )) = ey f

Hinge Loss
A final loss function to consider, typically used with support vector
machines, is the hinge loss function.
L(Y, f ( X )) = max(1 y f , 0)

2.5
2.0
1.5

Loss

1.0
0.5
0.0

Which of these might work best may be specific to the situation, but
the gist is that they penalize negative values (misclassifications) more
heavily and increasingly so the worse the misclassification (except for
misclassification error, which penalizes all misclassifications equally),
with their primary difference in how heavy that penalty is. At right is
a depiction of the loss as a functions above, taken from Hastie et al.
(2009).

Misclassification
Exponential
Binomial Deviance
Squared Error
Support Vector

3.0

Here negative values of y f are misclassifications, and so correct

classifications do not contribute to the loss. We could also note it as
(1 y f )+ , i.e. summing only those positive values of 1 y f .

Regularization
I T I S I M P O RTA N T T O N O T E that a model fit to a single data set might
do very well with the data at hand, but then suffer when predicting independent data 11 . Also, oftentimes we are interested in a best subset
of predictors among a great many, and in this scenario the estimated
coefficients are overly optimistic. This general issue can be improved
by shrinking estimates toward zero, such that some of the performance
in the initial fit is sacrificed for improvement with regard to prediction.
Penalized estimation will provide estimates with some shrinkage,
and we can use it with little additional effort with our common procedures. Concretely, lets apply this to the standard linear model, where
we are finding estimates of that minimize the squared error loss.
= arg min (y X)2

In words, were finding the coefficients that minimize the sum of the
squared residuals. With the approach to regression here we just add a
penalty component to the procedure as follows.

In terminology we will discuss further

later, such models might have low bias
but notable variance.

Applications in R

p
= arg min (y X)2 + j
j =1

In the above equation, is our penalty term12 for which larger values will result in more shrinkage. Its applied to the L1 or Manhattan
norm of the coefficients, 1 , 2 ... p , i.e. not including the intercept 0 ,
and is the sum of their absolute values (commonly referred to as the
lasso13 ). For generalized linear and additive models, we can conceptually express a penalized likelihood as follows:
p
l p ( ) = l ( ) j

This can be set explicitly or also

estimated via a validation approach. As
we do not know it beforehand, we can
estimate it on a validation data set (not
the test set) and then use the estimated
value when estimating coefficients via
cross-validation with the test set. We will
talk more about validation later.
13
See Tibshirani (1996) Regression
shrinkage and selection via the lasso.

j =1

As we are maximizing the likelihood the penalty is a subtraction,

but nothing inherently different is shown. This basic idea of adding
a penalty term will be applied to all machine learning approaches,
but as shown, we can apply such a tool to classical methods to boost
prediction performance.
It should be noted that we can go about the regularization in different ways. For example, using the squared L2 norm results in what is
called ridge regression (a.k.a. Tikhonov regularization)14 , and using a
weighted combination of the lasso and ridge penalties gives us elastic
net regularization.

Interestingly, the lasso and ridge

regression results can be seen as a
Bayesian approach using a zero mean
Laplace and Normal prior distribution
respectively for the j .

R Example
In the following example, we take a look at the lasso approach for a
standard linear model. We add the regularization component, with a
fixed penalty for demonstration purposes15 . However you should
insert your own values for in the optim line to see how the results
are affected.
sqerrloss_reg = function(beta, X, y, lambda=.1){
mu = X%*%beta
sum((y-mu)^2) + lambda*sum(abs(beta[-1]))
}
out3 = optim(par=c(0,0,0), fn=sqerrloss_reg, X=X, y=y)
rbind(c(out1$par, out1$value),
c(coef(out2),sum(resid(out2)^2)),
c(out3$par, out3$value) )
##
(Intercept) X[, 2] X[, 3]
## [1,]
0.2702 0.7336 1.048 351.1
## [2,]
0.2701 0.7337 1.048 351.1
## [3,]
0.2704 0.7328 1.047 351.3

From the above, we can see in this case that the predictor coefficients have indeed shrunk toward zero slightly while the residual sum
of squares has increased just a tad.

As noted previously, in practice

would be estimated via some validation
procedure.

Machine Learning

In general, we can add the same sort of penalty to any number of

models, such as logistic regression, neural net models, recommender
systems etc. The primary goal again is to hopefully increase our ability
to generalize the selected model to new data. Note that the estimates
produced are in fact biased, but we have decreased the variance with
new predictions as a counterbalance, and this brings us to the topic of
the next section.

Bias-Variance Tradeoff
I N M O S T O F S C I E N C E we are concerned with reducing uncertainty in
our knowledge of some phenomenon. The more we know about the
factors involved or related to some outcome of interest, the better we
can predict that outcome upon the influx of new information. The initial step is to take the data at hand, and determine how well a model
or set of models fit the data in various fashions. In many applications
however, this part is also more or less the end of the game as well16 .
Unfortunately, such an approach in which we only fit models to one
data set does not give a very good sense of generalization performance,
i.e. the performance we would see with new data. While typically not
reported, most researchers, if they are spending appropriate time with
the data, are actually testing a great many models, for which the best
is then provided in detail in the end report. Without some generalization performance check however, such performance is overstated when
it comes to new data.
In the following consider a standard linear model scenario, e.g.
with squared-error loss function and perhaps some regularization,
and a data set in which we split the data in some random fashion into
a training set, for initial model fit, and a test set, which is a separate
and independent data set, to measure generalization performance17 .
We note training error as the (average) loss over the training set, and
test error as the (average) prediction error obtained when a model
resulting from the training data is fit to the test data. So in addition to
the previously noted goal of finding the best model (model selection),
we are interested further in estimating the prediction error with new
data (model performance).

Bias & Variance

Conceptually18 , with the standard model Y = f ( X ) + e with we can
think of the expected prediction error at a specific input X = x0 as:
Errorx0 = Irreducible Error + Bias2 + Variance

I should note that I do not make any

particular claim about the quality of such
analysis. In many situations the cost of
data collection is very high, and for all
the current enamorment with big data,
a lot of folks will never have access to
big data for their situation (e.g. certain
clinical populations). In these situations
getting new data for which one might
make predictions is extremely difficult.

In typical situations there are parameters specific to some analytical

technique for which one would have
no knowledge and which must be estimated along with the usual parameters
of the standard models. The penalty
parameter in regularized regression is
one example of such a tuning parameter.
In the best case scenario, we would also
have a validation set, where we could
determine appropriate values for such
parameters based on performance with
the validation data set, and then assess
generalization performance on the test
set when the final model has been chosen. However, methods are available
to us in which we can approximate the
validation step in other ways.
18
Much of the following is essentially
a paraphrase of parts of Hastie et al.
(2009, Chap. 2 & 7).

Applications in R

In other words, we have three components to our general notion of

prediction error:
e2 An initial variance of the target around the true mean f ( x0 ) (unavoidable).
Bias2 the amount the average of our estimate varies from the true
mean.
Variance the variance of the target value about its mean.
Slightly more formally, we can present this as follows, with h0 our
estimated (hypothesized) value:
Errorx0 = Var (e) + (E[h0 ] f ( x0 ))2 + Var (h0 )

Low Bias

Low Variance

High Variance

0.6
0.4
0.2

Prediction Error

0.8

1.0

High Bias

0.0

Outlining a general procedure, we start by noting the prediction error

on a training data set with multiple models of varying complexity (e.g.
increasing the number of predictor variables), and then assess the
performance of the chosen models in terms of prediction error on the
test set. We then perform the same activity for a total of 100 simulated
data sets, for each level of complexity.
The results from this process might look like the image to the right
taken from Hastie et al. (2009). With regard to the training data, we
have errortrain for one hundred training sets for each level of model
complexity. The bold blue line notes this average error over the 100
sets by model complexity. The bold red line the average test error
(errortest ) across the 100 test data sets.
Ideally wed like to see low bias and variance, but things are not
so easy. One thing we can see clearly is that errortrain is not a good
estimate of errortest , which is now our focus in terms of performance. If
we think of the training error as what we would see in typical research
where one does everything with a single data set, we are using the
same data set to fit the model and assess error. As the model is adapted
to that data set specifically, it will be overly optimistic in the estimate
of the error, that optimism being the difference between the error
rate we see based on the training data versus the average of what we
would get with many test data sets. We can think of this as a problem
of overfitting to the training data. Models that do not incorporate any
regularization or validation process of any kind are likely overfit to the
data presented.
Generally speaking, the more complex the model, the lower the
bias, but the higher the variance, as depicted in the graphic. Specifically however, the situation is more nuanced, where type of problem

1.2

The Tradeoff

Model Complexity (df)

Machine Learning

(classification with 0-1 loss vs. continuous response with squared error
loss19 ) and technique (a standard linear model vs. regularized fit) will
exhibit different bias-variance relationships.

Diagnosing Bias-Variance Issues & Possible Solutions

19
See Friedman (1996) On Bias, Variance, 0/1 Loss and the Curse of Dimensionality for the unusal situations that
can arise in dealing with classification
error with regard to bias and variance.

Lets assume a regularized linear model with a standard data split into
training and test sets. We will describe different scenarios with possible
solutions.

Low Variance

High Variance
X

XX
XXX
X
High Bias

X
X

XX
XXX
X

Low Bias

X
X
X

Figure adapted from Domingos (2012).

1.0

https://libraries.mit.edu/
archives/exhibits/purse/

0.0

0.5

1.0

1.0
1.5

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

Cross-Validation
As noted in the previous section, in machine learning approaches we
are particularly concerned with prediction error on new data. The simplest validation approach would be to split the data available into a

0.0
0.5

0.5

1.0

With bias issues our training error is high and test error is not too
different from training error (underfitting problem). Adding new predictors/features, interaction terms, polynomials etc. can help here.
Additionally reducing the penalty parameter would also work with
even less effort, though generally it should be estimated rather than
explicitly set.

1.5

High Bias

0.0

0.5

1.0

When variance is a problem, our training error is low while test error is
relatively high (overfitting problem). Implementing more shrinkage or
other penalization to model complexity may help with the issue. In this
case more data may help as well.

1.5

High Variance

0.5

1.0

Starting with the worst case scenario, poor models may exhibit high
bias and high variance. One thing that will not help this situation (perhaps contrary to intuition) is adding more data, i.e. increasing N. You
cant make a silk purse out of a sows ear (usually20 ), and adding more
data just gives you a more accurate picture of how awful your model
is. One might need to rework the model, e.g. adding new predictors
or creating them via interaction terms, polynomials, or other smooth
functions as in additive models, or simply collecting better and/or
more relevant data.

X X
X X
X

1.5

Worst Case Scenario

Figure inspired by Murphy (2012, figure

6.5) showing the bias-variance tradeoff.
Sample (left) and average (right) fits of
linear regression using a gaussian radial
basis function expansion. The green line
represents the true relationship. The
top row shows low variance between
one fit and the next (left) but notable
bias (right) in that the average fit is off.
Compare to the less regularized (high
variance, low bias) situation of the
bottom row. See the kernlab package for
the fitting function used.

Applications in R

training and test set as discussed previously. We estimate the model

on the training data, and apply the model to the test data, get the predictions and measure our test error, selecting whichever model results
in the least test error. A hypothetical learning curve display the results
of such a process is shown to the right. While fairly simple, other approaches are more commonly used and result in better estimates of
performance21 .

One technique that might be utilized for larger data sets, is to split
the data into training, validation and final test sets. For example, one
might take the original data and create something like a 60-20-20%
split to create the needed data sets. The purpose of the initial validation set is to select the optimal model and determine the values of
tuning parameters. These are parameters which generally deal with
how complex a model one will allow, but for which one would have
little inkling as to what they should be set at before hand (e.g. our
shrinkage parameter). We select models/tuning parameters that minimize the validation set error, and once the model is chosen examine
test set error performance. In this way performance assessment is still
independent of the model development process.

Test Error

Adding Another Validation Set

0.6

0.4

0.2

0.0

100

Model Complexity Parameter

Along with some of the other works

cited, see Harrell (2001) for a good
discussion of model validation.

K-fold Cross-Validation
In many cases we dont have enough data for such a split, and the
split percentages are arbitrary anyway and results would be specific
to the specific split chosen. Instead we can take a typical data set and
randomly split it into = 10 equal-sized (or close to it) parts. Take
the first nine partitions and use them as the training set. With chosen
model, make predictions on the test set. Now do the same but this time
use the 9th partition as the holdout set. Repeat the process until each
of the initial 10 partitions of data have been used as the test set. Average the error across all procedures for our estimate of prediction error.
With enough data, this (and the following methods) could be used as
the validation procedure before eventual performance assessment on
an independent test set with the final chosen model.

Leave-one-out Cross-Validation
Leave-one-out (LOO) cross-validation is pretty much the same thing
but where = N. In other words, we train a model for all observations
except the th one, assessing fit on the observation that was left out.

Partition 1

Partition 2

Partition 3

Iteration 1

Train

Test

Iteration 2

Train

Test

Train

Iteration 3

Test

Train

An illustration of 3-fold classification.

Machine Learning

We then cycle through until all observations have been left out once to
obtain an average accuracy.
Of the two, K-fold may have relatively higher bias but less variance, while LOO would have the converse problem, as well as possible
computational issues22 . K-folds additional bias would be diminished
would with increasing sample sizes, and generally 5 or 10-fold crossvalidation is recommended.

For squared-error loss situations, there

is a Generalized cross-validation (GCV)
that can be estimated more directly
without actually going to the entire LOO
procedure, and functions similarly to
AIC.

Bootstrap
With a bootstrap approach, we draw B random samples with replacement from our original data set, creating B bootstrapped data sets of
the same size as the original data. We use the B data sets as training
sets and, using the original data as the test set, average the prediction
error across the models.

Other Stuff
Along with the above there are variations such as repeated cross validation, the .632 bootstrap and so forth. One would want to do a bit of
investigating, but -fold and bootstrap approaches generally perform
well. If variable selection is part of the goal, one should be selecting
subsets of predictors as part of the cross-validation process, not at
some initial data step.

Model Assessment & Selection

I N T Y P I C A L M O D E L C O M PA R I S O N within the standard linear model
framework, there are a number of ways in which we might assess
performance across competing models. For standard OLS regression
we might examine adjusted-R2 , or with the generalized linear models
we might pick a model with the lowest AIC23 . As we have already
discussed, in the machine learning context we are interested in models
that reduce e.g. squared error loss (regression) or misclassification
error (classification). However in dealing with many models some
differences in performance may be arbitrary.

Beyond Classification Accuracy: Other Measures of Performance

In typical classification situations we are interested in overall accuracy.
However there are situations, not uncommon, in which simple accuracy isnt a good measure of performance. As an example, consider

In situations where it is appropriate to

calculate in the first place, AIC can often
compare to the bootstrap and k-fold
cross-validation approaches.

Applications in R

the prediction of the occurrence of a rare disease. Guessing a nonevent every time might result in 99.9% accuracy, but that isnt how we
would prefer to go about assessing some classifiers performance. To
demonstrate other sources of classification information, we will use the
following 2x2 table that shows values of some binary outcome (0 =
non-event, 1 = event occurs) to the predictions made by some model
for that response (arbitrary model). Both a table of actual values, often
called a confusion matrix24 , and an abstract version are provided.

Predicted

1
0

1
41
16

Actual
0
21
13

Predicted

1
0

1
A
C

This term has always struck me as

highly sub-optimal.

Actual
0
B
D

True Positive, False Positive, True Negative, False Negative Above, these are A, B,
D, and C respectively.
Accuracy Number of correct classifications out of all predictions ((A+D)/Total).
In the above example this would be (41+13)/91, about 59%.
Error Rate 1 - Accuracy.
Sensitivity is the proportion of correctly predicted positives to all true positive
events: A/(A+C). In the above example this would be 41/57, about 72%.
High sensitivity would suggest a low type II error rate (see below), or high
statistical power. Also known as true positive rate.
Specificity is the proportion of correctly predicted negatives to all true negative
events: D/(B+D). In the above example this would be 13/34, about 38%.
High specificity would suggest a low type I error rate (see below). Also
known as true negative rate.
Postive Predictive Value (PPV) proportion of true positives of those that are
predicted positives: A/A+B. In the above example this would be 41/62,
about 66%.
Negative Predictive Value (NPV) proportion of true negatives of those that are
predicted negative: D/C+D. In the above example this would be 13/29,
about 45%.
Precision See PPV.
Recall See sensitivity.
Lift Ratio of positive predictions given actual positives to the proportion of
positive predictions out of the total: (A/(A+C))/((A+B)/Total). In the
above example this would be (41/(41+16))/((41+21)/(91)), or 1.05.
F Score (F1 score) Harmonic mean of precision and recall: 2*(Precision*Recall)/(Precision+Recall).
In the above example this would be 2*(.66*.72)/(.66+.72), about .69.
Type I Error Rate (false positive rate) proportion of true negatives that are
incorrectly predicted positive: B/B+D. In the above example this would be
21/34, about 62%. Also known as alpha.
Type II Error Rate (false negative rate) proportion of true positives that are
incorrectly predicted negative: C/C+A. In the above example this would be
16/57, about 28%. Also known as beta.

Machine Learning

False Discovery Rate proportion of false positives among all positive predictions: B/A+B. In the above example this would be 21/62, about 34%.
Often used in multiple comparison testing in the context of ANOVA.
Phi coefficient A measure of association: (A*D - B*C)/(sqrt((A+C)*(D+B)*(A+B)*(D+C))).
In the above example this would be .11.

Note the following summary of several measures where N+ and

N are the total true positive values and total true negative values
respectively, and T+ , F+ , T and F are true positive, false positive,
etc.25 :

Predicted

1
0

Actual
1
T+ /N+ = TPR = sensitivity = recall
F /N+ = FNR = Type II

0
F+ /N = FPR = Type I
T /N = TNR = specificity

There are many other measures such as area under a Receiver Operating Curve (ROC), odds ratio, and even more names for some of
the above. The gist is that given any particular situation you might be
interested in one or several of them, and it would generally be a good
idea to look at a few.

Process Overview
D E S P I T E T H E F A C A D E O F A P O L I S H E D P R O D U C T one finds in published research, most of the approach with the statistical analysis
of data is full of data preparation, starts and stops, debugging, reanalysis, tweaking and fine-tuning etc. Statistical learning is no different in this sense. Before we begin with explicit examples, it might be
best to give a general overview of the path well take.

Data Preparation
As with any typical statistical project, probably most of the time will be
spent preparing the data for analysis. Data is never ready to analyze
right away, and careful checks must be made in order to ensure the
integrity of the information. This would include correcting errors of
entry, noting extreme values, possibly imputing missing data and so
forth. In addition to these typical activities, we will discuss a couple
more things to think about during this initial data examination when
engaged in machine learning.

Define Data and Data Partitions

As we have noted previously, ideally we will have enough data to create a hold-out, test, or validation data set. This would be some random

Table based on table 5.3 in Murphy

(2012)

Applications in R

partition of the data such that we could safely conclude that the data
in the test set comes from the same population as the training set. The
training set is used to fit the initial models at various tuning parameter
settings, with a best model being that which satisfies some criterion
on the validation set (or via a general validation process). With final
model and parameters chosen, generalization error will be assessed
with the the performance of the final model on the test data.

Feature Scaling
Even with standard regression modeling, centering continuous variables (subtracting the mean) is a good idea so that intercepts and zero
points in general are meaningful. Standardizing variables so that they
have similar variances or ranges will help some procedures find their
minimums faster. Another common transformation is min-max normalization26 , which will transfer a scale to a new one of some chosen
minimum and maximum. Note that whatever approach is done, it must
be done after any explicit separation of data. So if you have separate
training and test sets, they should be scaled separately.

score

min

scorenew = max oldmin old (maxnew

old
old
minnew ) + minnew

Feature Engineering
If were lucky well have ideas on potential combinations or other
transformations of the predictors we have available. For example, in
typical social science research there are two-way interactions one is
often predisposed to try, or perhaps one can sum multiple items to a
single scale score that may be more relevant. Another common technique is to use a dimension reduction scheme such as principal components, but this can (and probably should) actually be an implemented
algorithm in the ML process27 .
One can implement a variety of such approaches in ML as well to
create additional potentially relevant features, even automatically,
but as a reminder, a key concern is overfitting, and doing broad construction of this sort with no contextual guidance would potentially be
prone to such a pitfall. In other cases it may simply be not worth the
time expense.

For example, via principal components

or partial least squares regression.

Discretization
While there may be some contextual exceptions to the rule, it is
generally a pretty bad idea in standard statistical modeling to discretize/categorize continuous variables28 . However some ML procedures will work better (or just faster) if dealing with discrete valued
predictors rather than continuous. Others even require them; for example, logic regression needs binary input. While one could pick arbitrary

See Harrell (2001) for a good summary of reasons why not to.

Machine Learning

intervals and cutpoints in an unsupervised fashion such as picking

equal range bins or equal frequency bins, there are supervised algorithmic approaches that will use the information in the data to produce
some optimal discretization.
Its generally not a good idea to force things in data analysis, and
given that a lot of data situations will be highly mixed, it seems easier
to simply apply some scaling to preserve the inherent relationships
in the data. Again though, if one has only a relative few continuous
variables or a context in which it makes sense to, its probably better to
leave continuous variables as such.

Model Selection
With data prepared and ready to analyze, one can use a validation
process to come up with a viable model. Use an optimization procedure or a simple grid search over a set of specific values to examine
models at different tuning parameters. Perhaps make a finer search
once an initial range of good performing values is found, though one
should not split hairs over arbitrarily close performance. Select a best
model given some criterion such as overall accuracy, or if concerned
about over fitting, select the simplest model within one standard error
of the accuracy of the best, or perhaps the simplest within X% of the
best model. For highly skewed classes, one might need to use a different measure of performance besides accuracy. If one has a great many
predictor variables, one may use the model selection process to select
features that are most important.

Model Assessment
With tuning parameters/features chosen, we then examine performance on the independent test set (or via some validation procedure).
For classification problems, consider other statistics besides accuracy
as measures of performance, especially if classes are unbalanced. Consider other analytical techniques that are applicable and compare
performance among the different approaches. One can even combine
disparate models predictions to possibly create an even better classifier29 .

Opening the Black Box

I T S N O W T I M E T O S E E S O M E O F T H I S I N A C T I O N . In the following
we will try a variety of techniques so as to get a better feel for the sorts

The topic of ensembles is briefly noted

later.

Applications in R

of things we might try out.

The Dataset
We will use the wine data set from the UCI Machine Learning data
repository. The goal is to predict wine quality, of which there are 7 values (integers 3-9). We will turn this into a binary classification task to
predict whether a wine is good or not, which is arbitrarily chosen as
6 or higher. After getting the hang of things one might redo the analysis as a multiclass problem or even toy with regression approaches,
just note there are very few 3s or 9s so you really only have 5 values
to work with. The original data along with detailed description can
be found here, but aside from quality it contains predictors such as
residual sugar, alcohol content, acidity and other characteristics of the
wine30 .
The original data is separated into white and red data sets. I have
combined them and created additional variables: color and its numeric version white indicating white or red, and good, indicating scores
greater than or equal to 6 (denoted as Good). The following will show
some basic numeric information about the data.
wine = read.csv("http://www.nd.edu/~mclark19/learn/data/goodwine.csv")
summary(wine)
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

fixed.acidity
Min.
: 3.80
1st Qu.: 6.40
Median : 7.00
Mean
: 7.21
3rd Qu.: 7.70
Max.
:15.90
chlorides
Min.
:0.009
1st Qu.:0.038
Median :0.047
Mean
:0.056
3rd Qu.:0.065
Max.
:0.611
pH
Min.
:2.72
1st Qu.:3.11
Median :3.21
Mean
:3.22
3rd Qu.:3.32
Max.
:4.01
white
Min.
:0.000
1st Qu.:1.000
Median :1.000
Mean
:0.754
3rd Qu.:1.000
Max.
:1.000

volatile.acidity citric.acid
residual.sugar
Min.
:0.08
Min.
:0.000
Min.
: 0.60
1st Qu.:0.23
1st Qu.:0.250
1st Qu.: 1.80
Median :0.29
Median :0.310
Median : 3.00
Mean
:0.34
Mean
:0.319
Mean
: 5.44
3rd Qu.:0.40
3rd Qu.:0.390
3rd Qu.: 8.10
Max.
:1.58
Max.
:1.660
Max.
:65.80
free.sulfur.dioxide total.sulfur.dioxide
density
Min.
: 1.0
Min.
: 6
Min.
:0.987
1st Qu.: 17.0
1st Qu.: 77
1st Qu.:0.992
Median : 29.0
Median :118
Median :0.995
Mean
: 30.5
Mean
:116
Mean
:0.995
3rd Qu.: 41.0
3rd Qu.:156
3rd Qu.:0.997
Max.
:289.0
Max.
:440
Max.
:1.039
sulphates
alcohol
quality
color
Min.
:0.220
Min.
: 8.0
Min.
:3.00
red :1599
1st Qu.:0.430
1st Qu.: 9.5
1st Qu.:5.00
white:4898
Median :0.510
Median :10.3
Median :6.00
Mean
:0.531
Mean
:10.5
Mean
:5.82
3rd Qu.:0.600
3rd Qu.:11.3
3rd Qu.:6.00
Max.
:2.000
Max.
:14.9
Max.
:9.00
good
Bad :2384
Good:4113

I think it would be interesting to have

included characteristics of the people
giving the rating.

Machine Learning

R Implementation
I will use the caret package in R. Caret makes implementation of validation, data partitioning, performance assessment, and prediction
and other procedures about as easy as it can be in this environment.
However, caret is mostly using other R packages31 that have more
information about the specific functions underlying the process, and
those should be investigated for additional information. Check out
the caret home page for more detail. The methods selected here were
chosen for breadth of approach, to give a good sense of the variety of
techniques available.
In addition to caret, its a good idea to use your computers resources as much as possible, or some of these procedures may take
a notably long time, and more so with the more data you have. Caret
will do this behind the scenes, but you first need to set things up. Say,
for example, you have a quad core processor, meaning your processor
has four cores essentially acting as independent CPUs. You can set up R
for parallel processing with the following code, which will allow caret
to allot tasks to three cores simultaneously32 .

In the following, the associated

packages and functions used are:
caret knn
nnet nnet
randomForest randomForest
kernlab ksvm

You typically want to leave at least one

core free so you can do other things.

library(doSNOW)
registerDoSNOW(makeCluster(3, type = "SOCK"))

fixed.acidity

volatile.acidity

citric.acid

residual.sugar

library(corrplot)
corrplot(cor(wine[, -c(13, 15)]), method = "number", tl.cex = 0.5)

I ran regressions to examine the r-squared for each predictor in a

model as if it were the dependent variable predicted by the other input
variables. The highest was for density at over 96%, and further investigation suggested color and either sulfur dioxide are largely captured
by the other variables already also. These will not be considered in the
following models.
Caret has its own partitioning function we can use here to separate
the data into training and test data. There are 6497 total observations

1
0.22

white

quality

alcohol

sulphates

density

total.sulfur.dioxide

free.sulfur.dioxide

chlorides

residual.sugar

citric.acid

volatile.acidity

This data set is large enough to leave a holdout sample, allowing us

to initially search for the best of a given modeling approach over a
grid of tuning parameters specific to the technique. To iterate previous
discussion, we dont want test performance contaminated with the
tuning process. With the best model at t tuning parameter(s), we will
assess performance with prediction on the holdout set.
I also made some decisions to deal with the notable collinearity in
the data, which can severely hamper some methods. We can look at
the simple correlation matrix to start

fixed.acidity

Feature Selection & The Data Partition

0.22 0.32 0.11 0.3 0.280.33 0.46 0.25 0.3 0.1 0.080.49
1 0.38 0.2 0.38 0.350.41 0.27 0.26 0.23 0.040.270.65

0.32 0.38 1
0.11 0.2 0.14

0.14 0.04 0.13 0.2

1 0.13 0.4

0.1 0.33 0.06 0.01 0.09 0.19

0.8

0.6

0.5 0.55 0.270.190.360.04 0.35

0.4

chlorides

0.3 0.38 0.04 0.13 1

free.sulfur.dioxide

0.280.35 0.13 0.4 0.2

total.sulfur.dioxide

0.330.41 0.2

density

0.2 0.28 0.36 0.04 0.4 0.26 0.2 0.51

0.5 0.28 0.72

0.72 0.03 0.150.190.18 0.06 0.47

0.46 0.27 0.1 0.55 0.36 0.03 0.03

0.03 0.240.280.270.04 0.7

0.25 0.26 0.330.27 0.04 0.150.24 0.01

0.01 0.26 0.690.310.39

0.2

0.19 0.12 0.02 0.33

0.4

sulphates

0.3 0.23 0.06 0.19 0.4 0.190.28 0.26 0.19

0.04 0.49

alcohol

0.1 0.040.010.360.260.180.270.69 0.12

0.44 0.03

0.6

0.12

0.8

quality

white

0.080.27 0.09 0.04 0.2 0.06 0.040.31 0.02 0.04 0.44

0.490.65 0.19 0.35 0.51 0.47 0.7 0.390.330.49 0.03 0.12

1
1

Applications in R

of which I will put 80% into the training set. The function createDataPartition will produce indices to use as the training set. In addition to
this, we will normalize the continuous variables to the [0,1] range. For
the training data set, this will be done as part of the training process,
so that any subsets under consideration are scaled separately, but for
the test set we will go ahead and do it now.
library(caret)
set.seed(1234) #so that the indices will be the same when re-run
trainIndices = createDataPartition(wine$good, p = 0.8, list = F)
wanted = !colnames(wine) %in% c("free.sulfur.dioxide", "density", "quality",
"color", "white")
wine_train = wine[trainIndices, wanted] #remove quality and color, as well as density and others
wine_test = wine[-trainIndices, wanted]
residual.sugar

sulphates

total.sulfur.dioxide

volatile.acidity
1.0

Lets take an initial peek at how the predictors separate on the target. In the following Im predicting the pre-possessed data so as to
get the transformed data. Again, well leave the preprocessing to the
training part, but here it will put them on the same scale for visual
display.

0.8
0.6
0.4
0.2
0.0

alcohol

chlorides

citric.acid

fixed.acidity

1.0
0.8
0.6
0.4
0.2
0.0

wine_trainplot = predict(preProcess(wine_train[,-10], method="range"),

wine_train[,-10])
featurePlot(wine_trainplot, wine_train$good, "box")

Yes

Feature

For the training set, it looks like alcohol content, volatile acidity and
chlorides separate most with regard to good classification. While this
might give us some food for thought, note that the figure does not give
insight into interaction effects, which methods such as trees will get at.

k-nearest Neighbors
Consider the typical distance matrix33 that is often used for cluster
analysis of observations34 . If we choose something like Euclidean distance as a metric, each point in the matrix gives the value of how far an
observation is from some other, given their respective values on a set of
variables.
K-nearest neighbors approaches exploit this information for predictive purposes. Let us take a classification example, and k = 5
neighbors. For a given observation xi , find the 5 closest neighbors in
terms of Euclidean distance based on the predictor variables. The class
that is predicted is whatever class the majority of the neighbors are labeled as35 . For continuous outcomes we might take the mean of those
neighbors as the prediction.
So how many neighbors would work best? This is an example of
a tuning parameter, i.e. k, for which we have no knowledge about its
value without doing some initial digging. As such we will select the
tuning parameter as part of the validation process.

See, for example, the function dist in

R.
34

Often referred to as unsupervised

learning as there is not target/dependent
variable.

See the knn.ani function in the animation package for a visual demonstration

Yes

Machine Learning

The caret package provides several techniques for validation such as

k-fold, bootstrap, leave-one-out and others. We will use 10-fold cross
validation. We will also set up a set of values for k to try out36 .
set.seed(1234)
cv_opts = trainControl(method="cv", number=10)
knn_opts = data.frame(.k=c(seq(3, 11, 2), 25, 51, 101)) #odd to avoid ties
results_knn = train(good~., data=wine_train, method="knn",
preProcess="range", trControl=cv_opts,
tuneGrid = knn_opts)
results_knn
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

For whatever tuning parameters are

sought, the train function will expect a
dataframe with a . before the parameter
name as the column name. Note also
you can just specify a tuning length
instead. See the help file for the train
function.

5199 samples
9 predictors
2 classes: 'Bad', 'Good'
Pre-processing: re-scaling to [0, 1]
Resampling: Cross-Validation (10 fold)
Summary of sample sizes: 4679, 4679, 4680, 4679, 4679, 4679, ...
Resampling results across tuning parameters:
k
3
5
7
9
10
20
50
100

Accuracy
0.8
0.7
0.7
0.7
0.7
0.7
0.7
0.7

Kappa
0.5
0.4
0.4
0.4
0.4
0.4
0.4
0.4

Accuracy SD
0.02
0.009
0.02
0.02
0.02
0.02
0.02
0.02

Kappa SD
0.04
0.02
0.04
0.04
0.04
0.04
0.04
0.04

Accuracy was used to select the optimal model using

The final value used for the model was k = 3.

the largest value.

In this case it looks like choosing the nearest five neighbors (k =

3) works best in terms of accuracy. Additional information regards the
variability in the estimate of accuracy, as well as kappa, which can be
seen as a measure of agreement between predictions and true values.
Now that k is chosen, lets see how well the model performs on the test
set.
preds_knn = predict(results_knn, wine_test[,-10])
confusionMatrix(preds_knn, wine_test[,10], positive='Good')
## Confusion Matrix and Statistics
##
##
Reference
## Prediction Bad Good
##
Bad 285 162
##
Good 191 660
##
##
Accuracy : 0.728
##
95% CI : (0.703, 0.752)
##
No Information Rate : 0.633
##
P-Value [Acc > NIR] : 2.76e-13
##

For some reason here and beyond, the

creation of this document rounds the
results of carets train, and changing
various options doesnt do anything.
When you run it yourself you should see
a range of slightly different values, e.g.
between .75 and .77.

##
##
##
##
##
##
##
##
##
##
##
##
##

Applications in R

Kappa : 0.407
Mcnemar's Test P-Value : 0.136
Sensitivity
Specificity
Pos Pred Value
Neg Pred Value
Prevalence
Detection Rate
Detection Prevalence

:
:
:
:
:
:
:

0.803
0.599
0.776
0.638
0.633
0.508
0.656

'Positive' Class : Good

We get a lot of information here, but to focus on accuracy, we get

around 72.8%. The lower bound (and p-value) suggests we are statistically predicting better than the no information rate (i.e., just guessing
the more prevalent not good category), and sensitivity and positive
predictive power are good, though at the cost of being able to distinguish bad wine. Perhaps the other approaches will have more success,
but note that the caret package does have the means to focus on other
metrics such as sensitivity during the training process which might
help. Also feature combination or other avenues might help improve
the results as well.
Additional information reflects the importance of predictors. For
most methods accessed by caret, the default variable importance metric regards the area under the curve or AUC from a ROC analysis with
regard to each predictor, and is model independent. This is then normalized so that the least important is 0 and most important is 100.
Another thing one could do would require more work, as caret doesnt
provide this, but a simple loop could still automate the process. For
a given predictor x, re-run the model without x, and note the decrease (or increase for poor variables) in accuracy that results. One
can then rank order those results. I did so with this problem and notice that only alcohol content and volatile acidity were even useful for
this model. K nearest-neighbors is susceptible to irrelevant information (youre essentially determining neighbors on variables that dont
matter), and one can see this in that, if only those two predictors are
retained, test accuracy is the same (actually a slight increase).
alcohol

dotPlot(varImp(results_knn))

volatile.acidity

chlorides

fixed.acidity

citric.acid

total.sulfur.dioxide

Strengths & Weaknesses

sulphates

Strengths37

residual.sugar

Importance

Intuitive approach.
Robust to outliers on the predictors.

See table 10.1 in Hastie et al. (2009)

for a more comprehensive list for this
and the other methods discussed in this
section.

100

Machine Learning

Weaknesses
Susceptible to irrelevant features.
Susceptible to correlated inputs.
Ability to handle data of mixed types.
Big data. Though approaches are available that help in this regard.

Neural Nets
Neural nets have been around for a long while as a general concept
in artificial intelligence and even as a machine learning algorithm,
and often work quite well. In some sense they can be thought of as
nonlinear regression. Visually however, we can see them as layers of
inputs and outputs. Weighted combinations of the inputs are created
and put through some function (e.g. the sigmoid function) to produce
the next layer of inputs. This next layer goes through the same process
to produce either another layer or to predict the output, which is the
final layer38 . All the layers between the input and output are usually
referred to as hidden layers. If there were no hidden layers then it
becomes the standard regression problem.
One of the issues with neural nets is determining how many hidden
layers and how many hidden units in a layer. Overly complex neural
nets will suffer from high variance will thus be less generalizable, particularly if there is less relevant information in the training data. Along
with the complexity is the notion of weight decay, however this is the
same as the regularization function we discussed in a previous section,
where a penalty term would be applied to a norm of the weights.
A comment about the following: if you are not set up for utilizing
multiple processors the following might be relatively slow. You can
replace the method with nnet and shorten the tuneLength to 3 which
will be faster without much loss of accuracy. Also, the function were
using has only one hidden layer, but the other neural net methods
accessible via the caret package may allow for more, though the gains
in prediction with additional layers are likely to be modest relative
to complexity and computational cost. In addition, if the underlying
function39 has additional arguments, you may pass those on in the
train function itself. Here I am increasing the maxit, or maximum
iterations, argument.
results_nnet = train(good~., data=wine_train, method="avNNet",
trControl=cv_opts, preProcess="range",
tuneLength=5, trace=F, maxit=1000)
results_nnet

Input Layer

Hidden Layer

Output

There can be many output variables in

this approach.

For this example, ultimately the

primary function is nnet in the nnet
package

##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

Applications in R

decay
0
1e-04
0.001
0.01
0.1
0
1e-04
0.001
0.01
0.1
0
1e-04
0.001
0.01
0.1
0
1e-04
0.001
0.01
0.1
0
1e-04
0.001
0.01
0.1

Accuracy
0.7
0.7
0.7
0.7
0.7
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8

Kappa
0.4
0.4
0.4
0.4
0.4
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5

Accuracy SD
0.02
0.02
0.02
0.02
0.02
0.02
0.02
0.01
0.02
0.01
0.02
0.02
0.02
0.02
0.01
0.02
0.02
0.02
0.02
0.01
0.02
0.01
0.02
0.01
0.01

Kappa SD
0.04
0.04
0.04
0.04
0.04
0.04
0.03
0.03
0.04
0.03
0.04
0.04
0.05
0.03
0.03
0.05
0.04
0.03
0.04
0.03
0.05
0.02
0.04
0.03
0.03

Tuning parameter 'bag' was held constant at a value of FALSE

Accuracy was used to select the optimal model using the largest value.
The final values used for the model were size = 9, decay = 1e-04 and bag
= FALSE.

We see that the best model has 9 hidden layer nodes and a decay
parameter of 104 . Typically you might think of how many hidden
units you want to examine in terms of the amount of data you have
(i.e. estimated parameters to N ratio), and here we have a decent
amount. In this situation you might start with very broad values for
the number of inputs (e.g. a sequence by 10s) and then narrow your
focus (e.g. between 20 and 30), but with at least some weight decay
you should be able to avoid overfitting. I was able to get an increase in
test accuracy of about 1.5% using up to 50 hidden units40 .
preds_nnet = predict(results_nnet, wine_test[,-10])
confusionMatrix(preds_nnet, wine_test[,10], positive='Good')
## Confusion Matrix and Statistics

There are some rules of thumb, but

using regularization and cross-validation
is a much better way to guess.

Machine Learning

##
##
Reference
## Prediction Bad Good
##
Bad 295 113
##
Good 181 709
##
##
Accuracy : 0.773
##
95% CI : (0.75, 0.796)
##
No Information Rate : 0.633
##
P-Value [Acc > NIR] : < 2e-16
##
##
Kappa : 0.497
## Mcnemar's Test P-Value : 9.32e-05
##
##
Sensitivity : 0.863
##
Specificity : 0.620
##
Pos Pred Value : 0.797
##
Neg Pred Value : 0.723
##
Prevalence : 0.633
##
Detection Rate : 0.546
##
Detection Prevalence : 0.686
##
##
'Positive' Class : Good
##

We note improved prediction with the neural net model relative to the k-nearest neighbors approach, with increases in accuracy
(77.35%), sensitivity, specificity etc.

Strengths & Weaknesses

Strengths
Good prediction generally.
Incorporating the predictive power of different combinations of
inputs.
Some tolerance to correlated inputs.
Weaknesses
Susceptible to irrelevant features.
Not robust to outliers.
Big data with complex models.

Trees & Forests

Classification and regression trees provide yet another and notably
different approach to prediction. Consider a single input variable and
binary dependent variable. We will search all values of the input to
find a point where, if we partition the data at that point, it will lead to

Applications in R

the best classification accuracy. So for a single variable whose range

might be 1 to 10, we find that a cut at 5.75 results in the best classification if all observations greater than or equal to 5.75 are classified as
positive and the rest negative. This general approach is fairly straightforward and conceptually easy to grasp, and it is because of this that
tree approaches are appealing.
Now lets add a second input, also on a 1 to 10 range. We now
might find that even better classification results if, upon looking at the
portion of data regarding those greater than or equal to 5.75, that we
only classify positive if they are also less than 3 on the second variable.
At right is a hypothetical tree reflecting this.
The example tree here is based on the wine training data set. It is
interpreted as follows. If the alcohol content is greater than 10.63 %,
a wine is classified as good41 . For those less than 10.63, if its volatile
acidity is also less than .25, they are also classified as good, and of
the remaining observations, if they are at least more than 9.85% (i.e.
volatility >.25, alcohol between 9.85 and 10.625), they also get classified as good. Any remaining observations are classified as bad wines.
Unfortunately a single tree, while highly interpretable, does pretty
poorly for predictive purposes. In standard situations we will instead
use the power of many trees, i.e. a forest, based on repeated sampling
of the original data. So if we create 1000 new training data sets based
on random samples of the original data (each of size N, i.e. a bootstrap
of the original data set), we can run a tree for each, and assess the
predictions each tree would produce for the observations for a hold
out set (or simply those observations which werent selected during the
sampling process, the out-of-bag sample), in which the new data is
run down the tree to obtain predictions. The final class prediction for
an observation is determined by majority vote across all trees.
Random forests are referred to as an ensemble method, one that
is actually a combination of many models, and there are others well
mention later. In addition there are other things to consider, such
as how many variables to make available for consideration at each
split, and this is the tuning parameter of consequence here in our use
of caret (called mtry). In this case we will investigate subsets of 2
through 6 possible predictors. With this value determined via crossvalidation, we can apply the best approach to the hold out test data
set.
Theres a lot going on here to be sure: there is a sampling process
for cross-validation, there is resampling to produce the forest, there is
random selection of mtry predictor variables etc. But we are in the end
just harnessing the power of many trees, any one of which would be
highly interpretable.

X1 >=5.75

Negative

X2 < 3

Negative

Positive

alcohol|< 10.625

volatile.acidity < 0.2525

Good

alcohol < 9.85

Bad

Good

Results from the tree package.

Color me unsurprised by this finding.

Good

Machine Learning

set.seed(1234)
rf_opts = data.frame(.mtry=c(2:6))
results_rf = train(good~., data=wine_train, method="rf",
preProcess='range',trControl=cv_opts, tuneGrid=rf_opts,
n.tree=1000)
results_rf
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

Accuracy
0.8
0.8
0.8
0.8
0.8

Kappa
0.6
0.6
0.6
0.6
0.6

Accuracy SD
0.02
0.02
0.02
0.02
0.02

Kappa SD
0.04
0.04
0.04
0.04
0.04

Accuracy was used to select the optimal model using

The final value used for the model was mtry = 3.

the largest value.

The initial results look promising with mtry = 3 producing the best
initial result. Now for application to the test set.
preds_rf = predict(results_rf, wine_test[,-10])
confusionMatrix(preds_rf, wine_test[,10], positive='Good')
## Confusion Matrix and Statistics
##
##
Reference
## Prediction Bad Good
##
Bad 333
98
##
Good 143 724
##
##
Accuracy : 0.814
##
95% CI : (0.792, 0.835)
##
No Information Rate : 0.633
##
P-Value [Acc > NIR] : < 2e-16
##
##
Kappa : 0.592
## Mcnemar's Test P-Value : 0.00459
##
##
Sensitivity : 0.881
##
Specificity : 0.700
##
Pos Pred Value : 0.835
##
Neg Pred Value : 0.773
##
Prevalence : 0.633
##
Detection Rate : 0.558
##
Detection Prevalence : 0.668
##
##
'Positive' Class : Good
##

Applications in R

This is our best result so far with 81.43% accuracy, with a lower
bound well beyond the 63% wed have guessing. Random forests do
not suffer from some of the data specific issues that might be influencing the other approaches, such as irrelevant and correlated predictors,
and furthermore benefit from the combined information of many models. Such performance increases are not a given, but random forests
are generally a good method to consider given their flexibility.
Incidentally, the underlying randomForest function here allows one
to assess variable importance in a different manner42 , and there are
other functions used by caret that can produce their own metrics also.
In this case, randomForest can provide importance based on a version
of the decrease in inaccuracy approach we talked before (as well as
another index known as gini impurity). The same two predictors are
found to be most important and notably more than others- alcohol and
volatile.acidity.

Our previous assessment was model

independent.

Strengths & Weaknesses

Strengths
A single tree is highly interpretable.
Tolerance to irrelevant features.
Some tolerance to correlated inputs.
Good with big data.
Handling of missing values.
Weaknesses

Relatively less predictive in many situations.

Cannot work on (linear) combinations of features.

lab
class1

class2

Support Vector Machines

Support Vector Machines (SVM) will be our last example, and is perhaps the least intuitive. SVMs seek to map the input space to a higher
dimension via a kernel function, and in that transformed feature space,
find a hyperplane that will result in maximal separation of the data.
To better understand this process, consider the example to the right
of two inputs, x and y. Cursory inspection shows no easy separation
between classes. However if we can map the data to a higher dimension43 , shown in the following graph, we might find a more clear separation. Note that there are a number of choices in regard to the kernel
function that does the mapping, but in that higher dimension, the

Note that we regularly do this sort of

thing in more mundane circumstances.
For example, we map an Nxp matrix
to an NxN matrix when we compute a
distance matrix for cluster analysis.

Machine Learning

decision boundary is chosen which will result in maximum distance

(largest margin) between classes (following figures, zoom in to see
the margin on the second plot). Real data will not be so clean cut, and
total separation impossible, but the idea is the same.
set.seed(1234)
results_svm = train(good~., data=wine_train, method="svmLinear",
preProcess="range", trControl=cv_opts, tuneLength=5)
results_svm

class1
class2

##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##

Accuracy
0.7
0.7
0.7
0.7
0.7

Kappa
0.4
0.4
0.4
0.4
0.4

Accuracy SD
0.02
0.02
0.02
0.02
0.02

class1

Kappa SD
0.05
0.05
0.05
0.04
0.05

Accuracy was used to select the optimal model using

The final value used for the model was C = 0.5.

class2

the largest value.

preds_svm = predict(results_svm, wine_test[,-10])

confusionMatrix(preds_svm, wine_test[,10], positive='Good')
## Confusion Matrix and Statistics
##
##
Reference
## Prediction Bad Good
##
Bad 268 123
##
Good 208 699
##
##
Accuracy : 0.745
##
95% CI : (0.72, 0.769)
##
No Information Rate : 0.633
##
P-Value [Acc > NIR] : < 2e-16
##
##
Kappa : 0.43
## Mcnemar's Test P-Value : 3.89e-06
##
##
Sensitivity : 0.850
##
Specificity : 0.563
##
Pos Pred Value : 0.771
##
Neg Pred Value : 0.685
##
Prevalence : 0.633
##
Detection Rate : 0.539
##
Detection Prevalence : 0.699
##
##
'Positive' Class : Good
##

Applications in R

Results for the initial support vector machine do not match the
random forest for this data set, with accuracy of 74.5%. However, you
might choose a different kernel than the linear one used here, as well
as tinker with other options.

Strengths & Weaknesses

Strengths
Good prediction in a variety of situations.
Can utilize predictive power of linear combinations of inputs.
Weaknesses
Very black box.
Computational scalability.
Natural handling of mixed data types.

Other
I N T H I S S E C T I O N I N O T E S O M E O T H E R T E C H N I Q U E S one may come
across and others that will provide additional insight into machine
learning applications.

Unsupervised Learning
Unsupervised learning generally speaking involves techniques in which
we are utilizing unlabeled data. In this case we have our typical set of
features we are interested in, but no particular response to map them
to. In this situation we are more interested in the discovery of structure
within the data.

Clustering
Many of the techniques used in unsupervised are commonly taught
in various applied disciplines as various forms of "cluster" analysis.
The gist is we are seeking an unknown class structure rather than
seeing how various inputs relate to a known class structure. Common
techniques include k-means, hierarchical clustering, and model based
approaches (e.g. mixture models).

Machine Learning

Latent Variable Models

Sometimes the desire is to reduce the dimensionality of the inputs to a
more manageable set of information. In this manner we are thinking
that much of the data can be seen as having only a few sources of
variability, often called latent variables or factors. Again, this takes
familiar forms such as principal components and ("exploratory") factor
analysis, but would also include independence components analysis
and partial least squares techniques. Note also that these can be part
of a supervised technique (e.g. principal components regression) or
the main focus of analysis (as with latent variable models in structural
equation modeling).

LV1

LV2

Graphical Structure
Other techniques are available to understand structure among observations or features. Among the many approaches is the popular network
analysis, where we can obtain similarities among observations and examine visually the structure of those data points, where observations
are placed closer together that are more similar in nature. In still other
situations, we arent so interested in the structure as we are in modeling the relationships and making predictions from the correlations of
inputs.

Coburn
DeMint
Inhofe

Vitter
Enzi Allard
Bunning
Burr Ensign DoleGregg

Kyl

Chambliss
Cornyn
Hagel Sessions
Sununu
Shelby
Hatch McConnell
Corker Lott
Craig Isakson
Martinez
Graham
Crapo
Hutchison
Cochran
Roberts
Bond
Domenici
Grassley
McCain
Brownback
Bennett Thune
Alexander
Murkowski
Warner
Stevens
Voinovich Lugar

Coleman
Smith

Specter

Imputation
Collins

We can also use these techniques when we are missing data as a means
to impute the missing values44 . While many are familiar with this
problem and standard techniques for dealing with it, it may not be
obvious that ML techniques may also be used. For example, both knearest neighbors and random forest techniques have been applied to
imputation.
Beyond this we can infer values that are otherwise unavailable in a
different sense. Consider Netflix, Amazon and other sites that suggest
various products based on what you already like or are interested in.
In this case the suggested products have missing values for the user
which are imputed or inferred based on their available data and other
consumers similar to them who have rated the product in question.
Such recommender systems are widely used these days.

Ensembles
In many situations we can combine the information of multiple models
to enhance prediction. This can take place within a specific technique,

Snowe

Nelson (NE)
Pryor Salazar
Baucus
Lincoln Carper
Kennedy
Kerry
Tester
Cantwell Lieberman
Klobuchar Feinstein
Mikulski
Reid
Landrieu Leahy Akaka
Wyden
Sanders Cardin
Nelson (FL)
Murray
Byrd
CaseyHarkinKohl
Conrad
Menendez
Schumer
Lautenberg
Reed
DorganBingaman
Whitehouse
Webb
DurbinDodd
BrownBiden
Clinton
Bayh
Rockefeller
Feingold
Boxer LevinObama
Inouye
Stabenow
McCaskill

Applications in R

e.g. random forests, or between models that utilize different techniques. I will discuss some standard techniques, but there are a great
variety of forms in which model combination might take place.

Bagging
Bagging, or bootstrap aggregation, uses bootstrap sampling to create
many data sets on which a procedure is then performed. The final
prediction is based on an average of all the predictions made for each
observation. In general, bagging helps reduce the variance while leaving bias unaffected. A conceptual outline of the procedure is provided.
Model Generation
For B number of iterations:
1. Sample N observations with replacement B times to create B data
sets of size N.
2. Apply the learning technique to each of B data sets to create t models.
3. Store the t results.
Classification
For each of t number of models:
1. Predict the class of N observations of the original data set.
2. Return the class predicted most often.

Boosting
With boosting we take a different approach to refitting models. Consider a classification task in which we start with a basic learner and
apply it to the data of interest. Next the learner is refit, but with more
weight (importance) given to misclassified observations. This process
is repeated until some stopping rule is reached. An example of the
AdaBoost algorithm is provided (in the following I is the indicator
function).
Set initial weights wi to 1/N.
for m = 1 : M

Fit a classifier m with given weights to the data resulting in

(m)
predictions f i
that minimizes some loss function.
N

Compute the error rate errm =

(m)

I( y i 6 = f i

i =1

(m)

i =1

Compute m = log[(1 errm )/errm ]

Machine Learning

(m)

Set wi wi exp[m I(yi 6= f i

)]

}
M

Return sgn[ m f (m) ]

m =1

Boosting can be applied to a variety of tasks and loss functions, and

in general is highly resistant to overfitting.

Stacking
Stacking is a method that can generalize beyond a single fitting technique, though it can be applied in a fashion similar to boosting for a
single technique. While the term can refer to a specific technique, here
we will use it broadly to mean any method to combine models of different forms. Consider the four approaches we demonstrated earlier:
k-nearest neighbors, neural net, random forest, and the support vector
machine. We saw that they do not have the same predictive accuracy,
though they werent bad in general. Perhaps by combining their respective efforts, we could get even better prediction than using any
particular one.
The issue then how we might combine them. We really dont have
to get too fancy with it, and can even use a simple voting scheme as in
bagging. For each observation, note the predicted class on new data
across models. The final prediction is the class that receives the most
votes. Another approach would be to use a weighted vote, where the
votes are weighted by their respective accuracies.
Another approach would use the predictions on the test set to create a data set of just the predicted probabilities from each learning
scheme. We can then use this data to train a meta-learner using the
test labels as the response. With the final meta-learner chosen, we then
retrain the original models on the entire data set (i.e. including the
test data). In this manner the initial models and the meta-learner are
trained separately and you get to eventually use the entire data set to
train the original models. Now when new data becomes available, you
feed them to the base level learners, get the predictions, and then feed
the predictions to the meta-learner for the final prediction.

Feature Selection & Importance

We hit on this topic some before, but much like there are a variety of
ways to gauge performance, there are different approaches to select
features and/or determine their importance. Invariably feature selection takes place from the outset when we choose what data to collect
in the first place. Hopefully guided by theory, in other cases it may be
restricted by user input, privacy issues, time constraints and so forth.

Applications in R

But once we obtain the initial data set however, we may still want to
trim the models under consideration.
In standard approaches we might have in the past used forward
or other selection procedure, or perhaps some more explicit model
comparison approach. Concerning the content here, take for instance
the lasso regularization procedure we spoke of earlier. Less important
variables may be shrunk entirely to zero, and thus feature selection is
an inherent part of the process, and is useful in the face of many, many
predictors, sometimes outnumbering our sample points. As another
example, consider any particular approach where the importance
metric might be something like the drop in accuracy when the variable
is excluded.
Variable importance was given almost full weight in the discussion
of typical applied research in the past, based on statistical significance
results from a one-shot analysis, and virtually ignorant of prediction
on new data. We still have the ability to focus on feature performance
with ML techniques, while shifting more of the focus toward prediction
at the same time. For the uninitiated, it might require new ways of
thinking about how one measures importance though.

Textual Analysis
In some situations the data of interest is not in a typical matrix form
but in the form of textual content, i.e. a corpus of documents (loosely
defined). In this case, much of the work (like in most analyses but perhaps even more so) will be in the data preparation, as text is rarely if
ever in a ready-to-analyze state. The eventual goals may include using
the word usage in the prediction of an outcome, perhaps modeling the
usage of select terms, or examining the structure of the term usage
graphically as in a network model. In addition, machine learning processes might be applied to sounds (acoustic data) to discern the speech
characteristics and other information.

Bayesian Approaches
It should be noted that the approaches outlined in this document
are couched in the frequentist tradition. But one should be aware
that many of the concepts and techniques would carry over into the
Bayesian perspective, and even some machine learning techniques
might only be feasible or make more sense within the Bayesian framework (e.g. online learning).

Machine Learning

More Stuff
Aside from what has already been noted, there still exists a great many
applications for ML such as data set shift 45 , deep learning46 , semisupervised learning47 , online learning48 , and many more.

Summary
Cautionary Notes
A standard mantra in machine learning and statistics generally is that
there is no free lunch. All methods have certain assumptions, and if
those dont hold the results will be problematic at best. Also, even if
in truth learner A is better than B, B can often outperform A in the
finite situations we actually deal with in practice. Furthermore, being
more complicated doesnt mean a technique is better. As previously
noted, incorporating regularization and cross-validation goes a long
way toward to improving standard techniques, and they may perform
quite well in some situations.

Used when fundamental changes

occur between the data a learner is
trained on and the data coming in for
further analysis.
46
Learning at different levels of representation, e.g. from an image regarding
a scene to the concepts used to describe
it.
47
Learning with both labeled and
unlabeled data.
48
Learning from a continuous stream of
data.

Some Guidelines
Here are some thoughts to keep in mind, though these may be applicable to applied statistical practice generally.
More data beats a cleverer algorithm, but a lot of data is not enough by
itself49 .

Domingos (2012)

Avoid overfitting.
Let the data speak for itself.
"Nothing is more practical than a good theory."50
While getting used to ML, it might be best to start from simpler approaches and then work towards more black box ones that require
more tuning. For example, naive Bayes logistic regression knn
svm.
Drawing up a visual path of your process is a good way to keep your
analysis on the path to your goal. Some programs can even make this
explicit (e.g. RapidMiner, Weka).
Keep the tuning parameter/feature selection process separate from the
final test process for assessing error.
Learn multiple models, selecting the best or possibly combining them.

Kurt Lewin, and iterated by V. Vapnik

for the machine learning context.

Applications in R

Conclusion
It is hoped that this document sheds some light on some areas that
might otherwise be unfamiliar to some applied researchers. The field
of statistics has rapidly evolved over the past two decades. The tools
available are myriad, and expanding all the time. Rather than being
overwhelmed, one should embrace the choice available, and have some
fun with your data.

Machine Learning

Brief Glossary of Common

Terms
bias could mean the intercept (e.g. in neural nets), typically refers to
the bias in bias-variance decomposition
regularization, penalization, shrinkage The process of adding a penalty
to the size of coefficients, thus shrinking them towards zero but
resulting in less overfitting (at an increase to bias)
classifier specific model or technique (i.e. function) that maps observations to classes
confusion matrix a table of predicted class membership vs. true class
membership
hypothesis a specific model h( x ) of all possible in the hypothesis space
H
input, feature, attribute independent variable, predictor variable, column
instance, example observation, row
learning model fitting
machine learning a form of statistics utilizing various algorithms with a
goal to generalize to new data situations
supervised has a dependent variable
target, label dependent variable, response, the outcome of interest
unsupervised no dependent variable; think clustering, PCA etc.
weights coefficients, parameters

Applications in R

References
Breiman, L. (2001). Statistical modeling: The two cultures (with
comments and a rejoinder by the author). Statistical Science,
16(3):199231. Mathematical Reviews number (MathSciNet):
MR1874152.
Domingos, P. (2012). A few useful things to know about machine
learning. Commun. ACM, 55(10).
Harrell, F. E. (2001). Regression Modeling Strategies: With Applications
to Linear Models, Logistic Regression, and Survival Analysis. Springer.
Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of
Statistical Learning: Data Mining, Inference, and Prediction, Second
Edition. Springer, 2nd ed. 2009. corr. 10th printing. edition.
Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective.
The MIT Press.
Wood, S. N. (2006). Generalized additive models: an introduction with
R, volume 66. CRC Press.
I had a lot of sources in putting this together, but I note these in particular as I feel they can either provide the appropriate context to begin,
help with the transition from more standard approaches, or serve as a
definitive reference for various methods.

AHA ACLS Precourse Self Assessment Answers 2023
92% (24)
AHA ACLS Precourse Self Assessment Answers 2023
32 pages
USMLE World Step 3 High Yield Notes 90 Pages
95% (58)
USMLE World Step 3 High Yield Notes 90 Pages
90 pages
ACLS Exam Version A
98% (41)
ACLS Exam Version A
11 pages
ACLS Post Test Answer Key 2024
92% (12)
ACLS Post Test Answer Key 2024
12 pages
ACLS Pretest Answers 2024
94% (17)
ACLS Pretest Answers 2024
29 pages
Les Miserables School Edition
100% (10)
Les Miserables School Edition
309 pages
The Manual of Free Energy Devices and Systems (1991)
93% (208)
The Manual of Free Energy Devices and Systems (1991)
128 pages
Seventy-Eight Degrees of Wisdom - A Book of Tarot, Revised - Rachel Pollack
98% (41)
Seventy-Eight Degrees of Wisdom - A Book of Tarot, Revised - Rachel Pollack
370 pages
Healing Prayer by Dr. Cindy Trimm
94% (32)
Healing Prayer by Dr. Cindy Trimm
10 pages
Byford Dolphin - Text
43% (7)
Byford Dolphin - Text
8 pages
Interventions Cheat Sheet
100% (5)
Interventions Cheat Sheet
3 pages
TNCC 9th Exam
100% (6)
TNCC 9th Exam
35 pages
Joe Tippens' Protocol and Scientific Backing PDF
86% (7)
Joe Tippens' Protocol and Scientific Backing PDF
2 pages
The One-Minute Cure Synopsis
97% (29)
The One-Minute Cure Synopsis
4 pages
Bessel Van Der Kolk - The Body Keeps The Score - Brain, Mind, and Body in The Healing of Trauma-Penguin (2014)
100% (12)
Bessel Van Der Kolk - The Body Keeps The Score - Brain, Mind, and Body in The Healing of Trauma-Penguin (2014)
490 pages
DR Sebi Cure For Herpes - Def
82% (11)
DR Sebi Cure For Herpes - Def
95 pages
House of Leaves - Mark Z Danielewski
81% (21)
House of Leaves - Mark Z Danielewski
750 pages
Lundy - Why Does He Do That PDF
79% (42)
Lundy - Why Does He Do That PDF
1,020 pages
Anatomy & Physiology Coloring - Brain Answers
100% (6)
Anatomy & Physiology Coloring - Brain Answers
5 pages
Trauma and Dissociation Informed Internal Family Systems How To Successfully Treat C PTSD and Disso
100% (2)
Trauma and Dissociation Informed Internal Family Systems How To Successfully Treat C PTSD and Disso
221 pages
Kiss Your Dentist Goodbye A Do It Yourself Mouth Care System For Healthy - Clean Gums and Teeth by El
100% (14)
Kiss Your Dentist Goodbye A Do It Yourself Mouth Care System For Healthy - Clean Gums and Teeth by El
210 pages
Haunted Self
100% (15)
Haunted Self
440 pages
Do You Like Big Girls V01
18% (28)
Do You Like Big Girls V01
161 pages
Viktor Frankl - Man S Search For Meaning
94% (33)
Viktor Frankl - Man S Search For Meaning
99 pages
Encyclopedia of Medicinal Herbs
95% (22)
Encyclopedia of Medicinal Herbs
260 pages
Principles of ICD 10 Coding PDF
100% (14)
Principles of ICD 10 Coding PDF
473 pages
The Cure For All Diseases
98% (121)
The Cure For All Diseases
631 pages
The Hold Me Tight Workbook A Couples Guide For A Lifetime of Love (Dr. Sue Johnson) (Z-Library)
100% (3)
The Hold Me Tight Workbook A Couples Guide For A Lifetime of Love (Dr. Sue Johnson) (Z-Library)
187 pages
Complete Mouth Care System
No ratings yet
Complete Mouth Care System
24 pages
Battlefield WKBK
100% (10)
Battlefield WKBK
85 pages
The Preppers Medical Handbook by William Forgey
100% (14)
The Preppers Medical Handbook by William Forgey
321 pages
Adult ADHD Test 18 Question ADHD-ASRS-v1-1 Adhd-Npf - Com Quality Approved
100% (14)
Adult ADHD Test 18 Question ADHD-ASRS-v1-1 Adhd-Npf - Com Quality Approved
3 pages
The Ultimate Guide To Red Light - Whitten, Ari
100% (12)
The Ultimate Guide To Red Light - Whitten, Ari
199 pages
Adrenochrome
80% (5)
Adrenochrome
2 pages
Borax - The Inexpensive Detox, Arthritis, Osteoporosis and Mycoplasma Cure
90% (10)
Borax - The Inexpensive Detox, Arthritis, Osteoporosis and Mycoplasma Cure
10 pages
DSM Iv
100% (33)
DSM Iv
915 pages
The Borax Conspiracy
100% (5)
The Borax Conspiracy
3 pages
Nikola Tesla - The Inventions, Researches and Writings of Nikola Tesla
100% (39)
Nikola Tesla - The Inventions, Researches and Writings of Nikola Tesla
509 pages
DSM 5 Chart
93% (30)
DSM 5 Chart
2 pages
The NCLEX-RN Exam Study Guide Premium Edition - Proven Methods To Pass The NCLEX-RN Examination With Confidence (Belinelli, Rachel Media Group, Scientia) PDF
82% (11)
The NCLEX-RN Exam Study Guide Premium Edition - Proven Methods To Pass The NCLEX-RN Examination With Confidence (Belinelli, Rachel Media Group, Scientia) PDF
509 pages
Mark Sloan, Ray Peat, Raymond Peat - The Ultimate Guide to Methylene Blue_ Remarkable Hope for Depression, COVID, AIDS & Other Viruses, Alzheimer’s, Autism, Cancer, Heart Disease, ... Targeting Mitoch
100% (15)
Mark Sloan, Ray Peat, Raymond Peat - The Ultimate Guide to Methylene Blue_ Remarkable Hope for Depression, COVID, AIDS & Other Viruses, Alzheimer’s, Autism, Cancer, Heart Disease, ... Targeting Mitoch
171 pages
What Is My Trauma Type - 4F's PDF
100% (22)
What Is My Trauma Type - 4F's PDF
1 page
Circumcision Photo Guide: What To Expect After Surgery: Patient and Family Education
17% (12)
Circumcision Photo Guide: What To Expect After Surgery: Patient and Family Education
2 pages
Cognitive Therapy Therapy Techniques For Retraining Your Brain - Jason M. Satterfield
97% (71)
Cognitive Therapy Therapy Techniques For Retraining Your Brain - Jason M. Satterfield
222 pages
Pain Secrets
100% (9)
Pain Secrets
790 pages
Anatomy Coloring Book
100% (10)
Anatomy Coloring Book
215 pages
8th Grade Math Textbook CMS
100% (3)
8th Grade Math Textbook CMS
338 pages
Project Looking Glass: QAnon Post 3094 Ir0nbelly
93% (28)
Project Looking Glass: QAnon Post 3094 Ir0nbelly
89 pages
The Complete Book of Family Homeopathic Medicine (PDFDrive) PDF
91% (11)
The Complete Book of Family Homeopathic Medicine (PDFDrive) PDF
184 pages
7 Day Keto Meal Plan
100% (3)
7 Day Keto Meal Plan
2 pages
Psychology 1
83% (12)
Psychology 1
755 pages
Top Secret Medicine
100% (29)
Top Secret Medicine
9 pages
Ehlers-Danlos Syndrome: A Multidisciplinary Approach
100% (4)
Ehlers-Danlos Syndrome: A Multidisciplinary Approach
370 pages
Precalculus With Trigonometry
94% (31)
Precalculus With Trigonometry
791 pages
Exponential and Logarithmic Functions
No ratings yet
Exponential and Logarithmic Functions
8 pages
Computation and Measurement
No ratings yet
Computation and Measurement
12 pages
Differential Operator
No ratings yet
Differential Operator
4 pages
Example 9
No ratings yet
Example 9
3 pages
EXAMPLE 7 The Following: Manipulate
No ratings yet
EXAMPLE 7 The Following: Manipulate
3 pages
Contraction of Tensors
No ratings yet
Contraction of Tensors
3 pages
Solution 2: 7.3 Trigonometric Functions
No ratings yet
Solution 2: 7.3 Trigonometric Functions
2 pages
The QR Decomposition
No ratings yet
The QR Decomposition
3 pages
The Cholesky Decomposition
No ratings yet
The Cholesky Decomposition
4 pages
Water Tank 10,000 L 10 Control Panel: Top View
No ratings yet
Water Tank 10,000 L 10 Control Panel: Top View
1 page
Integer Programming
No ratings yet
Integer Programming
5 pages
Bosch
No ratings yet
Bosch
128 pages
The Problem Can Be Translated Into Python With The
No ratings yet
The Problem Can Be Translated Into Python With The
2 pages
Sakho I. Introduction To Quantum Mechanics 2. 2020 PDF
No ratings yet
Sakho I. Introduction To Quantum Mechanics 2. 2020 PDF
297 pages
The Arbitrage Pricing Theory Model
No ratings yet
The Arbitrage Pricing Theory Model
3 pages
Force and
No ratings yet
Force and
6 pages
Note That The Equations of Steps 1 and 2
No ratings yet
Note That The Equations of Steps 1 and 2
2 pages
And Vertical Forces Acting
No ratings yet
And Vertical Forces Acting
6 pages
Calculations and Numerical Accuracy
No ratings yet
Calculations and Numerical Accuracy
4 pages
The Forces On The
No ratings yet
The Forces On The
8 pages
Tugboat A Is Pushing
No ratings yet
Tugboat A Is Pushing
9 pages
2 Dynamics
No ratings yet
2 Dynamics
4 pages
What Parents and Teachers Should Know About Adhd: Center For Children and Families
No ratings yet
What Parents and Teachers Should Know About Adhd: Center For Children and Families
2 pages
Trigger Point Book
No ratings yet
Trigger Point Book
84 pages
Healing Codes
96% (27)
Healing Codes
17 pages
Great Sex Guide
59% (46)
Great Sex Guide
122 pages
Hulda - Clark.the - Cure.for - All.advanced - Cancers 1999
100% (17)
Hulda - Clark.the - Cure.for - All.advanced - Cancers 1999
633 pages
BPD and Olanzapine
100% (1)
BPD and Olanzapine
3 pages
EKG Study Guide
100% (7)
EKG Study Guide
45 pages
The Pocket Manual of OMT Osteopathic Manipulative Treatment For Physicians 2E 2010 PDF Koudiai VRG PDF
100% (2)
The Pocket Manual of OMT Osteopathic Manipulative Treatment For Physicians 2E 2010 PDF Koudiai VRG PDF
346 pages
Dissociative Identity Disorder
No ratings yet
Dissociative Identity Disorder
1 page
LPN Notes
100% (5)
LPN Notes
223 pages
Take Back Your Life Now - Powerful Guide For Recovering From Narcissistic Abuse
No ratings yet
Take Back Your Life Now - Powerful Guide For Recovering From Narcissistic Abuse
63 pages
Eds Diagnostic Tool
100% (2)
Eds Diagnostic Tool
27 pages
Pharmacology Cheat Sheet - Generic Drug Stems - Nurseslabs
100% (1)
Pharmacology Cheat Sheet - Generic Drug Stems - Nurseslabs
12 pages