Introduction to Machine

Learning with Scikit-Learn

Plan of Study
- Preface
- What is Machine Learning?
- An Architecture for ML Data Products
- What is Scikit-Learn?
- Data Handling and Loading
- Model Evaluation
- Regressions
- Classification
- Clustering
- Workshop
 Preface

Why I didn’t want to teach a Machine

Learning course for DDL.
 Smoothing Planning

Probability Computer Vision

Normalization Reinforcement

Distributions Neural Models

Artificial
Statistics
Bayes Theorem Intelligence Computer Vision

Regression Anomaly Detection

Logits
Natural Language Processing
Entropy

Big Data
Computer
Data Mining Science Optimization

Function Approximation Graph Algorithms

Is Machine Learning a one semester course?

 Statistician Expert Programmer

Machine Learning
Practitioner Domains

Machine Learning
Hacker Practitioner Continuum Academic
What is Machine Learning?
Learning by Example
Given a bunch of examples (data) extract a
meaningful pattern upon which to act.

Problem Domain Machine Learning Class

Infer a function from labeled data Supervised learning

Find structure of data without feedback Unsupervised learning

Interact with environment towards goal Reinforcement learning

Context to Data Mining & Statistics

Statistics Data Mining

Users

Data

Machine Learning
Computer
Science
Machine or App
Types of Algorithms by Output
Input training data to fit a model which is then
used to predict incoming inputs into ...

Type of Output Algorithm Category

Output is one or more discrete classes Classification (supervised)

Output is continuous Regression (supervised)

Output is membership in a similar group Clustering (unsupervised)

Output is the distribution of inputs Density Estimation

Output is simplified from higher dimensions Dimensionality Reduction

Classification

Given labeled input data (with two or more labels), fit a

function that can determine for any input, what the label is.
Regression

Given continuous input data fit a function that is able to

predict the continuous value of input given other data.
Clustering

Given data, determine a pattern of associated data points

or clusters via their similarity or distance from one another.
Dimensions and Features
In order to do machine learning you need a data set containing
instances (examples) that are composed of features from which
you compose dimensions.

Instance: a single data point or example composed of fields

Feature: a quantity describing an instance
Dimension: one or more attributes that describe a property

from sklearn.datasets import load_digits

digits = load_digits()

X = digits.data # X.shape == (n_samples, n_features)

y = digits.target # y.shape == (n_samples,)
Your Task
Given a data set of instances of size N, create
a model that is fit from the data (built) by
extracting features and dimensions. Then use
that model to predict outcomes …
1. Data Wrangling (normalization, standardization, imputing)
2. Feature Analysis/Extraction
3. Model Selection/Building
4. Model Evaluation
5. Operationalize Model
A Tour of Machine Learning
Algorithms
Models: Instance Methods
Compare instances in data set with a similarity
measure to find best matches.
- Suffers from curse of dimensionality.
- Focus on feature representation and
similarity metrics between instances

● k-Nearest Neighbors (kNN)

● Self-Organizing Maps (SOM)
● Learning Vector Quantization (LVQ)
Models: Regression
Model relationship of independent variables, X
to dependent variable Y by iteratively
optimizing error made in predictions.

● Ordinary Least Squares

● Logistic Regression
● Stepwise Regression
● Multivariate Adaptive Regression Splines (MARS)
● Locally Estimated Scatterplot Smoothing (LOESS)
Models: Regularization Methods
Extend another method (usually regression),
penalizing complexity (minimize overfit)
- simple, popular, powerful
- better at generalization

● Ridge Regression
● LASSO (Least Absolute Shrinkage & Selection Operator)
● Elastic Net
Models: Decision Trees
Model of decisions based on data attributes.
Predictions are made by following forks in a
tree structure until a decision is made. Used for
classification & regression.

● Classification and Regression Tree (CART)

● Decision Stump
● Random Forest
● Multivariate Adaptive Regression Splines (MARS)
● Gradient Boosting Machines (GBM)
Models: Bayesian
Explicitly apply Bayes’ Theorem for
classification and regression tasks. Usually by
fitting a probability function constructed via the
chain rule and a naive simplification of Bayes.

● Naive Bayes
● Averaged One-Dependence Estimators (AODE)
● Bayesian Belief Network (BBN)
Models: Kernel Methods
Map input data into higher dimensional vector
space where the problem is easier to model.
Named after the “kernel trick” which computes
the inner product of images of pairs of data.

● Support Vector Machines (SVM)

● Radial Basis Function (RBF)
● Linear Discriminant Analysis (LDA)
Models: Clustering Methods
Organize data into into groups whose members
share maximum similarity (defined usually by a
distance metric). Two main approaches:
centroids and hierarchical clustering.

● k-Means
● Affinity Propegation
● OPTICS (Ordering Points to Identify Cluster Structure)
● Agglomerative Clustering
Models: Artificial Neural Networks
Inspired by biological neural networks, ANNs are
nonlinear function approximators that estimate
functions with a large number of inputs.
- System of interconnected neurons that activate
- Deep learning extends simple networks recursively

● Perceptron
● Back-Propagation
● Hopfield Network
● Restricted Boltzmann Machine (RBM)
● Deep Belief Networks (DBN)
Models: Ensembles
Models composed of multiple weak models that
are trained independently and whose outputs
are combined to make an overall prediction.

● Boosting
● Bootstrapped Aggregation (Bagging)
● AdaBoost
● Stacked Generalization (blending)
● Gradient Boosting Machines (GBM)
● Random Forest
Models: Other
The list before was not comprehensive, other
algorithm and model classes include:
● Conditional Random Fields (CRF)
● Markovian Models (HMMs)
● Dimensionality Reduction (PCA, PLS)
● Rule Learning (Apriori, Brill)
● More ...
An Architecture for Operationalizing
Machine Learning Algorithms
 Build Phase

Feature
Training Data Vectors

Estimation
Algorithm

Labels

Operational Phase

New Data Feature Predictive

Prediction
Vector Model

Architecture of Machine Learning Operations

Feedback!
 Initial Fixtures

Model Building

Service/API

Feedback

The Learning Part of Machine Learning

Deploying Machine Learning as a Web Service
Annotation Service Example
 Architecture Demo
https://github.com/DistrictDataLabs/product-classifier
What is Scikit-Learn?
Extensions to SciPy (Scientific Python) are
called SciKits. SciKit-Learn provides machine
learning algorithms.
● Algorithms for supervised & unsupervised learning
● Built on SciPy and Numpy
● Standard Python API interface
● Sits on top of c libraries, LAPACK, LibSVM, and Cython
● Open Source: BSD License (part of Linux)
Probably the best general ML framework out there.
Where did it come from?
Started as a Google summer of code project in
2007 by David Cournapeau, then used as a
thesis project by Matthieu Brucher.

In 2010, INRIA pushed the first public release,

and sponsors the project, as do Google,
Tinyclues, and the Python Software
Foundation.
Who uses Scikit-Learn?
Primary Features
- Generalized Linear Models
- SVMs, kNN, Bayes, Decision Trees, Ensembles
- Clustering and Density algorithms
- Cross Validation
- Grid Search
- Pipelining
- Model Evaluations
- Dataset Transformations
- Dataset Loading
A Guide to Scikit-Learn
Scikit-Learn API
Object-oriented interface centered around the
concept of an Estimator:
“An estimator is any object that learns from data; it may
be a classification, regression or clustering algorithm or
a transformer that extracts/filters useful features from
raw data.”

- Scikit-Learn Tutorial
class Estimator(object):

def fit(self, X, y=None):

"""Fits estimator to data. """
# set state of ``self``
return self

def predict(self, X):

"""Predict response of ``X``. """
# compute predictions ``pred``
return pred

The Scikit-Learn Estimator API

Estimators
- fit(X,y) sets the state of the estimator.
- X is usually a 2D numpy array of shape
(num_samples, num_features).
- y is a 1D array with shape (n_samples,)
- predict(X) returns the class or value
- predict_proba() returns a 2D array of
shape (n_samples, n_classes)
from sklearn import svm

estimator = svm.SVC(gamma=0.001)
estimator.fit(X, y)
estimator.predict(x)

Basic methodology
Wrapping fit and predict
We’ve already discussed a broad workflow, the
following is a development workflow:

Feature Feature
Raw Data
Extraction Evaluation

Load &
Build Model Evaluate Model
Transform Data
class Transformer(Estimator):

def transform(self, X):

"""Transforms the input data. """
# transform ``X`` to ``X_prime``
return X_prime

from sklearn import preprocessing

Xt = preprocessing.normalize(X) # Normalizer
Xt = preprocessing.scale(X) # StandardScaler

imputer =Imputer(missing_values='Nan',
strategy='mean')
Xt = imputer.fit_transform(X)

Transformers
Cross Validation (classification)
Assess how model will generalize to independent data set
(e.g. data not in the training set).

1. Divide data into training and test splits

2. Fit model on training, predict on test
3. Determine accuracy, precision and recall
4. Repeat k times with different splits then average as F1

Predicted Class A Predicted Class B

Actual A True A False B #A

Actual B False A True B #B

#P(A) #P(B) total

from sklearn import metrics
from sklearn import cross_validation as cv

splits = cv.train_test_split(X, y, test_size=0.2)

X_train, X_test, y_train, y_test = splits

model = ClassifierEstimator()
model.fit(X_train, y_train)

expected = y_test
predicted = model.predict(X_test)

print metrics.classification_report(expected, predicted)

print metrics.confusion_matrix(expected, predicted)
print metrics.f1_score(expected, predicted)

Cross Validation in Scikit-Learn

MSE & Coefficient of Determination
In regressions we can determine how well the
model fits by computing the mean square error
and the coefficient of determination.

MSE = np.mean((predicted-expected)**2)

R2 is a predictor of “goodness of fit” and is a

value ∈ [0,1] where 1 is perfect fit.
from sklearn import metrics
from sklearn import cross_validation as cv

splits = cv.train_test_split(X, y, test_size=0.2)

X_train, X_test, y_train, y_test = splits

model = RegressionEstimator()
model.fit(X_train, y_train)

expected = y_test
predicted = model.predict(y_test)

print metrics.mean_squared_error(expected, predicted)

print metrics.r2_score(expected, predicted)

K-Part Cross Validation

Other Evaluation
How to evaluate clusters?
Visualization (but only in 2D)
Standardized Data Model Demo
(Wheat Kernel Sizes)
A Tour of Scikit-Learn
Questions, Comments?