What is Scikit-Learn (sklearn)?

Data

?
 Why Scikit-Learn?
• Built on NumPy and Matplotlib (and Python)
• Has many in-built machine learning models
• Methods to evaluate your machine learning models
• Very well-designed API
What are we going to cover?

A Scikit-Learn workflow
 What are we going to cover?
• An end-to-end Scikit-Learn workflow

• Getting data ready (to be used with machine learning models)

• Choosing a machine learning model

• Fitting a model to the data (learning patterns)

• Making predictions with a model (using patterns)

• Evaluating model predictions

• Improving model predictions

• Saving and loading models

 Where can you get help?

• Follow along with the code

• Try it for yourself
• Press SHIFT + TAB to read the docstring
• Search for it
• Try again
• Ask
Let’s model!
 Supervised learning

Heart disease?

Classification Regression

• “Is this example one thing or another?”

• “How much will this house sell for?”
• Binary classification = two options
• “How many people will buy this app?”
• Multi-class classification = more than two options
 One Hot Encoding
A process used to turn categories into numbers.

Car Colour Car Red Green Blue

0 Red 0 1 0 0

1 Green 1 0 1 0

2 Blue 2 0 0 1

3 Red 3 1 0 0
Classification and Regression
metrics
Classification Regression

Accuracy R2 (r-squared)

Precision Mean absolute error (MAE)

Recall Mean squared error (MSE)

F1 Root mean squared error (RMSE)

Bold = default evaluation in Scikit-Learn

Confusion matrix anatomy

• True positive = model predicts 1 when truth is 1

• False positive = model predicts 1 when truth is 0
• True negative = model predicts 0 when truth is 0
• False negative = model predicts 0 when truth is 1
Classification report anatomy
• Precision - Indicates the proportion of positive identifications (model
predicted class 1) which were actually correct. A model which produces
no false positives has a precision of 1.0.
• Recall - Indicates the proportion of actual positives which were correctly
classified. A model which produces no false negatives has a recall of 1.0.
• F1 score - A combination of precision and recall. A perfect model
achieves an F1 score of 1.0.
• Support - The number of samples each metric was calculated on.
• Accuracy - The accuracy of the model in decimal form. Perfect accuracy
is equal to 1.0.
• Macro avg - Short for macro average, the average precision, recall and
F1 score between classes. Macro avg doesn’t class imbalance into
effort, so if you do have class imbalances, pay attention to this metric.
• Weighted avg - Short for weighted average, the weighted average
precision, recall and F1 score between classes. Weighted means each
metric is calculated with respect to how many samples there are in each
class. This metric will favour the majority class (e.g. will give a high value
when one class out performs another due to having more samples).
Which classification metric should
you use?
• Accuracy is a good measure to start with if all classes are balanced (e.g. same amount
of samples which are labelled with 0 or 1).
• Precision and recall become more important when classes are imbalanced.
• If false positive predictions are worse than false negatives, aim for higher precision.
• If false negative predictions are worse than false positives, aim for higher recall.
• F1-score is a combination of precision and recall.
Which regression metric should you
use?
• R2 is similar to accuracy. It gives you a quick indication of how well your model might be doing.
Generally, the closer your R2 value is to 1.0, the better the model. But it doesn't really tell exactly
how wrong your model is in terms of how far off each prediction is.
• MAE gives a better indication of how far off each of your model's predictions are on average.
• As for MAE or MSE, because of the way MSE is calculated, squaring the differences between
predicted values and actual values, it amplifies larger differences. Let's say we're predicting the
value of houses (which we are).
• Pay more attention to MAE: When being $10,000 off is twice as bad as being $5,000 off.
• Pay more attention to MSE: When being $10,000 off is more than twice as bad as being
$5,000 off.
 Improving a model
(via hyperparameter tuning)

Cooking time: 1 hour

Temperature: 180ºC

Cooking time: 1 hour

Temperature: 200ºC
Tuning Hyperparameters by Hand
The most important concept in
machine learning
(the 3 sets)

Course materials Practice exam Final exam

(training set) (validation set) (test set)

The ability for a machine learning model to perform

Generalization well on data it hasn’t seen before.
 Cross-validation
5-fold Cross-validation
100 patient records
Normal Train & Test Split

100 patient records

Split 20 80 patient records

80 patient records 20

Training split (80%) Test split (20%)

Model is trained on training data, and evaluated on the test

data.

Model is trained on 5 different versions of training data, and

evaluated on 5 different versions of the test data.
 5-fold Cross-validation

100 patient records

Normal Train & Test Split

100 patient records

20 80 patient records

80 patient records 20

Training split (80%) Test split (20%)

Figure 1.0: Model is trained on training data, and evaluated

on the test data.

Figure 2.0: Model is trained on training data, and evaluated

on the test data.
 Things to remember
• All data should be numerical

• There should be no missing values

• Manipulate the test set the same as the training set

• Never test on data you’ve trained on

• Tune hyperparameters on validation set OR use cross-validation

• One best performance metric doesn’t mean the best model