Lecture 2: ML Pipeline

CSC 484 / 584, DA 515

Fall 2024

REF: Chapter 2: End to End ML

Ch2. An Example: end to end

1. Look at the big picture.

2. Get the data.
3. Discover and visualize the data to gain
insights.

4. Prepare the data for Machine Learning

algorithms.
5. Select models and train them.
6. Fine-tune your models.
7. Present your solution.
2

8. Launch, monitor, and maintain your

ML pipeline example

# Create the pipeline (not totally matched the pipeline above)

pipeline = make_pipeline(StandardScaler(), PCA(n_components=8),
RandomForestClassifier(criterion='gini', n_estimators=50,
max_depth=2, random_state=1))
# Fit the pipeline
pipeline.fit(X_train, y_train)
# Evaluate the model
print('Model Accuracy: %.3f' % pipeline.score(X_test, y_test))
3
Questions you might ask:
 What does the dataset include?
 District median price along with population, income…

 What are the business objectives?

 (summary, viz, prediction…)

 What do we have currently? Complex rules (low accuracy)

 What kind of problem is it?

 Supervised/Non-Supervised

 Classification/Regression
 Which algorithm(linear, poly, neural, kernel, tree, k-nn)

4
Example: California Housing Price (1990)
 Given California housing prices(by district)
 Train a model to predict a district’s median
housing price

5
Load in the dataset
 Demo Code:
 Create an isolated Environment: DA515
 Install sklearn (code: pip install scikit-learn)

 Download the dataset (check the book code)

 For us, the data is saved in disk in the same folder as your
code is in.
housing = pd.read_csv(“CA_housing.csv”)

 For my case: data is saved in subfolder “datasets”

housing = pd.read_csv(“./datasets/CA_housing.csv”) 6
Take a Quick Look at the Data Structure
The first 5 rows:

7
Explore the dataset
 Check data numbers of rows and cols
 Check data types(int, float, categorical…)

 Check missing values

 (Column total bedroom: 207 rows missing)

 Check duplicated data

 Check statistics(max, min, mean, …)

8
Missing values
 In this example, we do not have a lot of missing values:

 We need to fix the missing values later.

9
Categorical data
# check the categorical data
ocean_proximity 20640 non-null object

housing["ocean_proximity"].value_counts()
<1H OCEAN 9034
INLAND 6496
NEAR OCEAN 2628
NEAR BAY 2270
ISLAND 5

There are 5 categories, counts are dif.

10
Plotting

# install matplotlib
! pip install matplotlib

# import matplotlib to memory

import matplotlib.pyplot as plt
housing.hist(bins=50, figsize=(20,15))
plt.show()

11
.hist()

12
Observations
 Data distribution varies. Need to have outlier removed.

 These attributes have very different scales.

 Finally, many histograms are tail-heavy: extend much

farther to the right

13
For Machine Learning
 Data pre-processing
 Missing values

 Categorical data

 Feature selection/engineering

 Data scaling
 Data Sampling:
 using stratify parameter:

This stratify parameter makes a split so that the proportion of values in

the sample produced will be the same as the proportion of values
provided to parameter stratify

14
For missing values 1/2
 You can:
1. Get rid of the corresponding districts.
2. Get rid of the whole attribute.
3. Set the values to some value (zero, the mean, the median, etc.).

median = housing["total_bedrooms"].median()
housing["total_bedrooms"].fillna(median, inplace=True)

4. Find the closest neighbors and use the average

15
Use SimpleImputer to fill in 2/2

You can use Scikit-Learn: SimpleImputer:

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
imputer =
imputer.fit(housing[["total_bedrooms"]])
housing['total_bedrooms'] =
imputer.transform(housing[['total_bedrooms
']])

16
Text and Categorical Attributes
 Check the first 10 categorical “ocean_proximity” data
samples
housing_cat = housing[["ocean_proximity"]]
housing_cat.head(10)

 Computer cannot deal with text data directly

17
Use value_count or distinct
# count all distinct values.
housing_cat.ocean_proximity.value_counts()

<1H OCEAN 7276

INLAND 5263
NEAR OCEAN 2124
NEAR BAY 1847
ISLAND 2

There are 5 categories, counts are dif.

18
Convert:
categories text => ordinal numbers

Scikit-Learn provides
OrdinalEncoder to change 5
categorical data using number
list: [0 1 2 3 4] to represents:

[‘<1H OCEAN', 'INLAND',

'ISLAND', 'NEAR BAY', 'NEAR
OCEAN’]

Don’t Use it:

Reason: Not
Ordinal data
19
DATA TYPES
 Variables Types:
 Continuous
 Discrete:
 Ordinal: can order such as A, B, C, D, or Mon, Tue,
 Nominal: blue, red, .. Or banana, apple, orange, …
 Text -> encode to numerical
 Image -> use pixel
 Voice -> text
 ….

20
Correct Encoding: One-Hot encoding

There are total 5 categories:

use list to encode: [x1 , x2 , x3, x4, x5]
xi is 1 for yes, 0 for no

1-hot example:

21
Problem: TOO MANY VARIABLES
USE one-hot encoder
# Use pd.get_dummies( )
housing_cat_1hot =
pd.get_dummies(housing_cat).astype(int)

# then merge it with numerical attribute

housing = housing.join(housing_cat_1hot)

Now we have 14 features

Feature selection/engineering
 Feature engineering:
create some new features which make more
sense.

 For example: rooms_per_household is better than

total_rooms

 Feature selection:
keep only the important relevant features
 There are several diff methods
 Recursive elimination, Important Ranking, PCA, etc.

 We talk about Correlation 23

Experiments of attribute combinations

What you really want is the number of rooms per

household

housing["rooms_per_household"]
=housing["total_rooms"]/housing["households"]

housing["bedrooms_per_room"] =
housing["total_bedrooms"]/housing["total_rooms"]

housing["population_per_household"]=housing["population"]/
housing["households"]

24
Matrix of Correlations

25
Standard correlation coefficient of various
datasets
(source: Wikipedia; public domain image)

26
Feature Selection
Correlations: median_house_value
#Looking for Correlations:
# for ML, keep only the important features
(Assume:linear relation)

27
Separate X and y: features and
target
 Separate the dependent Y/independent variables X:
# training X
X = housing.drop("median_house_value", axis=1)

# label Y
y = housing["median_house_value"]

28
Splitting and Scaling
 Which one needs to be done first?

 Book Code did splitting first

Splitting with random sampling methods. This is generally
fine if your dataset is large enough (especially relative to
the number of attributes)

 I prefer do it later.

29
Feature Scaling
 Now, all data are numerical
----------------------------------------------------------------------------
 Scaling is very important:
 For distance computation
 For optimization

 Two ways:
 Normalization: (x-x_min)/(x_max – x_min) => [0,1]
 Standardization: (x- mu)/sigma => N(0, 1)

30
Why do data scaling:
Lesson of the widow's mite
 This poor widow put in more than all the other contributors
to the treasury. For they have all contributed from their
surplus wealth, but she, from her poverty, has contributed
all she had, her whole livelihood (Wikipedia)

31
 Source of figure:

Feature Scaling http://cs231n.github.io/neural-

networks-2/

𝑦 =𝑏+ 𝑤1 𝑥 1+ 𝑤2 𝑥 2

𝑥2 𝑥2

𝑥1 𝑥1

Make different features have the same scaling

Feature Selection
 Add combined features
 Remove less relevant features:

33
 Feature Scaling 𝑦 =𝑏+ 𝑤1 𝑥 1+ 𝑤2 𝑥 2

w1 w1
1, 2 …… x1  y 1, 2 …… x1  y
w2 w2

100, 200 …… x2 b 1, 2 …… x2 b

w2 Loss L w2 Loss L

w1 w1
Data Splitting: 80 vs 20
Scikit-Learn provides a few functions to split datasets into
multiple subsets in various ways. The simplest function is
train_test_split(),

# random sampling
X_train, X_test, y_train, y_test =
train_test_split(X, y, test_size=0.2,
random_state= 100)

35
Now visualization for training data

36
ML models
 Regression: for continuous data (Y)
 Linear, or Poly
 Tree
 Random Forest

 K-NN(HOMEWORK)

 SVM
 ANN
 Kernel
 ….
37
Regression Evaluation Metrics (1/3)
1. Mean Square Error(MSE) or Root Mean Square
Error(RMSE):

38
Regression Evaluation Metrics (2/3)
2. Mean Absolute Error(MAE):

39
Regression Evaluation Metrics (3/3)
3. Square/Adjusted R Squared:
 Simple linear Regression ( r2 instead of R2 )
 The R2 quantifies the degree of any linear correlation between Yobs and Ypred,
or assess the goodness-of-fit

https://en.wikipedia.org/wiki/Coefficient_of_determination

40
 Limitation of using R squared
 R-squared Is Not Valid for Nonlinear Regression
https://statisticsbyjim.com/regression/r-squared-invalid-nonlinear-regression/#:~:text=Nonlinear%20regression%20is%20an
%20extremely,just%20don't%20go%20together
.

 If you use R-squared for nonlinear models, their study

indicates you will experience the following problems:

 R-squared is consistently high for both excellent and appalling models.

 R-squared will not rise for better models all of the time.
 If you use R-squared to pick the best model, it leads to the proper
model only 28-43% of the time.

 More info: https://en.wikipedia.org/wiki/Coefficient_of_determination

41
3 Steps of ML
 Import a model
 Train: fit (X, y)
 Evaluate:
 MSE(Mean Squared Error)
 RMSE(Root Mean Squared Error (RMSE).)
 Cross Validation

42
Grid Search: looking for best user-defined
parameters

43
Example:
 Forest: Total 12 combinations + 6 combinations

44
Short Summary

 After hyperparameter tuning,

Compare the RMSEs

 We also need to avoid

overfitting

 Feature Selection can be

done differently

 Feature Importance from

Random Forest:
45
Final Pipeline
 Example Bayesian Algorithm

46
Finally

 Evaluate Your System on the Test

Set
 Launch, Monitor, and Maintain

Your System

47
 Homework: K-NN for Appraisal

Data: California Housing (chapter 2 )

Cannot use the sk-learn library

K_NN:K nearest neighbors:

• Lazy algorithm ，
• No training,
• No distribution assumption ，
• Based on feature similarity ，
• Used in classification by majority vote
• For regression, find the average price (scalar)

48
FYI: Real data sources
 UCI Data Repository
http://archive.ics.uci.edu/ml/index.php

 Kaggle
https://www.kaggle.com/datasets

 Google datasets
https://cloud.google.com/public-datasets/

 Government (Agriculture/Commerce/Education/FDA…. )
https://catalog.data.gov/dataset
49
 END
• Read book Chapter 2
• Practice the code
• Do your homework 1