Azure Machine Learning

© Jitesh Khurkhuriya
 Create an AzureML workspace Create models by using AzureML Designer

Run training scripts in an AzureML workspace

Manage data objects in an AzureML workspace

Set up AzureML Run Experiments Generate metrics from an experiment run

Manage experiment compute contexts
Workspace and Train Models
Automate the model training process

DP-100
Create production compute targets Use Automated ML to create optimal models
Deploy and Optimize and
Deploy a model as a service Consume Models Manage Models Use Hyperdrive to tune hyperparameters

Create a pipeline for batch inferencing Use model explainers to interpret models

Publish a designer pipeline as a web service Manage models

© Jitesh Khurkhuriya
Typical Machine Learning Workflow

Data Processing Build and Train Model Deploy and Monitor

Data Ingestion Data Validation Model Training

Model Deployment
Cross Validation
Feature Engineering Inferences/Predictions
Hyperparameter Tuning
Feature Selection Monitor Results
Model Selection

© Jitesh Khurkhuriya
Typical Machine Learning Workflow

© Jitesh Khurkhuriya
 Typical Machine Learning Workflow
Model Building and
Data Processing Hyperparameter Tuning Evaluate and Select
Selection

• ID • Logistic Regression • L1 and L2 Select Best model

• Credit Given • Boosted Decision Tree • Tolerance based on the goal
• Gender • Decision/Random Forest • Number of trees
• Education • Support Vector Machines • Max Depth
• Marital Status • Deep Learning • Splits per node
• ….. • Min Samples per node
• Default?

• Which Features to • Which Algorithm to • How to tune the • Compare the goals
select? Select? parameters?
• How to deal with
missing values?

© Jitesh Khurkhuriya
 Azure AutoML

Bring Dataset Rank Run Accuracy

Run 1 Data Processing + Algorithm + Tunning 62%
1 Run 5 92%
Run 2 Data Processing + Algorithm + Tunning 82% 2 Run 3 91%
3 Run 2 82%
Run 3 Data Processing + Algorithm + Tunning 91%
Define Goal 4 Run 4 74%

Run 4 Data Processing + Algorithm + Tunning 74% 5 Run 1 62%

Run 5 Data Processing + Algorithm + Tunning 92%

Constraints

© Jitesh Khurkhuriya
Note on norm_micro_recall

© Jitesh Khurkhuriya
 Recall

Predicted

False True

False 253 115 𝑇𝑃 46

𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.4107
Actual

𝑇𝑃 + 𝐹𝑁 46 + 66

True 66 46

© Jitesh Khurkhuriya
 Recall

Predicted

No Yes

No 253 115 𝑇𝑃 46
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.4107
Actual

𝑇𝑃 + 𝐹𝑁 46 + 66

Yes 66 46

© Jitesh Khurkhuriya
 Recall

Predicted

C1 C2

C1 253 115 𝑇𝑃 46
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.4107
Actual

𝑇𝑃 + 𝐹𝑁 46 + 66

C2 66 46

© Jitesh Khurkhuriya
 Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠

C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual

253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46

© Jitesh Khurkhuriya
 Macro Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠

C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual

253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46

𝑅𝑒𝑐𝑎𝑙𝑙𝐶1 + 𝑅𝑒𝑐𝑎𝑙𝑙𝐶2 0.4107 + 0.6875

𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = = = 0.5491
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑙𝑎𝑠𝑠𝑒𝑠 2

© Jitesh Khurkhuriya
 Normalized Macro Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠

C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual

253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46

𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 0.5491

𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 − 𝑅 1
𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 𝑅= 𝐶 → 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑙𝑎𝑠𝑠𝑒𝑠
1 −𝑅 𝐶

© Jitesh Khurkhuriya
 Normalized Macro Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠

C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual

253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46

𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 0.5491

𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 − 𝑅 0.5491 − 0.5

𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = = = 0.0982
1 −𝑅 1 − 0.5

© Jitesh Khurkhuriya
 Normalized Macro Recall
Predicted Predicted
C1 C2 C1 C2

C1 368 0 C1 368 0
Actual

Actual
C2 112 0 C2 0 112

368 368
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = =1 𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = =1
368 + 0 368 + 0

0 112
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = =0 𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = =1
0 + 112 112 + 0

𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 0.5 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 1

0.5 − 0.5 1 − 0.5

𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = =0 𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = =1
1 − 0.5 1 − 0.5

© Jitesh Khurkhuriya
Azure Machine Learning

Thank You..!!
© Jitesh Khurkhuriya