INTRODUCTION TO MLOPS

Introduction to MLOps
Bringing DevOps and Automation to Machine
Learning

Hei Chow
Solutions Architect

© 2021,
2022, Amazon Web Services, Inc. or its affiliates.
INTRODUCTION TO MLOPS

Current state of AI/ML

© 2022, Amazon Web Services, Inc. or its affiliates. 2

 INTRODUCTION TO MLOPS

State of machine learning

• Today
• 53% of POCs make it into production
• Average 9 months

- Gartner

Last decade By end of 2024

• Focusing mostly on building ML models • 75% of organizations will shift from piloting to
• Operationalization was an afterthought operationalizing AI

- Gartner

https://www.idgconnect.com/article/3583467/gartner-accelerating-ai-deployments-paths-of-least-resistance.html

© 2022, Amazon Web Services, Inc. or its affiliates. 3

INTRODUCTION TO MLOPS

Main Challenges
• Publishing a ML model is not • “IT leaders responsible for AI are
enough. discovering ‘AI pilot paradox’, where
launching pilots is deceptively easy but
• Managing the published ML
deploying them into production is
models is as important as
notoriously challenging.”
developing them.

• Chirag Dekate, Vice President

Analyst, Gartner

© 2022, Amazon Web Services, Inc. or its affiliates. 4

INTRODUCTION TO MLOPS

From DevOps to MLOps

© 2021,
2022, Amazon Web Services, Inc. or its affiliates. 5
INTRODUCTION TO MLOPS

The ML process

ML problem
Data collection
framing
Business
Problem
Feature Monitoring and
Data integration
engineering debugging

Data preparation Model training and

and cleaning parameter tuning Predictions

Data visualization
Model evaluation Model deployment
and analysis

NO YES

© 2022, Amazon Web Services, Inc. or its affiliates. 6

INTRODUCTION TO MLOPS

Phase 1: Research/Experiment

Data
Question: “Can we use ML to solve this?” Collection
• “Is it possible to … ?” Data
Monitoring Verificatio
• “Can we use this data to solve the following n
problem?”
• “Surely we must be able to …”
Serving
Feature
Infrastruct ML Code Extraction
Typical scenarios ure

• Scientific projects

• Proof-of-concepts (PoCs) Resource

Analysis
Manageme
Tools
nt
Process
Manageme
nt

© 2022, Amazon Web Services, Inc. or its affiliates. 7

INTRODUCTION TO MLOPS

Phase 2: Operational
Question: “How do we implement this method
at scale?” Data
Collection
• How do we pipe the data into the model in a timely Data
fashion? Monitoring Verificatio
• How do we collect, store and transform data so n
models can be retrained consistently?
• How do we build an A/B testing environment, in
order to test future model iterations? Serving
ML Feature
Infrastruct Code Extraction
ure

Typical scenarios
• After PoC, bringing your ML models to Resource
Analysis
Manageme
Tools
production Process
nt
Manageme
• Migration of existing models into ML platform nt

© 2022, Amazon Web Services, Inc. or its affiliates. 8

 INTRODUCTION TO MLOPS

MLOps – Why?

Agility Experiments Scalability Time to Market Business Owners

• Continuous and faster • Faster and Controlled • Ease integration of • Reduced time-to-market • Strong collaboration
deliveries Experiments new ML model • Faster planning and • Improve iterations
• Faster modifications • Faster integration of • Standarization of code delivery expectations
• Faster bug-fixing successful experiments • Lower operational
to other environments costs

© 2022, Amazon Web Services, Inc. or its affiliates. 9

INTRODUCTION TO MLOPS

MLOps – What?

ML + Dev + Ops = MLOps

Collaborative and experimental in nature | Automate as much as possible |
Continuous improvement of ML Models | Standardize and Scale

Models

+ +
ML Dev Ops
Data Monitor

© 2022, Amazon Web Services, Inc. or its affiliates. 10

INTRODUCTION TO MLOPS

ML Code and Data are Independent

Algorithm
Code Model architecture
Configuration

Model analysis
Model tuning
Model deployment
Training Model

Data validation
Data Shuffle and split
Transformation and
feature engineering

© 2022, Amazon Web Services, Inc. or its affiliates. 11

 INTRODUCTION TO MLOPS

How is MLOps different from DevOps?

DevOps MLOPS
Code versioning ✓ ✓
Compute environment ✓ ✓
Continuous integration/delivery (CI/CD) ✓ ✓
Monitoring in production ✓ ✓ MLOPS
Data provenance ✓ End-to-end ML
Datasets ✓ lifecycle
management
Models ✓
Hyperparameters ✓
Metrics ✓
Workflows ✓
https://medium.com/analytics-vidhya/mlops-the-epoch-of-productionizing-ml-models-4eec06d93623

© 2022, Amazon Web Services, Inc. or its affiliates. 12

 INTRODUCTION TO MLOPS

MLOps practices

Data Scientists/ Software Engineer DevOps Engineer

Data Engineer ML Engineer

Business
Stakeholder Training pipeline Deployment pipeline

Project Data Model Model Model Model

Data Deploy Integrate Operate
Identified Preparation Build Training Artifact Registry

Feedback

© 2022, Amazon Web Services, Inc. or its affiliates. 13

 INTRODUCTION TO MLOPS

ML lifecycle management

Model
evaluation and Productionize Testing and Monitoring and
Model building Deployment
experimentatio model quality Observability
n
Data Model Code

<> <> <>

Train Test Application
code code code

Code & model

Candidate Chosen in production
Productionized Model
models model
model

Train Test Metrics Test Production

data data data data

© 2022, Amazon Web Services, Inc. or its affiliates. 14

INTRODUCTION TO MLOPS

Automating ML Workflows
using SageMaker

© 2021,
2022, Amazon Web Services, Inc. or its affiliates.
 INTRODUCTION TO MLOPS

Technology components in MLOps

MLOPS • ML development,
experimentation,
• Create and manage collaboration
workflows • Compute/training
environment
• Automate ML steps Task
• Model registry
& pipelines orchestratio ML platform • Feature store
n • Model deployment
• Implement CI/CD
• Monitoring in
• Form a Directed production
Acyclic Graph (DAG) • Hyperparameter
Open optimization
AWS native DIY SageMaker
source • Dataset management
A
B C
Amazon EKS Amazon ECS Amazon
D SageMaker

E Amazon EC2 Amazon ECR

AWS Deep AWS Deep

Learning AMIs Learning
Containers

© 2022, Amazon Web Services, Inc. or its affiliates. 16

INTRODUCTION TO MLOPS

Amazon SageMaker
Most complete, end-to-end ML service

Integrated Workbench
Capabilities designed specifically for ML, data
preparation, experiment management,
and workflows

Managed Infrastructure
Designed for ultra low latency and high
throughput, automatic scaling, and
distributed training

Managed Tooling
Purpose-built from the ground up to
work together including auto ML,
collaboration, debugger, profiler, bias
analyzer, and explainability

© 2022, Amazon Web Services, Inc. or its affiliates. 17

INTRODUCTION TO MLOPS

Amazon SageMaker Features

Amazon SageMaker
PREPARE BUILD TRAIN & TUNE DEPLOY & MANAGE

SageMaker Ground Truth SageMaker Studio Managed Training Managed Deployment

Label training data for machine Notebooks Distributed infrastructure Fully managed, ultra low latency,
learning Jupyter notebooks with elastic management high throughput
compute and sharing
SageMaker Data Wrangler SageMaker Experiments Kubernetes & Kubeflow
Aggregate and prepare data for Built-in and Bring Capture, organize, and compare Integration
machine learning your-own Algorithms every step Simplify Kubernetes-based
Dozens of optimized algorithms machine learning
SageMaker Processing or bring your own Automatic
Built-in Python, BYO R/Spark Model Tuning Multi-Model Endpoints
Local Mode Hyperparameter optimization Reduce cost by hosting multiple
SageMaker Feature Store Test and prototype on your local models per instance
Store, update, retrieve, and share machine Distributed Training
features Libraries SageMaker Model Monitor
SageMaker Autopilot Training for large datasets Maintain accuracy of deployed
SageMaker Clarify Automatically create machine and models models
Detect bias and understand learning models with full
model predictions visibility SageMaker Debugger SageMaker Edge Manager
Debug and profile training runs Manage and monitor models on
SageMaker JumpStart edge devices
Pre-built solutions for common Managed Spot Training
use cases Reduce training cost by 90% SageMaker Pipelines
Workflow orchestration and
automation

SageMaker Studio
Integrated development environment (IDE) for ML

© 2022, Amazon Web Services, Inc. or its affiliates. 18

 INTRODUCTION TO MLOPS

Technology components in MLOps

MLOPS • ML development,
experimentation,
• Create and manage collaboration
workflows • Compute/training
environment
• Automate ML steps Task
• Model registry
& pipelines orchestratio ML platform • Feature store
n • Model deployment
• Implement CI/CD
• Monitoring in
• Form a Directed production
Acyclic Graph (DAG) • Hyperparameter
Open optimization
AWS native DIY SageMaker
source • Dataset management
A
B C Pipelines
Kubeflow Amazon EKS Amazon ECS Amazon
D SageMaker

E AWS Step
MLflow Amazon EC2 Amazon ECR
Functions

AWS Deep AWS Deep

Apache Airflow Learning AMIs Learning
Containers

© 2022, Amazon Web Services, Inc. or its affiliates. 19

INTRODUCTION TO MLOPS

Task orchestration
Open source 3rd party options Native AWS options

MLflow Apache Kubeflow AWS Step Amazon

Airflow Functions SageMaker
Pipelines
Open source Platform to author, ML toolkit Serverless Managed ML
platform for schedule and for pipeline pipelines in
the ML monitor workflows Kubernetes orchestration SageMaker Studio
lifecycle

Native integration with SageMaker

Apache Airflow Kubeflow & Kubernetes
• SageMaker Operators in Apache Airflow • SageMaker Components for
• Amazon Managed Workflows Kubeflow Pipelines
for Apache Airflow • SageMaker Operators for
Kubernetes
(managed Airflow service)
© 2022, Amazon Web Services, Inc. or its affiliates. 20
INTRODUCTION TO MLOPS

Amazon SageMaker Pipelines

Components
1
Project 2 Automated 3
Pipelines Model Build Model Registry
Workflows
Source Code Model
</> Control Deployment
Pipeline

Built-In Central Store to

Triggers manage models

End-to-End Traceability & Integration

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Amazon SageMaker Pipelines

Components – Pipelines

1
Project 2 Automated 3
Pipelines Model Build Model Registry
Workflows
Source Code Model
</> Control Deployment
Pipeline

Built-In Central Store to

Triggers manage models

End-to-End Traceability & Integration

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Amazon SageMaker Pipelines

Components – Pipelines

Supported Steps:
• Processing
• Training
• Tuning
• Conditional
• Register Model
• Create Model

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Amazon SageMaker Pipelines

Components – Model Registry

1
Project 2 Automated 3
Pipelines Model Build Model Registry
Workflows
Source Code Model
</> Control Deployment
Pipeline

Built-In Central Store to

Triggers manage models

End-to-End Traceability & Integration

© 2022, Amazon Web Services, Inc. or its affiliates.

 INTRODUCTION TO MLOPS

Amazon SageMaker Pipelines

Components – Model Registry

• Catalog models for production

• Manage model versions

• Track model performance metrics
• Associate metadata with a model

• Manage the approval status of a model

• Deploy models to production (with Projects)

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Amazon SageMaker Projects

High Level Services View
Build, Train, Deploy Template

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Amazon SageMaker Projects

using third-party source control and Jenkins

https://aws.amazon.com/blogs/machine-learning/create-amazon-sagemaker-projects-using-third-party-source-control-and-jenkins/

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Amazon SageMaker Pipelines

Built-In Triggers

1
Project 2 Automated 3
Pipelines Model Build Model Registry
Workflows
Source Code Model
</> Control Deployment
Pipeline

Built-In Central Store to

Triggers manage models

End-to-End Traceability & Integration

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Creating retraining strategies

1 - Scheduled
Example:
”We know we want to
train every week on
Saturday at 23:00” Amazon EventBridge
Scheduled event

2 - Event Driven Automated

Pipelines Model Build Model Registry
Workflows
Example:
Model
Trigger Deployment
Pipeline
retraining
workflow Central Store to
AWS Glue Amazon EventBridge manage models
Job Status AWS event

3 - Metric Based
Example:

Model Quality / Amazon CloudWatch

Data Drift Alert Event (event-based)

© 2022, Amazon Web Services, Inc. or its affiliates. 29

INTRODUCTION TO MLOPS

Amazon SageMaker Model Monitor

Supported Features:
• Automatic data collection
• Continuous monitoring
• Flexible Monitoring Rules
• Visual data analysis
• CloudWatch integration

© 2022, Amazon Web Services, Inc. or its affiliates. 30

 Automate ML workflows to
scale model development

Build CI/CD pipelines for ML to

accelerate model deployment
Amazon
SageMaker Catalog model versions, metadata, metrics,
and approvals for traceability and reusability
MLOps
Streamline the ML lifecycle Track lineage for troubleshooting
and compliance

Maintain accuracy of predictions

after models are deployed

Enhance governance and security

© 2022, Amazon Web Services, Inc. or its affiliates.

INTRODUCTION TO MLOPS

Getting Started

© 2022, Amazon Web Services, Inc. or its affiliates. 32

 INTRODUCTION TO MLOPS

Getting started: Next steps

Discovery and Proof of AWS Partner Training and

Get Hands Dirty Concepts (PoC) Network (APN) Certification

© 2022, Amazon Web Services, Inc. or its affiliates. 33

INTRODUCTION TO MLOPS

Thank you! Fill in the event survey

and get USD 25 AWS Credits

© 2022, Amazon Web Services, Inc. or its affiliates. 34