OCTOBER 24, 2024

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MMBT5S6

Mastering SaaS Operations

Lessons from Amazon for Operational Excellence

Kilian Ruess
SaaS Advisor, AWS

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buy with Prime
AMAZON REFERENCE CASE

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buy with Prime

Stand out to Offer Prime Delight Prime Give Prime members

shoppers members an easy members with fast, post-purchase peace
checkout free delivery of mind

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buy with Prime

Shoppers see the Buy with Prime button

and delivery promise on merchant’s site.

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buy with Prime architecture from 50,000 ft
Customer-facing
layer Backend services

Button Catalogs
Merchant

APIs Orders

Developer
Payments
tools
…

…
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Buy with Prime architecture from 50,000 ft
Customer-facing
layer Backend services

Button Catalogs
Merchant

Isolation
APIs Orders

Developer
Payments
tools
Merchant 2
…

…
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operational Excellence

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operational excellence

Perform operations Make frequent, Refine operations

as code small, reversible changes procedures frequently

Anticipate failure Learn from all

operational failures

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://aws.amazon.com/architecture/well-architected/
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenant On-boarding
Perform operations
as code

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenant On-boarding
Perform operations
as code

Reproducible
Tenants

Ensure tenants are built

the same way to increase
maintainability

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenant On-boarding
Perform operations
as code

Reproducible Frictionless
Tenants On-Boarding

Ensure tenants are built Automated to reduce

the same way to increase on-boarding time and to
maintainability bring value to customers
as quickly as possible

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenant On-boarding
Perform operations
as code

Reproducible Frictionless Maintainable

Tenants On-Boarding Process

Ensure tenants are built Automated to reduce In code, to be able to

the same way to increase on-boarding time and to evolve this critical part of
maintainability bring value to customers the SaaS Control Plane
as quickly as possible

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenant On-boarding
1 Establish SaaS identity
Perform operations
3 as code
Authentication Onboarding Tenant
Provision
Token (tenant context) 4 2
(optional)
Tenant
user
Multi-tenant aware Tenant
application stack resources

6 Record tenant metrics 5 Log with tenant context

Metrics/metering Logging

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tenant On-boarding
CloudFormation Template Perform operations
as code

{
"Parameters": {
"tenantId": {
Merchants "Type": "String",
"Description": "Merchant owner ID"
Sign up }
},

Merchant onboarding service "Resources": {

"Resource1": {
Trigger
Create "Type": "Amazon::BwP::CatalogInstance",
creation of
Merchant … merchant "Properties": {
metadata "TenantId": {"Ref": "tenantId”}
resources
}
AWS CloudFormation },

"Resource2": {
"Type": "Amazon::BwP::OrderInstance",
"Properties": {
"TenantId": {"Ref": "tenantId"}
}
},

…
}
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operational excellence

Perform operations Make frequent, Refine operations

as code small, reversible changes procedures frequently

Anticipate failure Learn from all

operational failures

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updates enable agility and innovation
Make frequent,
small, reversible changes

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updates enable agility and innovation
Make frequent,
small, reversible changes

Downtimes

Maintenance windows
across many tenants are
not feasible, updates
need to run without
downtime

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updates enable agility and innovation
Make frequent,
small, reversible changes

Downtimes Tenant Isolation

Maintenance windows Risk of compromising

across many tenants are tenant isolation and fail
not feasible, updates compliance requirements
need to run without
downtime

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updates enable agility and innovation
Make frequent,
small, reversible changes

Downtimes Tenant Isolation User Experience

Maintenance windows Risk of compromising Users should not be

across many tenants are tenant isolation and fail impacted in their ability
not feasible, updates compliance requirements to perform work,
need to run without acceptance of updates at
downtime risk

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipeline
Gamma- Prod-
Source Build Alpha Beta Gamma Prod
OneBox OneBox

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipeline
Gamma- Prod-
Source Build Alpha Beta Gamma Prod
OneBox OneBox

• Application source
• Operation tool source
• Test source
• Infrastructure as code
• Static assets
• Dependency libraries
• Configurations

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipeline
Gamma- Prod-
Source Build Alpha Beta Gamma Prod
OneBox OneBox

• Compile
• Unit test
• Static analysis
• Code coverage check
• Code review check
• Store package and artifacts

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipeline
Gamma- Prod-
Source Build Alpha Beta Gamma Prod
OneBox OneBox

• Health checks + production like:

• Functional tests • Monitoring
• Security integration test • Alarms
• Integration tests • Synthetic Testing

Automatic rollback window

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipeline
Gamma- Prod-
Source Build Alpha Beta Gamma Prod
OneBox OneBox

Load balancer

OneBox Existing Fleet

New version! Original version

Original version

Original version

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pipeline
Gamma- Prod-
Source Build Alpha Beta Gamma Prod
OneBox OneBox

New version is live on Prod

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updates enable agility and innovation
Make frequent,
small, reversible changes

Downtimes Tenant Isolation User Experience

Maintenance window Risk of compromising Users should not be

across many tenants are tenant isolation and fail impacted in their ability
not feasible, updates compliance requirements to perform work,
need to run without acceptance of updates at
downtime risk

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How tenant isolation validation works

IAM Role 1
NotAllowListed AWS Account
Control plane
Matching TenantID Tenant metadata database

Test Runner Assume

instance
IAM Role 2
AllowListed AWS Account
Not matching TenantID
Data plane
Tenant business database

IAM Role 3
AllowListed AWS Account
Matching TenantID

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How tenant isolation validation works

Test Runner
instance

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How tenant isolation validation works

IAM Role 1
NotAllowListed AWS Account
Control
Tenant plane
metadata
Matching TenantID Tenant metadata database

Test Runner
instance
IAM Role 2
AllowListed AWS Account
Not matching TenantID
Data
Tenant plane
business data
Tenant business database

IAM Role 3
AllowListed AWS Account
Matching TenantID

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How tenant isolation validation works

IAM Role 1
NotAllowListed AWS Account
Control
Tenant plane
metadata
Matching TenantID Tenant metadata database

Test Runner Assume

instance
IAM Role 2
AllowListed AWS Account
Not matching TenantID
Data
Tenant plane
business data
Tenant business database

IAM Role 3
AllowListed AWS Account
Matching TenantID

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How tenant isolation validation works

IAM Role 1
NotAllowListed AWS Account
Control
Tenant plane
metadata
Matching TenantID Tenant metadata database

Test Runner
instance
IAM Role 2
AllowListed AWS Account
Not matching TenantID
Data
Tenant plane
business data
Tenant business database

IAM Role 3
AllowListed AWS Account
Matching TenantID

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real test cases for identity and isolation
Account ID
O X
Tenant ID Tenant ID
O X O X
Create
metadata
Tenant

Read
Update
Delete
O X X
Create
Tenant

Read
data

Update
Delete

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample test runner implementation

Test Runner IAM Role 1

Not matching TenantID
instance Tenant table

IAM Role 2
Matching TenantID

https://github.com/amzn/buy-with-prime-automate-tenant-validation
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Updates enable agility and innovation
Make frequent,
small, reversible changes

Downtimes Tenant Isolation User Experience

Maintenance window Risk of compromising Users should not be

across many tenants are tenant isolation and fail impacted in their ability
not feasible, updates compliance requirements to perform work,
need to run without acceptance of updates at
downtime risk

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Feature toggles
Tenant configuration

Off On Make frequent,

small, reversible changes
Tenant 1

Off On
Tenant 2

Off On
Tenant 3

New feature
Off On
Tenant 4

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Feature toggles
Tenant configuration

Off On Make frequent,

small, reversible changes
Tenant 1

Off On
Tenant 2

Off On
Tenant 3

New feature
Off On
Tenant 4

https://aws.amazon.com/blogs/mt/how-cyberark-implements-feature-flags-with-aws-appconfig/
https://aws.amazon.com/blogs/apn/simple-and-flexible-saas-entitlement-management-with-launchdarkly/
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operational excellence

Perform operations Make frequent, Refine operations

as code small, reversible changes procedures frequently

Anticipate failure Learn from all

operational failures

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Continuously improve

Learn from all

operational failures

Improve Optimize

Operating SaaS requires The promise of SaaS is

highest standards from that service providers
service providers, as they lower serving cost per
are responsible for many customer over time to
customers increase margin

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Culture of Operational Excellence

[OpsWin] Mechanism for communicating Learn from all

operational improvements to the wider operational failures
community. These celebrate work to reduce
latency, cost, errors to AWS and our customers
and inspire fellow teams to make improvements
in their services.
Presented in weekly Ops Metrics Meeting

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 41
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional Complexity in SaaS Operations

Health of a multi-tenant Usage and consumption trends

environment of individual tenants

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://catalog.workshops.aws/saas-operations/
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
For SaaS providers,
operating the service is the
business
Someone who has been there

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
Kilian Ruess
kilian-ruess

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 !
Please complete
the session survey

© 2021,
2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.