NetflixOSS – A Cloud Native

Architecture

LASER Sessions 2&3 – Overview

September 2013
Adrian Cockcroft
@adrianco @NetflixOSS
http://www.linkedin.com/in/adriancockcroft
 Presentation vs. Tutorial
• Presentation
– Short duration, focused subject
– One presenter to many anonymous audience
– A few questions at the end

• Tutorial
– Time to explore in and around the subject
– Tutor gets to know the audience
– Discussion, rat-holes, “bring out your dead”
 Attendee Introductions
• Who are you, where do you work
• Why are you here today, what do you need
• “Bring out your dead”
– Do you have a specific problem or question?
– One sentence elevator pitch
• What instrument do you play?
 Content
Why Public Cloud?
Migration Path
Service and API Architectures
Storage Architecture
Operations and Tools
Example Applications
More?
 Cloud Native
A new engineering challenge
Construct a highly agile and highly
available service from ephemeral and
assumed broken components
 How to get to Cloud Native

Freedom and Responsibility for Developers

Decentralize and Automate Ops Activities
Integrate DevOps into the Business Organization
 Four Transitions
• Management: Integrated Roles in a Single Organization
– Business, Development, Operations -> BusDevOps

• Developers: Denormalized Data – NoSQL

– Decentralized, scalable, available, polyglot

• Responsibility from Ops to Dev: Continuous Delivery

– Decentralized small daily production updates

• Responsibility from Ops to Dev: Agile Infrastructure - Cloud

– Hardware in minutes, provisioned directly by developers
 Netflix BusDevOps Organization
Chief Product
Officer

VP Product VP UI VP Discovery
VP Platform
Management Engineering Engineering

Directors Directors Directors Directors

Product Development Development Platform

Code, independently updated Developers + Developers + Developers +

continuous delivery DevOps DevOps DevOps

Denormalized, independently UI Data Discovery Platform

updated and scaled data Sources Data Sources Data Sources

Cloud, self service updated &

AWS AWS AWS
scaled infrastructure
Decentralized Deployment
 Asgard Developer Portal
http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html
 Ephemeral Instances
• Largest services are autoscaled
• Average lifetime of an instance is 36 hours
P
u
s
h

Autoscale Up
Autoscale Down
Netflix Member Web Site Home Page
Personalization Driven – How Does It Work?
 How Netflix Used to Work
Consumer
Electronics Oracle
Monolithic Web
AWS Cloud App
Services
MySQL
CDN Edge
Locations
Oracle
Datacenter
Customer Device Monolithic
(PC, PS3, TV…) Streaming App
MySQL

Content
Management
Limelight/Level 3
Akamai CDNs
Content Encoding
 How Netflix Streaming Works Today
Consumer
Electronics User Data
Web Site or
AWS Cloud Discovery API
Services
Personalization
CDN Edge
Locations
DRM
Datacenter
Customer Device
Streaming API
(PC, PS3, TV…)
QoS Logging

CDN
Management
and Steering
OpenConnect
CDN Boxes
Content Encoding
 The AWS Question

Why does Netflix use AWS when

Amazon Prime is a competitor?
 Netflix vs. Amazon Prime
• Do retailers competing with Amazon use AWS?
– Yes, lots of them, Netflix is no different

• Does Prime have a platform advantage?

– No, because Netflix also gets to run on AWS

• Does Netflix take Amazon Prime seriously?

– Yes, but so far Prime isn’t impacting our growth
Nov
2012
Streaming
Bandwidth

March
2013

Mean
Bandwidth
+39% 6mo
The Google Cloud Question

Why doesn’t Netflix use Google

Cloud as well as AWS?
 Google Cloud – Wait and See
Pro’s Con’s
• Cloud Native • In beta until recently
• Huge scale for internal apps • Few big customers yet
• Exposing internal services • Missing many key features
• Nice clean API model • Different arch model
• Starting a price war • Missing billing options
• Fast for what it does • No SSD or huge instances
• Rapid start & minute billing • Zone maintenance windows
But: Anyone interested is welcome to port NetflixOSS components to Google Cloud
 Cloud Wars: Price and Performance
What Changed: AWS vs. Private
No Change:
Everyone using GCS War Cloud $$ Locked in for
AWS or GCS gets
three years.
the price cuts and
performance
improvements, as
they happen. No
need to switch
vendor.
 The DIY Question

Why doesn’t Netflix build and run its

own cloud?
 Fitting Into Public Scale

1,000 Instances 100,000 Instances

Grey
Public Private
Area

Startups Netflix Facebook

 How big is Public?
AWS Maximum Possible Instance Count 4.2 Million – May 2013
Growth >10x in Three Years, >2x Per Annum - http://bit.ly/awsiprange

AWS upper bound estimate based on the number of public IP Addresses

Every provisioned instance gets a public IP by default (some VPC don’t)
 The Alternative Supplier
Question
What if there is no clear leader for a
feature, or AWS doesn’t have what
we need?
 Things We Don’t Use AWS For

SaaS Applications – Pagerduty, Appdynamics

Content Delivery Service
DNS Service
 CDN Scale

Gigabits Terabits

Akamai Netflix
AWS CloudFront Limelight Openconnect
Level 3 YouTube

Startups Facebook Netflix

 Content Delivery Service
Open Source Hardware Design + FreeBSD, bird, nginx
see openconnect.netflix.com
 DNS Service

AWS Route53 is missing too many features (for now)

Multiple vendor strategy Dyn, Ultra, Route53
Abstracted (broken) DNS APIs with Denominator
 Cost Process
reduction reduction

Lower Slow down Higher Speed up

margins developers margins developers

Less More More

Less revenue
competitive revenue competitive

What Changed?

Get out of the way of innovation

Best of breed, by the hour
Choices based on scale
 Availability Questions

Is it running yet?
How many places is it running in?
How far apart are those places?
 Netflix Outages
• Running very fast with scissors
– Mostly self inflicted – bugs, mistakes from pace of change
– Some caused by AWS bugs and mistakes

• Incident Life-cycle Management by Platform Team

– No runbooks, no operational changes by the SREs
– Tools to identify what broke and call the right developer

• Next step is multi-region active/active

– Investigating and building in stages during 2013
– Could have prevented some of our 2012 outages
 Real Web Server Dependencies Flow
(Netflix Home page business transaction as seen by AppDynamics)

Each icon is
three to a few
hundred
instances
across three Cassandra
AWS zones
memcached
Web service
Start Here
S3 bucket

Personalization movie group choosers

(for US, Canada and Latam)
Three Balanced Availability Zones
Test with Chaos Gorilla

Load Balancers

Zone A Zone B Zone C

Cassandra and Evcache Cassandra and Evcache Cassandra and Evcache
Replicas Replicas Replicas
 Isolated Regions

US-East Load Balancers EU-West Load Balancers

Zone A Zone B Zone C Zone A Zone B Zone C

Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas

More?
Highly Available NoSQL Storage

A highly scalable, available and

durable deployment pattern based
on Apache Cassandra
 Single Function Micro-Service Pattern
One keyspace, replaces a single table or materialized view
Single function Cassandra
Many Different Single-Function REST Clients Cluster Managed by Priam
Between 6 and 144 nodes

Stateless Data Access REST Service

Astyanax Cassandra Client

Over 50 Cassandra clusters

Over 1000 nodes
Over 30TB backup
Over 1M writes/s/cluster

Each icon represents a horizontally scaled service of three to Optional

hundreds of instances deployed over three availability zones Datacenter
Update Flow
Appdynamics Service Flow Visualization
 Stateless Micro-Service Architecture

Linux Base AMI (CentOS or Ubuntu)

Optional
Apache
frontend,
Java (JDK 6 or 7)
memcached,
non-java apps
AppDynamics

Monitoring
appagent
monitoring
Tomcat
Log rotation Application war file, base Healthcheck, status
to S3 GC and thread servlet, platform, client servlets, JMX interface,
AppDynamics dump logging interface jars, Astyanax Servo autoscale
machineagent
Epic/Atlas
 Cassandra Instance Architecture

Linux Base AMI (CentOS or Ubuntu)

Tomcat and
Priam on JDK Java (JDK 7)
Healthcheck,
Status
AppDynamics
appagent
monitoring
Cassandra Server
Monitoring
AppDynamics Local Ephemeral Disk Space – 2TB of SSD or 1.6TB disk
GC and thread holding Commit log and SSTables
machineagent dump logging
Epic/Atlas
 Cassandra at Scale

Benchmarking to Retire Risk

More?
 Scalability from 48 to 288 nodes on AWS
http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html

Client Writes/s by node count – Replication Factor = 3

1200000
1099837
1000000

800000
Used 288 of m1.xlarge
4 CPU, 15 GB RAM, 8 ECU
600000
537172 Cassandra 0.86
Benchmark config only
400000 366828 existed for about 1hr
200000 174373

0
0 50 100 150 200 250 300 350
Cassandra Disk vs. SSD Benchmark
Same Throughput, Lower Latency, Half Cost
http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
 2013 - Cross Region Use Cases
• Geographic Isolation
– US to Europe replication of subscriber data
– Read intensive, low update rate
– Production use since late 2011

• Redundancy for regional failover

– US East to US West replication of everything
– Includes write intensive data, high update rate
– Testing now
 Benchmarking Global Cassandra
Write intensive test of cross region replication capacity
16 x hi1.4xlarge SSD nodes per zone = 96 total
192 TB of SSD in six locations up and running Cassandra in 20 min

Validation
Test 1 Million reads Test
Load
Load 1 Million writes Load
After 500ms
CL.ONE (wait for
CL.ONE with no
one replica to ack)
Data loss

US-West-2 Region - Oregon US-East-1 Region - Virginia

Zone A Zone B Zone C Zone A Zone B Zone C

Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas

Inter-Zone Traffic Inter-Region Traffic

Up to 9Gbits/s, 83ms 18TB
backups
from S3
Managing Multi-Region Availability

AWS DynECT
Route53 Denominator
UltraDNS DNS

Regional Load Balancers Regional Load Balancers

Zone A Zone B Zone C Zone A Zone B Zone C

Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas

Denominator – manage traffic via multiple DNS providers with Java code
2013 Timeline - Concept Jan, Code Feb, OSS March, Production use May
 Incidents – Impact and Mitigation
Public Relations Y incidents mitigated by Active
Media Impact Active, game day practicing
PR
X Incidents YY incidents
High Customer mitigated by
Service Calls better tools and
CS
practices
XX Incidents
Affects AB YYY incidents
Test Results mitigated by better
Metrics impact – Feature disable
data tagging
XXX Incidents

No Impact – fast retry or automated failover

XXXX Incidents
 Cloud Native Big Data

Size the cluster to the data

Size the cluster to the questions
Never wait for space or answers
 Netflix Dataoven
From cloud RDS
Services
~100 Billion Ursula
Events/day Metadata

From C* Aegisthus
Terabytes of
Dimension
data
Data Pipelines
Gateways
Data Warehouse
Over 2 Petabytes

Hadoop Clusters – AWS EMR Tools

More?
1300 nodes 800 nodes Multiple 150 nodes Nightly
 Cloud Native Development
Patterns
Master copies of data are cloud resident
Dynamically provisioned micro-services
Services are distributed and ephemeral
 Datacenter to Cloud Transition Goals
• Faster
– Lower latency than the equivalent datacenter web pages and API calls
– Measured as mean and 99th percentile
– For both first hit (e.g. home page) and in-session hits for the same user
• Scalable
– Avoid needing any more datacenter capacity as subscriber count increases
– No central vertically scaled databases
– Leverage AWS elastic capacity effectively
• Available
– Substantially higher robustness and availability than datacenter services
– Leverage multiple AWS availability zones
– No scheduled down time, no central database schema to change
• Productive
– Optimize agility of a large development team with automation and tools
– Leave behind complex tangled datacenter code base (~8 year old architecture)
– Enforce clean layered interfaces and re-usable components
Datacenter Anti-Patterns

What do we currently do in the

datacenter that prevents us from
meeting our goals?
 Rewrite from Scratch

Not everything is cloud specific

Pay down technical debt
Robust patterns
Netflix Datacenter vs. Cloud Arch
Central SQL Database Distributed Key/Value NoSQL

Sticky In-Memory Session Shared Memcached Session

Chatty Protocols Latency Tolerant Protocols

Tangled Service Interfaces Layered Service Interfaces

Instrumented Code Instrumented Service Patterns

Fat Complex Objects Lightweight Serializable Objects

Components as Jar Files Components as Services

More?
 Tangled Service Interfaces
• Datacenter implementation is exposed
– Oracle SQL queries mixed into business logic

• Tangled code
– Deep dependencies, false sharing

• Data providers with sideways dependencies

– Everything depends on everything else

Anti-pattern affects productivity, availability

 Untangled Service Interfaces
Two layers:
• SAL - Service Access Library
– Basic serialization and error handling
– REST or POJO’s defined by data provider

• ESL - Extended Service Library

– Caching, conveniences, can combine several SALs
– Exposes faceted type system (described later)
– Interface defined by data consumer in many cases
Service Interaction Pattern
Sample Swimlane Diagram

More?
 NetflixOSS Details
• Platform entities and services

• AWS Accounts and access management

• Upcoming and recent NetflixOSS components

• In-depth on NetflixOSS components

 Basic Platform Entities
• AWS Based Entities
– Instances and Machine Images, Elastic IP Addresses
– Security Groups, Load Balancers, Autoscale Groups
– Availability Zones and Geographic Regions

• NetflixOS Specific Entities

– Applications (registered services)
– Clusters (versioned Autoscale Groups for an App)
– Properties (dynamic hierarchical configuration)
 Core Platform Services
• AWS Based Services
– S3 storage, to 5TB files, parallel multipart writes
– SQS – Simple Queue Service. Messaging layer.

• Netflix Based Services

– EVCache – memcached based ephemeral cache
– Cassandra – distributed persistent data store
 Cloud Security

Fine grain security rather than perimeter

Leveraging AWS Scale to resist DDOS attacks
Automated attack surface monitoring and testing
http://www.slideshare.net/jason_chan/resilience-and-security-scale-lessons-learned
 Security Architecture
• Instance Level Security baked into base AMI
– Login: ssh only allowed via portal (not between instances)
– Each app type runs as its own userid app{test|prod}

• AWS Security, Identity and Access Management

– Each app has its own security group (firewall ports)
– Fine grain user roles and resource ACLs

• Key Management
– AWS Keys dynamically provisioned, easy updates
– High grade app specific key management using HSM

More?
AWS Accounts
 Accounts Isolate Concerns
• paastest – for development and testing
– Fully functional deployment of all services
– Developer tagged “stacks” for separation

• paasprod – for production

– Autoscale groups only, isolated instances are terminated
– Alert routing, backups enabled by default

• paasaudit – for sensitive services

– To support SOX, PCI, etc.
– Extra access controls, auditing

• paasarchive – for disaster recovery

– Long term archive of backups
– Different region, perhaps different vendor
 Reservations and Billing
• Consolidated Billing
– Combine all accounts into one bill
– Pooled capacity for bigger volume discounts
http://docs.amazonwebservices.com/AWSConsolidatedBilling/1.0/AWSConsolidatedBillingGuide.html

• Reservations
– Save up to 71%, priority when you request reserved capacity
– Unused reservations are shared across accounts

• Cost Aware Cloud Architectures – with Jinesh Varia of AWS

http://www.slideshare.net/AmazonWebServices/building-costaware-
architectures-jinesh-varia-aws-and-adrian-cockroft-netflix

More?
 Cloud Access Control
Cloud Access
audit log
developers ssh/sudo www- • Userid wwwprod
Gateway prod
Security groups don’t allow
ssh between instances

Dal- • Userid dalprod

prod

Cass- • Userid cassprod

prod
 Our perspiration…
A Cloud Native Open Source Platform
See netflix.github.com
Example Application – RSS Reader

Zuul Traffic
Processing
and Routing

Z
U
U
L
 Zuul Architecture
http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html
 Ice – AWS Usage Tracking
http://techblog.netflix.com/2013/06/announcing-ice-cloud-spend-and-usage.html
NetflixOSS Continuous Build and Deployment

Github Maven AWS

NetflixOSS Central Base AMI
Source

Cloudbees
Dynaslave
Jenkins AWS
AWS Build
Aminator Baked AMIs
Slaves
Bakery

Odin Asgard AWS

Orchestration (+ Frigga) Account
API Console
More?
 NetflixOSS Services Scope

AWS Account
Asgard Console

Archaius
Config Service
Multiple AWS Regions
Cross region Priam C*
Eureka Registry

Pytheas
Dashboards
Exhibitor
Zookeeper 3 AWS Zones
Atlas
Edda History
Monitoring
Application Clusters Priam Evcache
Simian Army Autoscale Groups Cassandra Memcached
Genie, Lipstick
Instances Persistent Storage Ephemeral Storage
Hadoop Services

Zuul Traffic Mgr

Ice – AWS Usage
Cost Monitoring

More?
 NetflixOSS Instance Libraries

• Baked AMI – Tomcat, Apache, your code

Initialization • Governator – Guice based dependency injection

• Archaius – dynamic configuration properties client
• Eureka - service registration client

Service • Karyon - Base Server for inbound requests

• RxJava – Reactive pattern
• Hystrix/Turbine – dependencies and real-time status
Requests • Ribbon and Feign - REST Clients for outbound calls

• Astyanax – Cassandra client and pattern library

Data Access • Evcache – Zone aware Memcached client

• Curator – Zookeeper patterns
• Denominator – DNS routing abstraction

• Blitz4j – non-blocking logging

Logging • Servo – metrics export for autoscaling
• Atlas – high volume instrumentation
More?
 NetflixOSS Testing and Automation

• CassJmeter – Load testing for Cassandra

Test Tools • Circus Monkey – Test account reservation rebalancing

• Janitor Monkey – Cleans up unused resources

• Efficiency Monkey
Maintenance • Doctor Monkey
• Howler Monkey – Complains about AWS limits

• Chaos Monkey – Kills Instances

• Chaos Gorilla – Kills Availability Zones
Availability • Chaos Kong – Kills Regions
• Latency Monkey – Latency and error injection

• Conformity Monkey – architectural pattern warnings

Security • Security Monkey – security group and S3 bucket permissions
More?
Your perspiration – deadline Sept 15th
Boosting the @NetflixOSS Ecosystem
See netflix.github.com
In 2012 Netflix Engineering won this..
We’d like to give out prizes too
But what for?
Contributions to NetflixOSS!
Shared under Apache license
Located on github
How long do you have?

Entries open March 13th

Entries close September 15th
Six months…
 Who can win?

Almost anyone, anywhere…

Except current or former Netflix or
AWS employees
Who decides who wins?

Nominating Committee
Panel of Judges
 Judges

Aino Corry
Martin Fowler
Program Chair for Qcon/GOTO Simon Wardley Chief Scientist Thoughtworks
Strategist

Werner Vogels Yury Izrailevsky

CTO Amazon Joe Weinman VP Cloud Netflix
SVP Telx, Author “Cloudonomics”
 What are Judges Looking For?
Eligible, Apache 2.0 licensed
Original and useful contribution to NetflixOSS

Code that successfully builds and passes a test suite

A large number of watchers, stars and forks on github

NetflixOSS project pull requests

Good code quality and structure

Documentation on how to build and run it

Evidence that code is in use by other projects, or is running in production

 What do you win?
One winner in each of the 10 categories
Ticket and expenses to attend AWS
Re:Invent 2013 in Las Vegas
A Trophy
 How do you enter?
Get a (free) github account
Fork github.com/netflix/cloud-prize
Send us your email address
Describe and build your entry

Twitter #cloudprize
 Vendor Driven Portability
Interest in using NetflixOSS for Enterprise Private Clouds

“It’s done when it runs Asgard”

Functionally complete
Demonstrated March
Released June in V3.3

Vendor and end user interest

Offering $10K prize for integration work Openstack “Heat” getting there
Paypal C3 Console based on Asgard
 Takeaways

Cloud Native Manages Scale and Complexity at Speed

NetflixOSS makes it easier for everyone to become Cloud Native

@adrianco #netflixcloud @NetflixOSS