NFX301

How Netflix is using IPv6 to

enable hyperscale networking
Donavan Fritz (he/him)
Sr Network SRE
Netflix

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Why IPv6

Co-innovation between Netflix and AWS

Netflix progress

Lessons learned and best practices

How to get started and how to show IPv6 is worthwhile

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 AWS Cloud

us-east-1

Instances

AWS re:Invent 2021

NFX301
 AWS Cloud

us-west-2 us-east-1 eu-west-1

Instances Instances Instances

AWS re:Invent 2021

NFX301
 AWS Cloud

us-west-2 us-east-1 eu-west-1

Instances Instances Instances

Instances Instances Instances

Instances Instances Instances

AWS re:Invent 2021

NFX301
 Flat network

AWS Cloud

us-west-2 us-east-1 eu-west-1

VPC VPC VPC

Instances Instances Instances

VPC VPC VPC

Instances Instances Instances

VPC VPC VPC

Instances Instances Instances

AWS re:Invent 2021

NFX301
 Flat network

AWS Cloud

us-west-2 us-east-1 eu-west-1

VPC VPC VPC

Instances Instances Instances

VPC VPC VPC

Instances Instances Instances

VPC VPC VPC

Instances Instances Instances

Office
AWS Direct Connect AWS Direct Connect AWS Direct Connect building
AWS re:Invent 2021
NFX301
 Flat network
and now, containers
AWS Cloud

us-west-2 us-east-1 eu-west-1

VPC VPC VPC

Containers Containers Containers

VPC VPC VPC

Containers Containers Containers

VPC VPC VPC

Containers Containers Containers

Office
AWS Direct Connect AWS Direct Connect AWS Direct Connect building
AWS re:Invent 2021
NFX301
 Flat network
and now, containers

VPC

Titus container host

192.0.2.4
192.0.2.86

Containers Elastic network

interface

192.0.2.124
192.0.2.25
Containers Elastic network
interface

192.0.2.51
192.0.2.211

Containers Elastic network

interface

AWS re:Invent 2021

NFX301
 Where we have been

Flat network
Carry over from Amazon EC2 classic
Lots of client-side load balancing
Business logic

Containers
IP per container
Same network posture as EC2 instances

AWS re:Invent 2021

NFX301
 Where we have been Where we are going

Flat network Continued growth

Carry over from Amazon EC2 classic More accounts, more VPCs
Lots of client-side load balancing
Business logic

Containers On premises
IP per container Studio
Same network posture as EC2 instances Gaming (?)

AWS re:Invent 2021

NFX301
 Technical requirements

Flat network 100+ VPCs

Full IP reachability
Containers
Continued growth
On premises

AWS re:Invent 2021

NFX301
 Technical requirements

Flat network
Containers “N” IPs per ENI
Short-lived IPs
Continued growth
On premises

AWS re:Invent 2021

NFX301
 Technical requirements

Flat network
ENI density “N” IPs per ENI
Short-lived IPs
Continued growth
On premises

AWS re:Invent 2021

NFX301
 Technical requirements

Flat network
ENI density
Continued growth 1000+ (?) VPCs
Full IP reachability
On premises

AWS re:Invent 2021

NFX301
 Technical requirements

Flat network
ENI density
Continued growth
On premises Full IP reachability on premises

AWS re:Invent 2021

NFX301
 Flaws with current IPv4 approach

Flat network Continued growth

Not really “flat” (!) AWS limits (routing, etc.)
Public vs. private IPv4 Private IPv4 address exhaustion

ENI density On premises

AWS limits Private IPv4 address exhaustion
IPv4 address reuse No security groups (!)
EIP update lag Private IPv4 routing overhead

AWS re:Invent 2021

NFX301
 Flaws with current IPv4 approach

Flat network Continued growth

Not really “flat” (!) AWS limits (routing, etc.)
Public vs. private IPv4 Private IPv4 address exhaustion

ENI density On premises

AWS limits Private IPv4 address exhaustion
IPv4 address reuse No security groups (!)
EIP update lag Private IPv4 routing overhead

AWS re:Invent 2021

NFX301
 Flaws with current IPv4 approach

Flat network Continued growth

Not really “flat” (!) AWS routing limits (VPC peering, etc.)
Public vs. private IPv4 Private IPv4 address exhaustion

ENI density On premises

AWS limits Private IPv4 address exhaustion
IPv4 address reuse No security groups (!)
EIP update lag Private IPv4 routing overhead

AWS re:Invent 2021

NFX301
 Flaws with current IPv4 approach

Flat network Continued growth

Not really “flat” (!) AWS routing limits (VPC peering, etc.)
Public vs. private IPv4 Private IPv4 address exhaustion

ENI density On premises

AWS limits Private IPv4 address exhaustion
IPv4 address reuse AWS routing limits (AWS Direct Connect, etc.)
EIP update lag

AWS re:Invent 2021

NFX301
 What are we going to do?

AWS re:Invent 2021

NFX301
STRANGER THINGS

What are we going to do?

Partner with AWS

AWS re:Invent 2021
NFX301
 Ideas considered
“Amazon EC2 classic”
We’ve done this before!
Does not address ENI density

VPC VPC
Internet gateway Internet gateway

Instances Instances

AWS re:Invent 2021

NFX301
 Ideas considered
“Amazon EC2 classic” “Tiny bubbles”
We’ve done this before! Well defined pattern
Does not address ENI density Client-side load balancing
Does not address ENI density

VPC VPC

Endpoint Network Load

Balancer
Instances Instances

Network Load Endpoint

Balancer

AWS re:Invent 2021

NFX301
 Ideas considered
“Amazon EC2 classic” “Tiny bubbles”
We’ve done this before! Well defined pattern
Does not address ENI density Client-side load balancing
Does not address ENI density

VPC VPC

Endpoint Network Load

Balancer
Instances Instances

Network Load Endpoint

Balancer

AWS re:Invent 2021

NFX301
 How would we solve this in a
traditional network?

AWS re:Invent 2021

NFX301
THE QUEEN’S GAMBIT

How would we solve this in a

traditional network?

Route
a network
block to the
host
AWS re:Invent 2021
NFX301
 Can we do something similar in AWS?

AWS re:Invent 2021

NFX301
 Focus on ENI density

192.0.2.96 →
192.0.2.15 →
... 192.0.2.96/28 →
192.0.2.99 →
192.0.2.43 →

Elastic network interface Elastic network interface

Prefix delegation
AWS re:Invent 2021
NFX301
 Focus on ENI density

Source: https://twitter.com/_joemag_/status/1418345704964063232

AWS re:Invent 2021

NFX301
 How much IP space do we need?

AWS re:Invent 2021

NFX301
 How much IP space do we need?

192.168.0.0/16

172.16.0.0/12

10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

192.168.0.0/16

172.16.0.0/12

100.64.0.0/10

10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

192.168.0.0/16

172.16.0.0/12

100.64.0.0/10

Used on premises

10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

192.168.0.0/16

172.16.0.0/12

100.64.0.0/10

Used on premises
Used on AWS

10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

100.64.0.0/10

Used on premises
Used on AWS

10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

/28 per prefix 16 IPs /28

100.64.0.0/10

Used on premises
Used on AWS
Prefix delegation
10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

/28 per prefix 16 IPs /28

4 prefixes per ENI 64 IPs /26

100.64.0.0/10

Used on premises
Used on AWS
Prefix delegation
10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

/28 per prefix 16 IPs /28

4 prefixes per ENI 64 IPs /26

8K ENIs per zone 524k IPs /13

100.64.0.0/10

Used on premises
Used on AWS
Prefix delegation
10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

/28 per prefix 16 IPs /28

4 prefixes per ENI 64 IPs /26

8K ENIs per zone 524k IPs /13

3 zones per Region 1.5m IPs /12 & /13

100.64.0.0/10

Used on premises
Used on AWS
Prefix delegation
10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

/28 per prefix 16 IPs /28

4 prefixes per ENI 64 IPs /26

8K ENIs per zone 524k IPs /13

3 zones per Region 1.5m IPs /12 & /13

100.64.0.0/10
3 Regions 4.5m IPs /10 & /13

Used on premises
Used on AWS
Prefix delegation
10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

/28 per prefix 16 IPs /28

4 prefixes per ENI 64 IPs /26

8K ENIs per zone 524k IPs /13

3 zones per Region 1.5m IPs /12 & /13

100.64.0.0/10
3 Regions 4.5m IPs /10 & /13

Double it! 9m IPs /9 & /12

Used on premises
Used on AWS
Prefix delegation
10.0.0.0/8

AWS re:Invent 2021

NFX301
 How much IP space do we need?

AWS re:Invent 2021

NFX301
 How much IP space do we need?

A lot

ALTERED CARBON

AWS re:Invent 2021

NFX301
THE UMBRELLA ACADEMY

IPv6
AWS re:Invent 2021
NFX301
 Focus on ENI density

192.0.2.96 →
192.0.2.15 →
... 192.0.2.96/28 →
192.0.2.99 →
192.0.2.43 →

Elastic network interface Elastic network interface

AWS re:Invent 2021

NFX301
 Focus on ENI density

192.0.2.96 →
192.0.2.15 →
... 192.0.2.96/28 →
192.0.2.99 →
192.0.2.43 →

Elastic network interface Elastic network interface

2001:db8::96 →
2001:db8::15 →
... 2001:db8:96::/80→
2001:db8::99 →
2001:db8::43 →

Elastic network interface Elastic network interface

AWS re:Invent 2021

NFX301
 Does IPv6 solve business network requirements?

Flat network Continued growth

No NAT

ENI density On premises

AWS re:Invent 2021

NFX301
 Does IPv6 solve business network requirements?

Flat network Continued growth

No NAT

ENI density On premises

Prefix delegation

AWS re:Invent 2021

NFX301
 Does IPv6 solve business network requirements?

Flat network Continued growth

No NAT No IPv4 address exhaustion
AWS limits (routing, etc.)

ENI density On premises

Prefix delegation

AWS re:Invent 2021

NFX301
 Inter-VPC private IPv4 reachability
1 – Customer gateway

AWS Cloud

VPC VPC

Customer gateway

AWS re:Invent 2021

NFX301
 Inter-VPC private IPv4 reachability
1 – Customer gateway

2 – VPC peering

AWS Cloud

VPC VPC

Peering
connection

AWS re:Invent 2021

NFX301
 Inter-VPC private IPv4 reachability
1 – Customer gateway

2 – VPC peering

3 – AWS Transit Gateway

AWS Cloud

VPC VPC

AWS
Transit Gateway

AWS re:Invent 2021

NFX301
 Inter-VPC private IPv4 reachability
1 – Customer gateway

2 – VPC peering

3 – AWS Transit Gateway

4 – Internet gateway

AWS Cloud

VPC VPC
Internet gateway Internet gateway

AWS re:Invent 2021

NFX301
 Inter-VPC IPv6 reachability
1 – Customer gateway

2 – VPC peering

3 – AWS Transit Gateway

4 – Internet gateway

AWS Cloud

VPC VPC
Internet gateway Internet gateway

AWS re:Invent 2021

NFX301
 Does IPv6 solve business network requirements?

Flat network Continued growth

No NAT No IPv4 address exhaustion
AWS limits (routing, etc.)

ENI density On premises

Prefix delegation

AWS re:Invent 2021

NFX301
 Does IPv6 solve business network requirements?

Flat network Continued growth

No NAT No IPv4 address exhaustion
Network setup not explicitly required

ENI density On premises

Prefix delegation

AWS re:Invent 2021

NFX301
 Does IPv6 solve business network requirements?

Flat network Continued growth

No NAT No IPv4 address exhaustion
Network setup not explicitly required

ENI density On premises

Prefix delegation Network setup not explicitly required

AWS re:Invent 2021

NFX301
SQUID GAME

Should we really do this?

AWS re:Invent 2021

NFX301
LA CASA DE PAPEL

Should we really do this?

Yes!

AWS re:Invent 2021

NFX301
 IPv6 co-innovation with AWS
Prefix delegation

2001:db8:96::/80→

Elastic network interface

AWS re:Invent 2021

NFX301
 IPv6 open issues with AWS
Prefix delegation

IPv6 transition mechanism

1
DNS64

Container 3 Instance
2001:db8::100 192.0.2.200

2
NAT64

Service discovery must be done via DNS!

AWS re:Invent 2021

NFX301
 IPv6 open issues with AWS
Prefix delegation

IPv6 transition mechanism

Security group Security group

1
DNS64

Container 3 Instance
2001:db8::100 192.0.2.200

2
NAT64

Security group references do not work!

AWS re:Invent 2021

NFX301
 Netflix innovation within AWS
Prefix delegation

IPv6 transition mechanism

Security group Security group

Titus container host

TSA

Container Elastic network Instance

2001:db8::100 interface 192.0.2.200
2001:db8::/80
192.0.2.100

Security group references do work!

More info at the Netflix Linux Plumbers Conference presentation

https://bit.ly/nflx-tsa

AWS re:Invent 2021

NFX301
 LOST IN SPACE

Netflix AWS VPC

IPv6 adoption progress

IPv6

Prefix delegation

In production
AWS re:Invent 2021
NFX301
 Netflix AWS IPv6 adoption progress 2021

IPv6: 25.2%

IPv4: 74.8%

AWS re:Invent 2021

Started 2021 with IPv6 being <1% of all interservice flows in VPC
NFX301
 Lessons learned

Old code is not fun, especially Java

AWS re:Invent 2021

NFX301
 Lessons learned

Old code is not fun, especially Java

Assigning IPv6 to a node does not mean IPv6 is used

AWS re:Invent 2021

NFX301
 Lessons learned

Old code is not fun, especially Java

Assigning IPv6 to a node does not mean IPv6 is used

Happy Eyeballs masks IPv6 problems

AWS re:Invent 2021

NFX301
 Lessons learned

Old code is not fun, especially Java

Assigning IPv6 to a node does not mean IPv6 is used

Happy Eyeballs masks IPv6 problems

Little IPv6 support for AWS Managed Services

AWS re:Invent 2021
NFX301
 Netflix best practices

Communication

AWS re:Invent 2021

NFX301
 Netflix best practices

Communication

Use BYOIP for IPv6 to have summarizable

address space, similar to private IPv4

AWS re:Invent 2021

NFX301
 Netflix best practices

Communication

Use BYOIP for IPv6 to have summarizable

address space, similar to private IPv4

Overlay IPv6 with IPv4

AWS re:Invent 2021

NFX301
 Netflix best practices

Communication

Use BYOIP for IPv6 to have summarizable

address space, similar to private IPv4

Overlay IPv6 with IPv4

Match IPv4 IP range rules with corresponding

IPv6 IP range rules
AWS re:Invent 2021
NFX301
 How do I get started?

AWS re:Invent 2021

NFX301
 How do I get started?

Dual stack workstations and dev machines

AWS re:Invent 2021

NFX301
 How do I get started?

Dual stack workstations and dev machines

Enable IPv6 on the edge

Update security groups
Update DNS records

AWS re:Invent 2021

NFX301
 How do I get started?

Dual stack workstations and dev machines

Enable IPv6 on the edge

Update security groups
Update DNS records

Focus on workloads in VPC

Overlay IPv6 with IPv4
Edge inwards
AWS re:Invent 2021
NFX301
 How do I show IPv6 is worthwhile to the business?

Economics

IPv6 is faster

Expensive boundary: public IPv4 vs. private IPv4

Middle boxes and operational risk

AWS re:Invent 2021

NFX301
 We have started the
journey to IPv6 in VPC
and so should you

STRANGER THINGS

AWS re:Invent 2021

NFX301
Thank you!
Donavan Fritz
[email protected]
@DonavanFritz
linkedin.com/in/DonavanFritz

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.