PayPal’s Private Cloud @ Scale
Jigar Desai• @jigardesai • March 17 2016

©2015 PayPal Inc. Confidential and proprietary.

 Introduction to PayPal
Table of contents
Current Cloud Stack

SDN Journey

Future Cloud Stack

Q&A

©2015 PayPal Inc. Confidential and proprietary. 2

Introduction to PayPal
A Technology Perspective

©2015 PayPal Inc. Confidential and proprietary. 3

About PayPal
PayPal is a leading technology platform company that enables digital and mobile payments on behalf of
consumers and merchants worldwide. We put our customers at the center of everything we do. We
strive to increase our relevance for consumers, merchants, friends and family to access and move their
money anywhere in the world, anytime, on any platform and through any device.

©2015 PayPal Inc. Confidential and proprietary. 4

 Architecture @ PayPal
Robust Infrastructure, Reusable Platforms, Payments Operating System, Delightful Experiences
Experience
Customer Facing

Payments Operating System

Identity Login Payment Wallet Cart Risk Coupons Email POS Notify Merchant Invoice Ship Support

Technology Platform Analytics Platform

Infrastructure

Framework Data Access Cache SOA Messaging Tools Crawler Media Teradata BI FPTI Hadoop

Infrastructure & Operations Layer

Power Data Center Hardware Network Storage Monitoring Cloud Database Tools Operations

©2015 PayPal Inc. Confidential and proprietary. 5

Cloud@PayPal

©2015 PayPal Inc. Confidential and proprietary. 6

 Key Statistics
Developer Statistics

x 1000 x 10M x 1000 x 1000 x 1000

engineers Lines Releases/ Builds/ Deploy/day
year day

OpenStack Cloud

3 Regions 9 Availability
Zones

> 10K > 1000

x 100K x 10K x 10 PB
Physical Services
Cores VMs Storage
Servers

© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.

Impact On Life Of A Developer
2 years back

Steps to Production 25 Manual Steps Automated

Time to Market 8 Weeks 1 Day

QA Deployment 5 Hours 30 Minutes

Now
Build Time 2.5 Hours 5 Minutes

Prod Deployment 4 Hours < 30 Minutes

Infrastructure 1 Month 30 Minutes

© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.

The Current Cloud Stack

• 100% of web and mid tier

Web/Mid Tier Applications applications on OpenStack
• Proprietary PaaS on top of
OpenStack to automate E2E
Common Platforms and application development life
Services cycle
Platform as a Service • Support for polyglot
applications
OpenStack Cloud • Test and production
(Infrastructure as a Service) Common Infrastructure environments available on-
demand
Datacenters
(Geographic regions, Availability Zones)

©2015 PayPal Inc. Confidential and proprietary. 9

Cloud SDN Journey

©2015 PayPal Inc. Confidential and proprietary. 10

Why SDN at PayPal?

• Ability to logically isolate cloud resources (compute, storage, network) for

different business use cases needing different security policies while co-existing
on shared infrastructure
• Solve compute capacity islands by moving computes between security zones as
needed
• Programmatic APIs to enable both cloud users and operator reducing operational
overhead
• Use cases:
• External zone hosting beta apps reachable from internet but separated from
other zones
• Developer zone hosting all developer tools with no direct access from internet
but available from corp

©2015 PayPal Inc. Confidential and proprietary. 11

Our SDN Architecture

Neutron API Overlay

•PayPal deploys multiple VPCs (Virtual
Network Bridged
Network
Private Cloud)s in every AZ
•Every overlay VPC is running on a
per VPC
Neutron Core per VPC

LBaaS SDN Plugin

Plugin
separate routed network and firewalled off
LB Specific •Bridged VPCs are deployed using VRF
Driver(s) LB for
LB Agent1 … n Controllers (Virtual Routing and Forwarding)
SDN APIs SDN

Gateway
•SDN controllers are horizontally scaled

SDN Controller-1
SDN Controller-2 OVS DB 

LB Device SDN Controller-3
APIs
SDN Controller-4
SDN Controller-n

OVS DB
OVS DB
LB Pair-1
LB Pair-2
LB Pair - ..
LB Pair-n
Open vSwitch Open vSwitch

VM1 VM2 VM1 VM2

Global Platform & Infrastructure Overlay Hypervisors 1….n Bridge Hypervisors 1….n
SDN Challenges
• Off the shelf solutions are not battle-tested for scale well
• Example: Control plane doesn’t scale well beyond 2500 hypervisors
• Bit early in software maturity curve
• Example: Latency issues with early OVS implementation
• Security groups performance issues with multiple rules. Later solved with
“megaflows” feature
• Strong architectural principles don’t always mean strong implementation
• Initial SDN implementation didn’t have full separation between control plane
and data plane. Control plane outages resulted in site outages
• Security concerns specially around centralized control plane

©2015 PayPal Inc. Confidential and proprietary. 13

The Future

©2015 PayPal Inc. Confidential and proprietary. 14

The Future Cloud Stack

• New PaaS is being built

Web and Mid Tier Applications using Mesos and Docker on
top of OpenStack
• Goal is to achieve industry
Common Platforms and leading datacenter efficiency
Services and utilization
• High application resiliency
against infrastructure
OpenStack Cloud Public failures
(Infrastructure as a Service) Clouds Common Infrastructure • Hybrid cloud POC
Datacenters • PaaS support for stateful
(Geographic regions, Availability Zones) applications

©2015 PayPal Inc. Confidential and proprietary. 15

Container Networking

©2015 PayPal Inc. Confidential and proprietary. 16

Container Networking Needs
Needs:
• Containers as first class citizens of the network
• IP per container or per pod
• Container IP must be a private address routable within the AZ
• Low latency container to container communication
• Support up to 100 containers per host
• Distributed SLB
• IP Mobility within the AZ for stateful apps
• Distributed firewall

Exploratory Options:
Non-overlay: Ipvlan, Macvlan
Overlay: Vxlan, MPLS

©2015 PayPal Inc. Confidential and proprietary. 17

Questions?
Jdesai at paypal dot com