Kubernetes networking

Intro & deep dive

Murali Reddy, kube-router.io

me?
● Over decade working on cloud networking products
○ ASF CloudStack networking stack
○ SDN controller
○ NFV MANO orchestrator
○ Kubernetes networking
○ cloud-native mobile network
● Author of kube-router, Kubernetes turn-key networking solution
○ https://github.com/cloudnativelabs/kube-router
● Weave-net maintainer

@murali.reddy on Kubernetes slack, #kube-router

agenda
- networking model
- service discovery
- service load balancing
- network policies
- ingress

- service mesh
- multi cluster networking: service discovery, load balancing, security
- api gateway

Dual stack, cluster level services/policies, etc

evolution of cloud networking
● Infrastructure as API: ability to create network and services through API
● Shift from hardware ADC, middleboxes to software driven appliances
● Centralized to distributed load balancers, firewalls etc
● Networking (load balancer, proxy’s) to the edge
● Intent driven
● Network disaggregation
● microservice
○ service discovery
○ load balancing - both server and client side
○ east-west vs north-south traffic
○ microsegmentation
conceptual networking model -single cluster
conceptual networking model - multi cluster
Multi cloud and hybrid
single cluster requirements
- east-West traffic
- Pod-to-pod connectivity
- Service discovery
- Load balancing
- Network security
- Encryption
- north-south traffic
- Aaa (authentication, auditing, authorization)
- L4/L7 load balancing
- API gateways
- all traffic
- observability
Kubernetes network model
● https://kubernetes.io/docs/concepts/cluster-administration/networking/#the-kubernetes-ne
twork-model
● non-prescriptive of how the network should be designed
● fundamental requirements
○ All pods can communicate with all other pods without NAT
○ All nodes running pods can communicate with all pods (and vice-versa) without NAT
○ IP that a pod sees itself as is the same IP that other pods see it as
pod-to-pod connectivity
● CNI specification
● multiple CNI implementation options
○ Overlay solutions
■ Weave
■ Flannel
○ Direct routing
■ Calico
■ Kube-router
■ Flannel
○ Cloud network native solutions
■ Azure CNI
■ AWS VPC CNI
overlay pod networking
Pod a Pod b

a b
a b

Overlay network

Node A Node B

A B a b
A B a b
physical network
overlay pod networking (contd..)
● VXLAN, IP-IP, GRE tunneling
● Useful when you don't want to expose pod traffic to physical network
● Cloud provider does not allow encapsulated traffic
● Pod traffic gets dropped due to anti-spoofing policies (AWS src-dst checks)
● Overhead due to encapsulation
direct routing pod networking
Pod a Pod b

a b
a b

Node A Node B
route table route table

b node B a node A

a b a b
physical network
direct routing pod networking (contd ..)
Pod a Pod b

a b
a b

Node A bgp
Node B bgp

Route advertisement

a b a b

L3 routed network
direct routing pod networking (contd ..)
● No overhead due to encapsulation
● Suitable for on-prem clusters
● Direct access to the pod from outside the cluster
● Can achieve massive scale
● Fits naturally for multi-cluster scenarios
native VPC/cloud networking

Vpc router
Node 1 pod1 pod2

subnet1

Node 2 pod3 pod4

subnet2
native VPC/cloud networking (contd..)
● Pods are first class citizen
● Can use all services like ELB/ALB etc in case of AWS
● Treated same as VM instance
Network API resources
● Service, Endpoints, EndpointSlices
● Ingress
● NetworkPolicy
Lets see how in-cluster service-to-service talk
services: problem

Pod 1

Pod 2
Client
?
pod
Pod 3

● How to figure pod IP’s serving an app Pod 4

● Which one to connects
● Pods are ephermal
● Pods can become unhealthy Serving app
service abstraction

Pod 1
client connects to abstract service

Pod 2
Client
pod service
Pod 3

Pod 4

Service hides backend

Serving app
details
Services: what you specify
Services: what you get
service abstraction (contd ...)
● Services “expose” a group of pods declared through label selector
● A static VIP (virtual IP)
● Can use well known ports
● Automates service discovery
● Each service gets cluster resolvable DNS name

my-service.default.svc.cluster.local

Service name namespace Cluster DNS zone

service: data path
Core
DNS
Client does DNS service query Pod 1
Dns returns
service VIP
Pod 2

Service
pod proxy
Pod 3

node
Pod 4

Client
connects to Proxy
connects to Serving app
VIP
serving pod
Endpoint, endpointslices
● Represents the list of live pod IPs “behind” a Service
● Generally managed by system
● Out of band changes to achieve service reflection, VM’s etc
service: control plane

DNS

Service controller

proxy API server

Endpoints controller
node

● Service controller watches for service objects allocated cluster IP, Nodeport etc
● End points manager keeps endpoitns upto date
● DNS server updates records for the service
● Node proxy watches service, endpoints and configures to reflect desired state
services: nifty features
● Services without a pod selector
○ Reflect services in other clusters, or external services
○ Endpoint object should reflect the correct IP’s or service VIP
● Services without a pod selector but with ExternalName
○ Services of type ExternalName map a Service to a DNS nam
● Headless services
How external clients access in-cluster services
Service types
● ClusterIP
● NodePort
● LoadBalancer
● ExternalName
● Services with external IP configured
NodePort

Pod a Pod c
1.1.1.2 1.1.2.2
client :40001 :40001

:40002 Pod a :40002 Pod d

1.1.1.3 1.1.2.3

Src: client
Dst: 10.1.1.2 (node1) Node 1 Node 2
10.1.1.2 10.1.1.2
Src: 10.1.12
Dst: 1.1.2.3
NodePort (contd...)
With externalTrafficPolicy=local

Pod a Pod c
1.1.1.2 1.1.2.2
client :40001 :40001

:40002 Pod a :40002 Pod d

1.1.1.3 1.1.2.3

Src: client
Dst: 10.1.1.2 (node1) Node 1 Node 2
Src: client
10.1.1.2
Dst: 1.1.1.2 10.1.1.2
NodePort (contd...)
● No special/external infrastructure requirements
● Can’t use arbitrary/well known port
● Client has to pick a node
● Without externalTrafficPolicy=true we lose source IP
● Two hops
● With externalTrafficPolicy=true client has to pick a node with service pods
LoadBalancer with cloud provider LB
Cloud load balancer

Pod a Pod c
1.1.1.2 1.1.2.2
client VIP :40001 :40001

:40002 Pod a :40002 Pod d

1.1.1.3 1.1.2.3

Node 1 Node 2
10.1.1.2 10.1.1.2
Src: 10.1.12
Dst: 1.1.2.3
Src: client Src: client
Dst: VIP:port Dst: 10.1.1.2:40001
LoadBalancer with cloud provider LB
Cloud load balancer

Pod a Pod c
1.1.1.2 1.1.2.2
client VIP :40001 :40001

:40002 Pod a :40002 Pod d

1.1.1.3 1.1.2.3

Node 1 Node 2
10.1.1.2 10.1.1.2

Src: client Src: client

Dst: VIP:port Dst: 1.1.1.2:pod port
 LoadBalancer in on-prem with BGP + ECMP
Src: client Each node
Dst: vip:port advertises
service VIP
client L3 routers
VIP is available on
Src: client each node to
Dst: vip:port recieve traffic

vip vip
Pod a Pod c
Src: client 1.1.1.2 1.1.2.2
Dst: 1.1.1.2:port bgp bgp
Pod a Pod d
1.1.1.3 1.1.2.3

Node 1 10.1.1.2 Node 2 10.1.1.2

 LoadBalancer in on-prem with BGP + ECMP
Src: client
Dst: vip:port

client L3 routers
VIP is configured on
Src: client single node that
Dst: vip:port responds to ARP

vip
Pod a Pod c
Src: client 1.1.1.2 1.1.2.2
Dst: 1.1.1.2:port

Pod a Pod d
1.1.1.3 1.1.2.3

Node 1 10.1.1.2 Node 2 10.1.1.2

network policies
● Declarative way or intent based securing the applications
● Describes allowed call graph between services
● Works based on label selector primitive
● Intended for app owners, not for the cluster/network operator
● L4 only
● Standard allow,deny semantics
● Mostly for east-west traffic
● Supports ipblock for north-south traffic
network policies

Frontend backend pod Frontend backend pod

pod 1 1 pod 2 2
node1 node2

firewall firewall
Check is made if
Check is made if pod can receive
pod can send traffic from
traffic to backend fronent pod
pod2 Change is picked up
by each node and
necessary
netfilter/bpf rules etc User applies network
API server policy with egress &
are setup
ingress rule to permit
traffic from frontend pods
to back end pods
ingress
● API object that manages external access to the services in a cluster, typically
HTTP
● exposes HTTP and HTTPS routes from outside the cluster to services within
the cluster
● Ingress does not expose arbitrary ports or protocols uses uses a service of
type Service.Type=NodePort or Service.Type=LoadBalancer
● Describes HTTP proxy and routing rules
● Targets service for each rule
ingress
API gateway
● api gateway is used for application routing, rate limiting, security, request and
response handling and other application related tasks
● Overlaps some of the functionalities with ingress controllers
● Kong, ambassador, AWS api gateway etc
service mesh
With current Kubernetes networking lacks

● Identity - who is sending and receiving

● Access control - who may send and receive
● Encryption and zero trust
● Traffic management
● Telemetry
● Client side load balancing

Istio
● Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic.
● Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault injection.
● A pluggable policy layer and configuration API supporting access controls, rate limits and quotas.
● Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress.
● Secure service-to-service communication in a cluster with strong identity-based authentication and authorization.
conceptual networking model - multi cluster
Multi cloud and hybrid
thanks