#CiscoLive

Understanding Multicluster
Kubernetes Connectivity
Options
Shannon McFarland – CCIE #5245
Distinguished Engineer
@eyepv6
BRKETI-2003

#CiscoLive
Cisco Webex App

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here

4 Enter messages/questions in the Webex space

Webex spaces will be moderated https://ciscolive.ciscoevents.com/ciscolivebot/#BRKETI-2003

by the speaker until June 17, 2022.

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 • Multicluster Connectivity – Pain points
• Kubernetes Services, Ingress, Load
Balancer
Agenda • Cilium Cluster Mesh
• Service Mesh
• Linkerd
• Istio

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Multicluster Connectivity – What is all the
fuss about?
• There are several reasons for establishing connectivity between Kubernetes clusters
to include:
• Service load balancing
• Data replication Network
• Service dependencies Services
• Partner-provided service connectivity FW VPN
• etc..

• Today, many assumptions are made about the underlying infrastructure that exists
underneath and in between these clusters:
• Use ingress/load balancers and let basic networking and name resolution sort it out CNI CNI

• Intra-VPC/Intra-network – Deploy the clusters in the same VPCs/networks to facilitate

App App App App
easier connectivity Pod Pod Pod Pod
• Inter-VPC/Inter-network – Networking is already built and managed (Hybrid cloud,
VPC peering, etc.)

• Regardless of the assumptions or justification, something and someone has to deal

with service-to-service connectivity – let’s explore some options

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Multicluster Connectivity – Options Galore!
• There are many solutions for linking workloads that are hosted on different
Kubernetes clusters – let’s look at a few of them
• CNI-based
• Cilium Cluster Mesh – Global load balancing is great – Service-to-service can be
dicey
• Gateway-based
• Submariner – A Layer 3/4 centric approach – Service-to-service is a strength -
Not the smoothest implementation
• Application Service Mesh-based – Layer 4/7 networking, robust security
and observability
• Linkerd
• Istio
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
A Couple of Use Cases
Global Load Balancing
Service-to-Service/Pod-to-Pod
DN DN
S S
Load Balancer Load Balancer

Cluster1 Cluster2 Cluster1 Cluster2

Load Balancer Load Balancer Load Balancer Load Balancer

Service Service Service Service

Pod Pod Pod Pod

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 Reference

A Note About Shared Service Naming

Cilium Cluster Mesh Submariner Linkerd Istio Cisco Calisti
(Istio)

Service Unchanged – Global LB <SVC>.default.svc.clu

sterset.local
<SVC-CLUSTER>-
.default.svc.cluster.local
Unchanged – Global LB Same as
Name Custom service - Create Custom service – Create Istio
phantom/ghost service phantom/ghost service for
for service-to-service service-to-service use case
use case
Example my-custom-service-
name.default.svc.cluster.local
redis-
cart.default.svc.clusters
redis-cart-
cluster1.default.svc.cluster.lo
my-custom-service-
name.default.svc.cluster.local
Same as
et.local cal Istio

Special kind: Service

metadata:
Config name: redis-cart
annotations:
io.cilium/global-service: "true"

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Kubernetes
Services, Ingresses,
Load Balancers
 K8s Multicluster Connectivity – Using Ingress,
Services, LBs
Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

Load Balancer db-leader.cls1.example.com db-follower.cls2.example.com

Load Balancer

Ingress Ingress
Pain:
db-leader • Global DNS db-follower

Service • Multiple trust boundaries Service

• HTTP/TCP centric connections

K8s Cluster Deployment • Operational boundaries Deployment
K8s Cluster
• No single view of service
connectivity
db-leader- • Finger pointing db-follower-
pod pod
• Horrid root cause analysis

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Cilium CNI + Cilium
Cluster Mesh
Cilium Cluster Mesh
• eBPF-based Networking, Observability and Security:
• https://cilium.io/
• A CNCF project

• Cilium Cluster Mesh: https://docs.cilium.io/en/stable/gettingstarted/#cluster-mesh

• https://cilium.io/blog/2019/03/12/clustermesh
• It isn’t a traditional Application Service Mesh ☺
• Define globally load balanced services that span Kubernetes clusters
• etcd state shared via load-balancers / Nodes communicate over VXLAN / Encryption over
IPSec
• Selective load balancing to remote clusters is possible but difficult depending on the scenario
• Connect to external workloads (e.g., VMs)

• Outcome: It just worked, but it may not be what you need

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Microservices Demo Topology
Cluster1 Cluster2

https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 K8s Multicluster Connectivity – Global Service LB
Setup # cilium install --cluster-name shmcfarl-mc-2 --cluster-id 2 --context
[email protected] --inherit-ca shmcfarl@shmcfarl-mc-
1.us-west-2.eksctl.io

VXLAN Networking Service

VPC peering, Hybrid Cloud, etc.

DNS CA

Load Load # cilium install --cluster-name shmcfarl-mc-1 --cluster-id 1 Load

Balancer Balancer Balancer

Cilium Cluster Mesh

Ingress

redis-cart
frontend

K8s Cluster cartservice redis-cart K8s Cluster

redis-cart # cilium clustermesh enable --context [email protected]

# cilium clustermesh enable --context [email protected]

Control Plane Connectivity: LoadBalancer, NodePort, ClusterIP

redis-cart
# cilium clustermesh connect --context [email protected]
--destination-context [email protected]

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 K8s Multicluster Connectivity – Global Service LB
VXLAN
VXLAN Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

Load Load
Balancer Balancer
# cilium bpf tunnel list Node IP:
TUNNEL VALUE 10.11.121.200
172.17.0.0:0 10.10.123.229:0
Node IP: 172.18.0.0:0 10.11.121.200:0 Cluster CIDR:
10.10.123.229 172.18.0.0/16

Cluster CIDR:
172.17.0.0/16
K8s Cluster

K8s Cluster
Cluster 172.17.0.0/16 Cluster 172.18.0.0/16

10.10.123.2 10.11.121.2
29 00

VXLAN
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 K8s Multicluster Connectivity – Global Service LB
Global LB

Cilium Cluster Mesh

Ingress

Global service defined

redis-cart
apiVersion: v1
frontend kind: Service 172.18.115.114:6379
metadata:
name: redis-cart
redis-cart
K8s Cluster cartservice annotations: K8s Cluster
io.cilium/global-service: "true" 10.11.126.169:6379

redis-cart # cilium service list

172.17.17.201:6379 ID Frontend Service Type Backend
7 172.18.115.114:6379 ClusterIP 1 => 10.10.80.2:6379
2 => 10.11.126.169:6379

redis-cart 10.10.80.2:6379 # cilium service list

ID Frontend Service Type Backend
20 172.17.17.201:6379 ClusterIP 1 => 10.11.126.169:6379
2 => 10.10.80.2:6379
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 K8s Multicluster Connectivity – Service-to-
Service
Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

Load Balancer

Cilium Cluster Mesh

Ingress redis-cart-
cls1

redis-cart
apiVersion: v1
frontend kind: Service 172.18.115.114:6379
metadata:
name: redis-cart-cls1
redis-cart
K8s Cluster cartservice annotations:
io.cilium/global-service: "true 10.11.126.169:6379
10.96.144.119:6379 spec: K8s Cluster
redis-cart- type: ClusterIP
cls1 selector:
redis-cart apiVersion: v1
app: redis-cart kind: Service
metadata:
name: redis-cart-cls1
annotations:
redis-cart 10.244.1.231:6379 io.cilium/global-service: "true
NOTE: No selector spec:
for “redis-cart” app type: ClusterIP

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 K8s Multicluster Connectivity – Service-to-
Service
# cilium service list # cilium service list
ID Frontend Service Type Backend ID Frontend Service Type Backend
23 10.96.89.208:6379 ClusterIP 1 => 10.244.1.231:6379 26 10.97.136.93:6379 ClusterIP 1 => 10.245.1.219:6379
30 10.96.144.119:6379 ClusterIP 1 => 10.244.1.231:6379 32 10.97.246.126:6379 ClusterIP 1 => 10.244.1.231:6379

Load Balancer

Cilium Cluster Mesh

Ingress redis-cart-
cls1

redis-cart
10.97.136.93:6379 10.97.246.126:6379
frontend

redis-cart
K8s Cluster cartservice
10.245.1.219:6379
10.96.144.119:6379 K8s Cluster
redis-cart-
10.96.89.208:6379 cls1
redis-cart apiVersion: v1
kind: Service
metadata:
name: redis-cart-cls1
annotations:
redis-cart 10.244.1.231:6379 io.cilium/global-service: "true
NOTE: No selector spec:
for “redis-cart” app type: ClusterIP

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 K8s Multicluster Connectivity – Service-to-
Service
# cilium service list # cilium service list
ID Frontend Service Type Backend ID Frontend Service Type Backend
23 10.96.89.208:6379 ClusterIP 1 => 10.244.1.231:6379 26 10.97.136.93:6379 ClusterIP 1 => 10.245.1.219:6379
30 10.96.144.119:6379 ClusterIP 1 => 10.244.1.231:6379 32 10.97.246.126:6379 ClusterIP 1 => 10.244.1.231:6379

Load Balancer

Cilium Cluster Mesh

Ingress redis-cart-
cls1

redis-cart
10.97.136.93:6379 10.97.246.126:6379
frontend

redis-cart
K8s Cluster cartservice
10.245.1.219:6379
10.96.144.119:6379 K8s Cluster
redis-cart-
10.96.89.208:6379 cls1
redis-cart
127.0.0.1:6379> replicaof redis-cart-cls1.default.svc.cluster.local 6379
127.0.0.1:6379> keys *
1) "5c605f89-0f26-41e5-a3b2-fe6d1962be7f”

redis-cart 10.244.1.231:6379

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Submariner
Submariner
• Gateway-based multicluster connectivity for Kubernetes services:
https://submariner.io/
• A CNCF project
• What is it?
• Gateway-based with support for IPSec (libreswan), WireGuard and VXLAN
• Connect ‘exported’ services between clusters
• Can be used as a transport for other stuff like Istio: https://cloud.redhat.com/blog/set-up-
istio-multicluster-with-submariner-in-red-hat-advanced-cluster-management-for-kubernetes

• Outcome: It is a very bumpy deployment. Fairly ’smooth’ on OpenShift, but bumpy

on most other platforms due to out-of-date docs and buggy dependency scripts
• Things to watch out:
• MTU on pods – Must account for overhead of IPSec/Wireguard/VXLAN
• Security groups – pay close attention to the SG dependencies per encap type

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Microservices Demo Topology
Cluster1 Cluster2

Leader Follower

https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 K8s Multicluster Connectivity – Submariner Service Export
Setup # subctl show all
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
cluster2-worker cluster2 172.18.0.5 no libreswan 100.2.0.0/16, 10.2.0.0/16 connected 152.062µs

IPSec, WireGuard, VXLAN Networking Service

VPC peering, Hybrid Cloud, etc.

DNS CA

# subctl join broker-info.subm --clusterid cluster1

# subctl join broker-info.subm --clusterid cluster2

RA RA
Ingress
GA GA

redis-cart
frontend
Broker
# subctl deploy-broker
redis-cart
K8s Cluster cartservice K8s Cluster
LA LA
cluster-2
redis-cart
# subctl export service --namespace default redis-cart

RA=Route Agent
redis-cart
cluster-1
GA=Gateway Agent
LA=Lighthouse Agent
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 K8s Multicluster Connectivity – Submariner Service Export
Make the Service Known
IPSec, WireGuard, VXLAN Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

RA RA
Ingress
GA GA

redis-cart
frontend
Broker
# kubectl get serviceexport -A redis-cart
K8s Cluster cartservice NAMESPACE NAME AGE K8s Cluster
default redis-cart 11m
LA LA
cluster-2
redis-cart
# subctl export service --namespace default redis-cart
100.1.39.208:6379
100.1.39.208 = redis-
cart.default.svc.clusterset.local
redis-cart
# kubectl get serviceimport -A
cluster-1 NAMESPACE NAME TYPE IP AGE
submariner-operator redis-cart-default-cluster1 ClusterSetIP ["100.1.39.208"] 13m
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 K8s Multicluster Connectivity – Submariner Service Export
Redis Replication
IPSec, WireGuard, VXLAN Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

RA RA
Ingress
GA GA

redis-cart
frontend
Broker
# kubectl get serviceexport -A redis-cart
K8s Cluster cartservice NAMESPACE NAME AGE K8s Cluster
default redis-cart 11m
LA LA
cluster-2
redis-cart
# subctl export service --namespace default redis-cart
100.1.39.208:6379
100.1.39.108 = redis-
cart.default.svc.clusterset.local
127.0.0.1:6379> replicaof redis-cart.default.svc.clusterset.local 6379
redis-cart 127.0.0.1:6379> keys *
1) "1991ffa8-ffa0-49d4-91d2-d165590b35b0"
cluster-1
127.0.0.1:6379> client list
. . .
id=29 addr=100.1.39.208:6379
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Linkerd
 Reference

An open source service mesh

and CNCF project.

🔥 4 years in production
🔥 5,000+ Slack channel
members
🔥 10,000+ GitHub stars
🔥 100+ contributors

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 Reference

What does it do?

• Observability: Service-level golden
metrics: success rates, latencies,
throughput. Service topologies.

• Reliability: Retries, timeouts, load

balancing, circuit breaking

• Security: Transparent mTLS, cert

management and rotation, policy

In an ultralight package focused on

operational simplicity first and
foremost.

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Reference

Linkerd Design

• In short, "do less, not more"

• Just works: Zero config, out of the box, for any Kubernetes app
• Ultralight: Introduce the bare minimum perf and resource cost
• Simple: Reduce operational complexity in every possible way
• Minimal overhead:
• Control plane: Go. ~200mb RSS (excluding metrics data). (Repo: linkerd/linkerd2).
• Data plane: Rust. <10mb RSS (Resident Set Size), <1ms p99
(Repo: linkerd/linkerd2-proxy)

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 https://linkerd.io/2/reference/architecture/
Linkerd 2.x Architecture

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 Reference

Linkerd: How Do I Get It?

• Where to get it:

• https://linkerd.io/2/getting-started/
• Releases: https://github.com/linkerd/linkerd2/releases/

• Deploy a Kubernetes Cluster

• Deploy Linkerd
• Deploy (or add) Linkerd to your microservice(s)

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Reference

Get involved!
• Linkerd has a friendly, welcoming community! Join us!
• Development is all on https://github.com/linkerd
• Thriving community in the https://slack.linkerd.io/
• Formal announcements on the
CNCF https://lists.cncf.io/g/cncf-linkerd-users
• Linkerd is 100% Apache v2 licensed, owned by a neutral foundation
(https://www.cncf.io/), and is
https://linkerd.io/2019/10/03/linkerds-commitment-to-open-
governance/.

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Microservices Demo Topology
Cluster1 Cluster2

Leader Follower

https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 Reference
Linkerd Multicluster Setup – Pre-setup Stuff
https://linkerd.io/2.11/getting-started/ https://linkerd.io/2.11/features/multicluster/

• Create certs
# step certificate create root.linkerd.cluster.local root.crt root.key \
--profile root-ca --no-password --insecure

# step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \

--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca root.crt --ca-key root.key

• Install Linkerd and create an anchor of trust between the clusters

# linkerd install \
--identity-trust-anchors-file root.crt \
--identity-issuer-certificate-file issuer.crt \
--identity-issuer-key-file issuer.key \
| tee \
>(kubectl --context=linkerd-mc-1 apply -f -) \
>(kubectl --context=linkerd-mc-2 apply -f -)

• Follow the documented steps for addons such as Linkerd ‘viz’ (UI)
installation
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 K8s Multicluster Connectivity – Linkerd Multicluster
Setup
Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

Load Load Load

# for ctx in linkerd-mc-1 linkerd-mc-2; do
Balancer Balancer Balancer
echo "Installing on cluster: ${ctx} ........."
linkerd --context=${ctx} multicluster install | \
kubectl --context=${ctx} apply -f - || break
Ingress echo "-------------"
done

redis-cart
frontend

K8s Cluster cartservice redis-cart K8s Cluster

redis-cart linkerd-multicluster linkerd-gateway LoadBalancer 172.17.92.236

a43fbc6255d064eb5bcac425574e1cf0-2075359419.us-west-2.elb.amazonaws.com
4143:31142/TCP,4191:32606/TCP 16m app=linkerd-gateway

linkerd-multicluster linkerd-gateway LoadBalancer 172.20.206.27

redis-cart a00f524023a5c491293e318d97e27807-1943766645.us-west-2.elb.amazonaws.com
4143:32485/TCP,4191:30346/TCP 15m app=linkerd-gateway

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 K8s Multicluster Connectivity – Linkerd Multicluster
Link Setup # linkerd --context=linkerd-mc-1 multicluster link --cluster-name linkerd-mc-1 |
kubectl --context=linkerd-mc-2 apply -f -
# linkerd --context=linkerd-mc-2 multicluster link --cluster-name linkerd-mc-2 |
kubectl --context=linkerd-mc-1 apply -f -
Networking Service
VPC peering, Hybrid Cloud, etc.

DNS CA

Load Load Load

Balancer Balancer Balancer

Ingress

redis-cart
frontend

K8s Cluster cartservice redis-cart K8s Cluster

redis-cart
linkerd-multicluster linkerd-service-mirror-shmcfarl-mc-1-6b68f84b64-ghwkg

linkerd-multicluster linkerd-service-mirror-shmcfarl-mc-2-779d9f874d-w9g94
redis-cart

Rule: ‘Link’ on cluster you want to share to

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 Rule: ‘Link’ on cluster
Linkerd Multicluster – Service Export you want to share to

Load AWS ELB: Load

Balancer 52.13.223.167 Balancer

Multicluster Multicluster
Gateway Gateway

service mirror:
redis-cart-
linkerd-mc-1
172.20.124.207
service:
redis-cart

pod: redis- pod: redis-

cart cart

# linkerd --context=linkerd-mc-1 multicluster link --cluster-name linkerd-mc-1 |

kubectl --context=linkerd-mc-2 apply -f -
# kubectl --context=linkerd-mc-1 label svc -n default redis-cart mirror.linkerd.io/exported=true
Name: redis-cart # kubectl --context=linkerd-mc-2 get endpoints
Namespace: default NAME ENDPOINTS AGE
Labels: mirror.linkerd.io/exported=true kubernetes 10.10.154.115:443,10.10.189.103:443 2d3h
redis-cart-linkerd-mc-1 52.13.223.167:4143 145m
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Sample of “link” YAML output (highly reduced output) Reference
apiVersion: multicluster.linkerd.io/v1alpha1
kind: Link
metadata:
name: linkerd-mc-1
namespace: linkerd-multicluster
spec:
clusterCredentialsSecret: cluster-credentials-linkerd-mc-1
gatewayAddress: a9d97fc75ed1d43b19e2a3344ad734cc-1322698043.us-west-2.elb.amazonaws.com
gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
gatewayPort: "4143"
probeSpec:
path: /ready
period: 3s
port: "4191"
selector:
matchExpressions:
- key: mirror.linkerd.io/exported
operator: Exists
targetClusterDomain: cluster.local
targetClusterLinkerdNamespace: linkerd
targetClusterName: linkerd-mc-1
---
apiVersion: v1
kind: Service
metadata:
name: probe-gateway-linkerd-mc-1
namespace: linkerd-multicluster
labels:
mirror.linkerd.io/mirrored-gateway: "true"
mirror.linkerd.io/cluster-name: linkerd-mc-1
spec:
ports:
- name: mc-probe
port: 4191
protocol: TCP

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Linkerd Multicluster – Redis Replication
Load Load
Balancer Balancer

Multicluster Multicluster
Gateway Gateway

service mirror:
redis-cart-
linkerd-mc-1
172.20.124.207
service:
redis-cart

pod: redis- pod: redis-

cart cart

127.0.0.1:6379> replicaof redis-cart-linkerd-mc-1.default.svc.cluster.local 6379

127.0.0.1:6379> keys *
1) "72c6d371-812f-4778-80c8-c41366d956a6”

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Istio
Istio Overview
• An open-source project started by Google and IBM with help from the
Envoy team at Lyft
• https://istio.io/
• https://github.com/istio
• https://www.envoyproxy.io/

• https://istio.io/docs/concepts/what-is-istio/
• Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic
• Robust multicluster connectivity
• Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault
injection
• A pluggable policy layer and configuration API supporting access controls, rate limits and
quotas
• Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and
egress
• Secure service-to-service authentication with strong identity assertions between services in a
cluster
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
 Istio Architecture
https://istio.io/latest/docs/ops/deployment/architecture/
• istiod
• Pilot
• Handles service discovery and config data
• Provides the Envoy proxies with the mesh topology and
route rules
• Galley
• Validates user authored Istio API configuration on behalf of
other control plane components
• Top-level config ingestion, processing and distribution
• Citadel
• Provides certificates to the Envoy proxies for authentication
and authorization

• Envoy
• A proxy attached to every microservice
• The connection point for a microservice to attach to the
mesh

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Envoy
https://www.envoyproxy.io
• Implemented by Lyft • API driven

• A C++ based L4/L7 proxy • Traffic routing and splitting

• Can be used independently of any service • Transparent proxying

mesh (Istio)
• Health checks, circuit breakers, etc.

https://github.com/envoyproxy/envoy
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Istio: How Do I Get It?
• Where to get it:
• Istio currently is available directly from the Istio community at:
https://istio.io/about/community/join/
• It can also be built directly: https://github.com/istio/istio
• It can be enabled as an infrastructure option in GKE

• How to install it (Kubernetes):

• https://istio.io/docs/setup/getting-started/
• Kubernetes installation is a prerequisite
• Directly from the manifests included in the release
• Using Helm charts included in the release

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 Reference

Contribution
• Contribution Readme:
https://github.com/istio/community/blob/master/CONTRIBUTING.md
• Contributing to the Docs: https://istio.io/about/contribute/
• Istio Discussion: https://discuss.istio.io/

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Istio Multicluster
• https://istio.io/latest/docs/ops/deployment/deployment-models/
• Primary-Remote – single network
• Primary-Remote – multiple networks
• Multi-Primary – single network
• Multi-Primary – multiple networks

• “single network” ->> “flat networking”, “shared networking” = full reachability

between workloads without an Istio gateway
• “multiple network” – Workloads reach each other via an Istio gateway
• Pre-planning:
• Service naming
• Istio DNS proxy
• Service sharing/exposure – Control at the gateway or via Istio Authorization:
https://istio.io/latest/docs/reference/config/security/authorization-policy/

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Microservices Demo Topology
Cluster1 Cluster2

Leader Follower

https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Reference
Istio Multicluster Setup – Pre-setup Stuff
https://istio.io/latest/docs/setup/install/multicluster/before-you-begin/
• Create certs and secrets on first cluster and 2nd clusters
# make -f ../tools/certs/Makefile.selfsigned.mk root-ca
# make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
# kubectl create namespace istio-system
# kubectl create secret generic cacerts -n istio-system \
--from-file=cluster1/ca-cert.pem \
--from-file=cluster1/ca-key.pem \
--from-file=cluster1/root-cert.pem \
--from-file=cluster1/cert-chain.pem
# make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts
# kubectl create namespace istio-system
# kubectl create secret generic cacerts -n istio-system \
--from-file=cluster2/ca-cert.pem \
--from-file=cluster2/ca-key.pem \
--from-file=cluster2/root-cert.pem \
--from-file=cluster2/cert-chain.pem

• Export context info for future use with ‘kubectl’ and ‘istioctl’
#commands
# export CTX_CLUSTER1=istio-mc-1
export CTX_CLUSTER2=istio-mc-2

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 K8s Multicluster Connectivity – Istio Multicluster
Setup – “istio-mc-1”
Networking Service
VPC peering, Hybrid Cloud, etc.

istio- istio- DNS CA

ingress eastwest-gw

Load Load Load

# ***kubectl --context="${CTX_CLUSTER1}" label
Balancer Balancer Balancer
namespace istio-system
topology.istio.io/network=network1

Ingress
# cat <<EOF > istio-mc-1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator redis-cart
frontend spec:
values:
global:
meshID: mesh1
K8s Cluster cartservice multiCluster:
redis-cart K8s Cluster
clusterName: istio-mc-1
network: network1
istio-mc-2
EOF
redis-cart
# istioctl install --context="${CTX_CLUSTER1}" -f istio-mc-1.yaml

redis-cart
istio-mc-1 # samples/multicluster/gen-eastwest-gateway.sh \

network1 --mesh mesh1 --cluster istio-mc-1 --network network1 | \

istioctl --context="${CTX_CLUSTER1}" install -y -f -

***Do this on existing Istio deployment #CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 K8s Multicluster Connectivity – Istio Multicluster
Setup – “istio-mc-2”
Networking Service
VPC peering, Hybrid Cloud, etc.

istio- istio- DNS CA istio- istio-

ingress eastwest-gw ingress eastwest-gw
Load Load Load Load Load
# kubectl --context="${CTX_CLUSTER2}" label namespace
Balancer Balancer Balancer Balancer Balancer
istio-system topology.istio.io/network=network2

Ingress
# cat <<EOF > istio-mc-2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator redis-cart
frontend spec:
values:
global:
meshID: mesh1
K8s Cluster cartservice multiCluster:
redis-cart K8s Cluster
clusterName: istio-mc-2
network: network2
istio-mc-2
EOF network2
redis-cart
# istioctl install --context="${CTX_CLUSTER2}" -f istio-mc-2.yaml

redis-cart
istio-mc-1 # samples/multicluster/gen-eastwest-gateway.sh \

network1 --mesh mesh1 --cluster istio-mc-2 --network network2 | \

istioctl --context="${CTX_CLUSTER2}" install -y -f -

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Warning: It is ALWAYS DNS that kills you ☺
Add to the Istio Operator Config
• By default, Istio does not enable DNS apiVersion: install.istio.io/v1alpha1

proxy for services that are exposed to kind: IstioOperator

spec:

another cluster meshConfig:

defaultConfig:
proxyMetadata:

https://istio.io/latest/docs/ops/configuratio
# Enable basic DNS proxying
• ISTIO_META_DNS_CAPTURE: "true"

n/traffic-management/dns-
proxy/#getting-started OR edit the config post-deployment
# kubectl edit istiocontrolplanes -n istio-system
meshConfig:
• Without enabling DNS proxy, “redis-cart- defaultConfig:
. . .<output_summarized>

cls1.default.svc.cluster.local” will not be proxyMetadata:

ISTIO_META_ALS_ENABLED: "true"

resolvable on the 2nd cluster ISTIO_META_DNS_CAPTURE: "true"

PROXY_CONFIG_XDS_AGENT: "true"

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 K8s Multicluster Connectivity – Istio Multicluster
Expose Services – “istio-mc-1”
Networking Service Expose services
VPC peering, Hybrid Cloud, etc.

istio- istio- DNS CA istio- istio-

ingress eastwest-gw ingress eastwest-gw
Load Load Load # cat <<EOF > expose-services.yaml Load Load
Balancer Balancer Balancer apiVersion: networking.istio.io/v1alpha3 Balancer Balancer
kind: Gateway
metadata:
name: cross-network-gateway
Ingress
spec:
selector:
istio: eastwestgateway
servers: redis-cart
frontend - port:
number: 15443
name: tls
protocol: TLS
K8s Cluster cartservice tls:
redis-cart K8s Cluster
mode: AUTO_PASSTHROUGH
hosts:
istio-mc-2
- "*.local” network2
redis-cart EOF
# kubectl --context="${CTX_CLUSTER1}" apply -n

istio-mc-1
istio-system -f expose-services.yaml

redis-cart
network1 Service-specific Example:
hosts:
- ”<SVC>.default.svc.cluster.local”
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 Reference
Istio Multicluster Setup – Endpoint Discovery
• Install remote secrets in both clusters so that each cluster has API
server access to the other cluster
# istioctl x create-remote-secret \
--context="${CTX_CLUSTER1}" \
--name=istio-mc-1 | \
kubectl apply -f - --context="${CTX_CLUSTER2}"

# istioctl x create-remote-secret \
--context="${CTX_CLUSTER2}" \
--name=istio-mc-2 | \
kubectl apply -f - --context="${CTX_CLUSTER1}"

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Istio Multicluster – Redis Replication
Service Mirror – Phantom/Ghost Services
istio- name: cross-network-gateway
eastwest-gw spec:
Load selector:
Balancer AWS LB: 52.11.49.96 istio: eastwestgateway
servers:
- port:
number: 15443
name: tls
protocol: TLS
172.17.43.144 tls:
redis-cart- mode: AUTO_PASSTHROUGH service: redis-cart
cls1
service:
redis-cart
pod: redis-
cart
replica

pod: redis-
cart 127.0.0.1:6379> replicaof redis-cart-cls1.default.svc.cluster.local 6379

# istioctl proxy-config endpoints --context $CTX_CLUSTER2 redis-cart-5b569cd47-6ppzm --cluster

"outbound|6379||redis-cart-cls1.default.svc.cluster.local"
ENDPOINT STATUS OUTLIER CHECK CLUSTER
52.11.49.96:15443 HEALTHY OK outbound|6379||redis-cart-cls1.default.svc.cluster.local

istioctl proxy-config listeners --context $CTX_CLUSTER2 redis-cart-5b569cd47-6ppzm --port 6379 -o json

. . .
"name": "172.17.43.144_6379",
. . .
"type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"statPrefix": "outbound|6379||redis-cart-cls1.default.svc.cluster.local",
"cluster": #CiscoLive
"outbound|6379||redis-cart-cls1.default.svc.cluster.local"
BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Example of redis-cart-cls1 service
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: redis-cart-cls1
spec:
type: ClusterIP
selector:
app: redis-cart
ports:
- name: tcp-redis
protocol: TCP
port: 6379
targetPort: 6379
EOF

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Cisco Calisti – A
Service Mesh
Manager
Cisco Calisti
https://calisti.app/
Operationalize the service mesh
Multi-cloud, multi-cluster connectivity and
observability
Connect any on-prem and public cloud together

Simplifies service mesh management

Single pane of glass, in depth metrics

Policy-based app networking & security

Policy management for DevOps teams
Introducing extensions
for Intersight Kubernetes Service

Traffic management ensures Complete application and Security at all layers between
smooth app updates health observability clusters and clouds

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
 Cisco Calisti Benefits
1 2 3
Multi-Cluster Observability Simplified mesh & traffic management Policy based n/w & Security

✓ Proactive issue resolution ✓ Complete Istio lifecycle mgmt. ✓ Simplified application

using SLO1, error budgeting, ✓ Ensure High Availability via deployment via security,
actionable alerting when automated tooling, metrics observability and platform
SLO’s are endangered ✓ Rich, comprehensive traffic management
✓ Faster root cause resolution operations focused dashboard ✓ Respond quickly to security
using timeline view, outlier ✓ Enterprise-grade security vulnerabilities via policy
detection, traffic hardening & lifecycle enforcement
tapping/tracing ✓ Avoid issues via canary
✓ Reduced risk of day 2
✓ Better visibility into service- deployments via canary deployments, circuit breakers
to-service performance upgrades ✓ DevOps friendly traffic
through Traffic Analytics debugging
✓ Reduce human error via config
validation
✓ VM-extensions for brownfield
and external service linkage
1Service Level Objectives
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Microservices Demo Topology
Cluster1 Cluster2

Leader Follower

https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Cisco Calisti Setup – From 19 steps to 3 ☺
https://calisti.app/
1) Install Cisco Calisti and identify the first cluster name:
# smm install -a --cluster-name smm-mc-1

2) Install Cisco Calisti with a full Istio control plane and attach the 2nd
cluster to the 1st cluster:
# smm istio cluster attach smm-mc-2.yaml --active-istio-control-plane

3) Enable Istio sidecar injection on a namespace:

# smm sidecar-proxy auto-inject on default

# smm istio cluster status

Clusters
---
Name Type Provider Regions Version Distribution Status Message
smm-mc-1 Local amazon [us-east-2] v1.21.2-13+d2965f0db10712 EKS Ready
smm-mc-2 Peer amazon [us-east-2] v1.21.2-13+d2965f0db10712 EKS Ready
ControlPlanes
---
Cluster Name Version Trust Domain Pods
Proxies
smm-mc-1 cp-v111x.istio-system 1.11.4 [cluster.local] [istiod-cp-v111x-75b7ccbb76-6szk9.istio-system] 32/32
smm-mc-2 cp-v111x.istio-system 1.11.4 [cluster.local] [istiod-cp-v111x-6f5d85c56f-vw2k7.istio-system] 5/5

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Demo
 K8s Multicluster Connectivity – Cisco Calisti
Multicluster
Networking Service
VPC peering, Hybrid Cloud, etc.

istio- istio- DNS CA istio- istio-

ingress meshexpansion ingress meshexpansion
Load Load Load Load Load
Balancer Balancer Balancer Balancer Balancer

Changes from
Ingress
earlier Istio
example: redis-cart
frontend
- New eastwest
K8s Cluster cartservice gateway name redis-cart K8s Cluster
- 2nd cluster’s smm-mc-2
smm-mc-2
redis-cart network name is
the same as the
redis-cart
smm-mc-1 cluster name
network1
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 Reference

2nd Warning: It is ALWAYS DNS that kills you ☺

Add to the Istio Operator Config
• By default, Istio does not enable DNS apiVersion: install.istio.io/v1alpha1

proxy for services that are exposed to kind: IstioOperator

spec:

another cluster meshConfig:

defaultConfig:
proxyMetadata:

https://istio.io/latest/docs/ops/configuratio
# Enable basic DNS proxying
• ISTIO_META_DNS_CAPTURE: "true"

n/traffic-management/dns-
proxy/#getting-started OR edit the config post-deployment
# kubectl edit istiocontrolplanes -n istio-system
meshConfig:
• Without enabling DNS proxy, “redis-cart- defaultConfig:
. . .<output_summarized>

cls1.default.svc.cluster.local” will not be proxyMetadata:

ISTIO_META_ALS_ENABLED: "true"

resolvable on the 2nd cluster ISTIO_META_DNS_CAPTURE: "true"

PROXY_CONFIG_XDS_AGENT: "true"

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
K8s Multicluster Connectivity – Cisco Calisti
Multicluster
• Brute force – Control which services are exposed/shared at the
gateway
# kubectl edit -n istio-system gw istio-cross-network-cp-v111x
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-cross-network-cp-v111x
spec:
servers:
- hosts:
- "*.local”

Service-specific Example:
hosts:
- ”<SVC>.default.svc.cluster.local”

• Istio Authorization Policy:

https://istio.io/latest/docs/reference/config/security/authorization-
policy/
• Microscopic control of which things talk to which other things and how
#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Cisco Calisti Multicluster – Redis Replication
Service Mirror – Phantom/Ghost Services
istio-
meshexpansion

Load
Balancer AWS LB: 52.14.79.204

10.100.55.204
redis-cart- service: redis-cart
cls1
service:
redis-cart
pod: redis-
cart
replica

pod: redis-
cart 127.0.0.1:6379> replicaof redis-cart-cls1.default.svc.cluster.local 6379

# istioctl proxy-config endpoints redis-cart-5b569cd47-brxgr --cluster "outbound|6379||redis-cart-

cls1.default.svc.cluster.local”
ENDPOINT STATUS OUTLIER CHECK CLUSTER
52.14.79.204:15443 HEALTHY OK outbound|6379||redis-cart-cls1.default.svc.cluster.local

# istioctl proxy-config listeners redis-cart-5b569cd47-brxgr --port 6379 -o json

. . .
"name": ”10.100.55.204_6379",
. . .
"type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"statPrefix": "outbound|6379||redis-cart-cls1.default.svc.cluster.local",
"cluster": #CiscoLive
"outbound|6379||redis-cart-cls1.default.svc.cluster.local"
BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
 Reference
Cisco Calisti – Multicluster – Multi-Control Plane

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Cisco Calisti - Topology Reference

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Summary
• Check out Cisco Calisti – Get started for free (Up to 10 nodes and 2
clusters): https://calisti.app/
• There are many options for connecting workloads in multiple Kubernetes
clusters – we just touched on a few
• Network Service Mesh - https://networkservicemesh.io/
• Many users leverage multicluster connectivity for cross-cluster load-
balancing of services
• For specialized per-service cross-cluster connectivity, special care must
be taken to select a solution that provides a balance of use-case flexibility
and operational supportability

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Technical Session Surveys
• Attendees who fill out a minimum of four
session surveys and the overall event
survey will get Cisco Live branded socks!

• Attendees will also earn 100 points

in the Cisco Live Game for every
survey completed.

• These points help you get on the

leaderboard and increase your chances
of winning daily and grand prizes.

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Pay for Learning with
Cisco Learning Credits
Cisco Learning and Certifications (CLCs) are prepaid training
vouchers redeemed directly
From technology training and team development to Cisco certifications and learning with Cisco.
plans, let us help you empower your business and career. www.cisco.com/go/certs

Learn Train Certify

Cisco U. Cisco Training Bootcamps Cisco Certifications and
IT learning hub that guides teams Intensive team & individual automation Specialist Certifications
and learners toward their goals and technology training programs Award-winning certification
program empowers students
Cisco Digital Learning Cisco Learning Partner Program and IT Professionals to advance
Subscription-based product, technology, Authorized training partners supporting their technical careers
and certification training Cisco technology and career certifications
Cisco Guided Study Groups
Cisco Modeling Labs Cisco Instructor-led and 180-day certification prep program
Network simulation platform for design, Virtual Instructor-led training with learning and support
testing, and troubleshooting Accelerated curriculum of product,
technology, and certification courses Cisco Continuing
Cisco Learning Network Education Program
Resource community portal for Recertification training options
certifications and learning for Cisco certified individuals

Here at the event? Visit us at The Learning and Certifications lounge at the World of Solutions

#CiscoLive BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 • Visit the Cisco Showcase
for related demos

• Book your one-on-one

Meet the Engineer meeting

• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs

your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

BRKETI-2003 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Thank you

#CiscoLive
#CiscoLive