CONFIDENTIAL designator

Adapt this presentation

according to the maturity
of you audience

OpenShift
Workshop OpenShift Service
Mesh

Vodafone Oman

Shah Zobair
Principal Consultant
March 02, 2022

V0000000
 CONFIDENTIAL designator

Contacts Red Hat

Role Mail

Zeeshan Shaikh Project Manager [email protected]

Shah Zobair Architect [email protected]

2
V0000000
 Agenda CONFIDENTIAL designator

What we’ll discuss

● Basic Concepts ● Basic configuration
● Istio ● Traffic Management
● OpenShift Service Mesh ● Security
● Jaeger ● Observability
● Kiali ● Advanced Configuration
● Prometheus ● Limitations
● Grafana
● Installation
● Hardware requirements
● Configuration prerequisites

3
V0000000
 CONFIDENTIAL designator

Basic Concepts

4
V0000000
 CONFIDENTIAL designator

Multi-Tiers Applications - Monolithic layout (1/2)

In early days we used to have multi tiers applications typically 3-tiers applications:

● Frontend server.
● Business logic layer: a very large monolithic app.
● Backend: can be a database, a filesystem, or an external API.

5
V0000000
 CONFIDENTIAL designator

Multi-Tiers Applications - Monolithic layout (2/2)

User Firewall Frontend Business Backend

Layer
When user made a request:

● It landed in the firewall, next to the frontend

server.
● Then to the business logic and finally the
backend.
Firewall External ● Accordingly a response came back to the user
API
following the reverse path.

6
V0000000
 CONFIDENTIAL designator

Multi-Tiers Applications - Scaling and resiliency problem (1/2)

● As more and more users start to come in, it would create a challenge.
● You could add more front end server behind a load balancer.

7
V0000000
 CONFIDENTIAL designator

Multi-Tiers Applications - Scaling and resiliency problem (2/2)

Frontend 1

User Firewall Business Backend

Layer
Frontend 2

● What happen when backend or external

service become unavailable?
● That would lead a huge backlog of requests
Firewall External
getting queued at business logic layer. API
● The load average of the business logic server
starts shooting up.
● The responses to the request would start
slowing down, eventually becoming
unavailable.
8
V0000000
 CONFIDENTIAL designator

Microservices Approach
Microservices are an architectural approach to building application that consists of distributed and loosely coupled services,
in a way that one team’s changes won’t break the entire app.

9
V0000000
 CONFIDENTIAL designator

Containerization: A common way to implement microservices

● Application are isolated.

● We can control resources each application could use.
● We can orchestrate them using Kubernetes platform
technology.
● We can scale them out or scale them as the need Firewall
may be.

Users

Firewall

External External
10 API 1 API 2
V0000000
 CONFIDENTIAL designator

Containerization introduces new challenges

What happened at scale when hundred or thousand microservice are running inside the container environment?

In case of troubleshooting, we have to trace individually requests that go the involved microservices.

If an external service is unavailable how to handle that?

11
V0000000
 CONFIDENTIAL designator

Microservices Fallacies

Service Service Service - Network is reliable

- Latency is zero
- Bandwidth is infinite
- Network is secure

Service Service Service

- Topology does not change
- There is one administrator
- Transport cost is zero
- Network is homogeneous

Service Service Service

12
V0000000
 CONFIDENTIAL designator

Supporting Services for Distributed Applications without

Service Mesh
To address the challenges a set of supporting services must be added to your code

Service
- Configuration
- Service Discovery
Tracing

Circuit Breaker
- Dynamic Routing
Routing

Svc Discovery
- Resilience
Config - Observability

Platform

13
V0000000
 CONFIDENTIAL designator

What is a service mesh?

A service mesh is a dedicated network for service-to-service communications.

When using a service mesh, the supporting services for distributed applications are provided via the
container platform and service mesh.

Service

Tracing

Circuit Breaker

Routing

Svc Discovery
Service
Config

Container Platform
Platform
(+ Service Mesh)

...2014 2018
14
V0000000
 Microservice architecture with/without a service mesh CONFIDENTIAL designator

15
V0000000
 CONFIDENTIAL designator

Why we need a service mesh?

Resilience for users

● Availability -> The application should be available.

● Performance -> The application should perform within a some defined parameters (ex: response time).

● Security -> Access to the data must be secure and prevented from any unattended malicious attack.

● Graceful error handling -> If external service are not available, how to handle properly without stucking the application.

16
V0000000
 CONFIDENTIAL designator

Why we need a service mesh?

Service Mesh for DevSecOps

● Auto discovery.

● Monitoring, tracing, RCA, observability.

● Testing: unit, integration, regression, chaos.

● Deployment: automated, A/B, canary, mirror.

● Security: who can access what.

17
V0000000
 CONFIDENTIAL designator

Istio

18
V0000000
 CONFIDENTIAL designator

Istio Project
Istio makes it easy to create a network of deployed services with load balancing, service-to-service
authentication, monitoring, and more, without any changes in the service code.

19
V0000000
 CONFIDENTIAL designator

Istio Architecture

The data plane is composed

of a set of intelligent proxies
Data Plane
(Envoy) deployed as
sidecars. These proxies
mediate and control all
network communication
between microservices.
They also collect and report
telemetry on all mesh
traffic.

Control Plane

The control plane manages

and configures the proxies
to route traffic.

20
V0000000
 CONFIDENTIAL designator

Istio Architecture - Envoy

istiod

Envoy is a high-performance proxy that

ENVOY ENVOY ENVOY
intercepts all inbound and outbound traffic for
all services in the service mesh.

SERVICE SERVICE SERVICE

POD POD POD

21
V0000000
 CONFIDENTIAL designator

Istio Architecture - istiod

istiod istiod provides service discovery,

configuration and certificate
management.

ENVOY ENVOY ENVOY

SERVICE SERVICE SERVICE

POD POD POD

22
V0000000
 CONFIDENTIAL designator

OpenShift
Service Mesh

23
V0000000
 CONFIDENTIAL designator

Red Hat OpenShift Service Mesh

Based on the open source Istio project, Red Hat OpenShift Service Mesh provides a platform for
behavioral insight and operational control over your networked microservices in a service mesh.

Service Service Service Service Service ANY

CONTAINER CONTAINER CONTAINER CONTAINER CONTAINER APPLICATION

OpenShift Service Mesh

(Istio + Jaeger + Kiali)

OpenShift Container Platform

(Enterprise Kubernetes)

ANY
INFRASTRUCTURE
Laptop Datacenter OpenStack Amazon Web Services Microsoft Azure Google Cloud

24
V0000000
 CONFIDENTIAL designator
OpenShift Service Mesh

OpenShift Service Mesh vs Istio

Additions for Red Hat’s Enterprise & Public Sector Customers

Multi-Tenant Security & Compliance Management, Monitoring & OpenShift

Architecture Focus Observability Integrations
Multiple meshes securely Control Plane and Data Plane Pre-configured Kiali, Jaeger Integrations with OpenShift
deployed within the same components execute with and Grafana for simplified components such as
cluster, with each mesh standard privileges. management, monitoring and OperatorHub, OpenShift
isolated and managed OpenSSL for FIPS observability. Routes and 3Scale API
independently. compliance. Management.

V0000000
 Multi-Tenancy - Mesh and non mesh (1/2) CONFIDENTIAL designator

Upstream Istio takes a single tenant approach.

Red Hat OpenShift Service Mesh supports multiple independent control planes within the cluster.

Non Mesh Service Mesh 1 Service Mesh n

Control Plane 1 Control Plane n

openshift-...
Namespace Namespace

kube-... Data Plane 1 Data Plane n

Namespace A
Non mesh managed Namespace C
namespace N...

Namespace z
Namespace x

- A namespace cannot be part of more than one data plane.

- A control plane cannot manage more than one data plane.
- All others namespaces in non mesh scope are managed using regular OpenShift Administration capabilities.
- Communication between meshes involves configuring one or more Gateways, as you would for accessing
external services. V0000000

26
 Multi-Tenancy - Mesh and non mesh (2/2) CONFIDENTIAL designator

Service Mesh n - A mesh is composed of one control plane and one data plane. Several mesh
Control Plane n
Namespace
can coexist inside the cluster.

Data Plane 2 - A control plane is implemented in a dedicated namespace called root

Namespace C namespace.

- A data plane is comprised of one or more namespaces that run your

Namespace z

applications you want to manage.

V0000000

27
 CONFIDENTIAL designator

Multi-Tenancy - Implementation

Multi-tenancy in OpenShift is implemented by the declaration of two specific CRDs.

ServiceMeshControlPlane - Defines the Service Mesh Control Plane

ServiceMeshMemberRoll - Defines list of namespaces member of the data plane

28
V0000000
 Multi-Tenancy - Network Policies (1/2)
CONFIDENTIAL designator

● OpenShift Service Mesh creates NetworkPolicy resources in control plane namespace and each namespaces member
of data plane.
● If you remove a namespace member from data plane, this NetworkPolicy resource is deleted from the project.

This also restricts ingress to only member projects.

If ingress from non-member projects is required, you need to create a NetworkPolicy to allow that traffic through.

29
V0000000
 Multi-Tenancy - Network Policies (2/2)
CONFIDENTIAL designator

When creating a new dataplane, each namespace belonging will have two Network Policies defined by default.

istio-mesh-basic: This policy will force all your pods to communicate only with pods included data plane namespace(s).
Data plane namespaces are identified with the namespace label maistra.io/member-of=<control plane namespace>

istio-expose-route: This policy allows Ingress Controller route traffic to be exposed for pod that will have explicit label
maistra.io/expose-route=”true”.

30
V0000000
 CONFIDENTIAL designator

Jaeger

31
V0000000
 Jaeger CONFIDENTIAL designator

Jaeger lets you perform tracing to monitor and troubleshoot transactions in complex distributed systems.

Based on your programming environment you need to

configure your application to use the library provided by the
1
Jaeger client, collect tracing data and push them to Jaeger
agent
6
Jaeger-agent in Red Hat OpenShift Service Mesh is an Envoy
2 proxy that collect Jaeger client data and push them to Jaeger
Collector.
4
3
5 3 Jaeger collector is part of the control plane.
1
All the collected data are stored in a backend storage.
4 In Red Hat OpenShift Service Mesh, use ElasticSearch as the
backend for Jaeger collected data.
2
5 A query service to query the data according various criteria.

6 A GUI to visualize various trace and spans.

A Jaeger instance is deployed in the control plane namespace.

32
V0000000
 CONFIDENTIAL designator

Kiali

33
V0000000
 CONFIDENTIAL designator

Kiali (1/2)
Kiali provides observability for your service mesh. By using Kiali you can view configurations, monitor traffic, and view and
analyze traces in a single console.

- OpenShift Service Mesh console

- Visualizes service mesh topology in real time
- Provides visibility into features like request routing, circuit breakers, request
rates, latency, etc.
- Inline edition of YAML representation of Istio resources, with powerful
semantic validation
- Actions to create, update, delete Istio configuration resources, driven by
wizards
- Custom metrics dashboards
- Integrated with distributed tracing

34
V0000000
 CONFIDENTIAL designator

Kiali (2/2)

A Kiali instance is deployed in the control plane namespace.

35
V0000000
 CONFIDENTIAL designator

Prometheus

36
V0000000
 CONFIDENTIAL designator

Prometheus

- Prometheus is an open source monitoring system and

time series database.
- You can use Prometheus with Istio to record metrics that
track the health of Istio and of applications within the
service mesh.
- You can visualize metrics using tools like Grafana and Kiali.
- The recommended approach for production-scale
monitoring of Istio meshes with Prometheus is to use
hierarchical federation in combination with a collection of
recording rules.

A Prometheus instance is deployed in the control plane namespace.

37
V0000000
 CONFIDENTIAL designator

Installation

38
V0000000
 CONFIDENTIAL designator

Installation Prerequisites

To install the Red Hat OpenShift Service Mesh Operator, you must first install these Operators:

Based on the open source Elasticsearch project that enables you to configure and
Elasticsearch
manage an Elasticsearch cluster for tracing and logging with Jaeger.

Based on the open source Jaeger project, lets you perform tracing to monitor and
Jaeger
troubleshoot transactions in complex distributed systems.

Based on the open source Kiali project, provides observability for your service mesh.
Kiali By using Kiali you can view configurations, monitor traffic, and view and analyze traces
in a single console.

Red Hat OpenShift Service Mesh does not support Istio installation profiles.
Red Hat OpenShift Service Mesh does not support canary upgrades of the service mesh.
39
V0000000
 Service Mesh Installation - Elasticsearch
CONFIDENTIAL designator

OpenShift Elasticsearch Operator is installed in the openshift-operators-redhat namespace.

V0000000
 CONFIDENTIAL designator

Service Mesh Installation - Jaeger

Jaeger Operator is installed in the openshift-operators namespace.

41
V0000000
 CONFIDENTIAL designator

Service Mesh Installation - Kiali

Kiali Operator is installed in the openshift-operators namespace.

42
V0000000
 CONFIDENTIAL designator

Service Mesh Installation - Service Mesh Operator

OpenShift Service Mesh Operator is installed in the openshift-operators namespace.

43
V0000000
 CONFIDENTIAL designator

Hardware
requirements

44
V0000000
 Control Plane resources
CONFIDENTIAL designator

Baseline: A mesh of 1000 services and 2000 sidecars.

Component CPU RAM

Istiod 1 1.5 GB

Ingress Gateway 0.5 50 MB

Egress Gateway 0.5 50 MB

Kiali 0.01 64 MB

Jaeger (All-in-one) 0.02 256 MB

Jaeger Query (Production) 0.03 384 MB

Jaeger Collector (Production) 0.01 128 MB

Jaeger ElasticSearch (Production) 3x1 3 x 16 GB

Prometheus 0.52 306 MB

Grafana 0.02 256 MB

45
V0000000
 Data Plane resources
CONFIDENTIAL designator

Baseline: A mesh of 1000 services and 2000 sidecars.

Component CPU RAM

Envoy 0.5 x 1000 (500) 50 MB x 1000 (50GB)

Additional Ingress Gateway 0.5 50 MB

Additional Egress Gateway 0.5 50 MB

46
V0000000
*The Envoy proxy adds 3.12 ms to the 90th percentile latency.
 Sizing consideration (1/3)
CONFIDENTIAL designator

Istiod’s CPU and memory requirements scale with the number of configurations and possible system states.

The CPU consumption scales with the following factors:

- The rate of deployment changes.

- The rate of configuration changes.

- The number of proxies connecting to Istiod.

47
V0000000
 Sizing consideration (2/3)
CONFIDENTIAL designator

Data plane performance depends on many factors, for example:

- Number of client connections.

- Target request rate.

- Request size and Response size.

- Number of proxy worker threads.

- Protocol.

- CPU cores.

- Number and types of proxy filters, specifically Envoy filter.

48
V0000000
 Sizing consideration (3/3)
CONFIDENTIAL designator

For Jaeger factors which need to be accounted for include:

- Sampling rate.

- Number of requests.

- How many services.

- Number of spans per service.

- Number of tags/log events per span.

- Size of the data values for those tags/log events.

49
V0000000
 CONFIDENTIAL designator

Configuration
prerequisites

50
V0000000
 Ingress Controller for Network policies
CONFIDENTIAL designator

Depending the infrastructure platform and Ingress Controller setting during cluster installation, you have to configure
the Ingress network traffic support for Network policies

First determinate the Ingress Controller endpointPublishingStrategy type.

Unless explicitly configured during cluster installation endpointPublishingStrategy type is set with the following value:
● LoadBalancerService (AWS / Azure / GCP / IBM Cloud)
● NodePortService (Bare metal)
● HostNetwork (other including vSphere)
$ oc get ingresscontroller default -n openshift-ingress-operator -o
jsonpath='{.status.endpointPublishingStrategy.type}{"\n"}'
HostNetwork

If endpointPublishingStrategy is set to HostNetwork, then the Ingress Controller pod runs on the host network. The
traffic is assigned the netid:0 Virtual Network ID (VNID).
Because Ingress Controller namespace has a different VNID, the Network Policy cannot match the default Ingress
controller.
You must label the default namespace (which have VNID=0) with network.openshift.io/policy-group: ingress to
allow traffic from Ingress

$ oc label ns default network.openshift.io/policy-group=ingress

51
V0000000
 Workload version labels CONFIDENTIAL designator

To correctly detect your workload in Kiali, you must have both version and app labels on the pod template of your workload
resource (Deployment, Deploymentconfig, Statefulset, etc).

Apply both version and app labels on:

- pod template
- pod selector
- workload metadata section

52
V0000000
 Services port labels CONFIDENTIAL designator

Although Istio can detect protocol used by services, sometime it could fail to find the protocol which prevents any
communication to the services.

It is crucial to give the protocol

name explicitly on every port
label of your services comprised
in your data plane:
- http
- https
- tcp
- ...
This way Istio will be able to route
the traffic

53
V0000000
 CONFIDENTIAL designator

Basic
Configuration

54
V0000000
 CONFIDENTIAL designator

Service Mesh Control Plane (1/2)

You need to create an empty namespace/project before creating the control plan.

The ServiceMeshControlPlane resource defines the configuration to be used during installation.

You can deploy the default configuration provided by Red Hat or customize the ServiceMeshControlPlane file to fit
your business needs.

55
V0000000
 CONFIDENTIAL designator

Service Mesh Control Plane (2/2)

Create the Service Mesh Control Plane. The customization of ServiceMeshControlPlane resource will be covered in
detail later.

56
V0000000
 CONFIDENTIAL designator

Service Mesh Member Roll (1/2)

The ServiceMeshMemberRoll lists the projects belonging to the control plane.
You must create a ServiceMeshMemberRoll resource named default in the same project as the
ServiceMeshControlPlane.

57
V0000000
 CONFIDENTIAL designator

Service Mesh Member Roll (2/2)

Create the Service Mesh Member Roll and all the projects belonging to the control plane.

58
V0000000
 Workloads support in Kiali CONFIDENTIAL designator

By default, Kiali will exclude the following workloads: DeploymentConfig, ReplicationController, Statefulsets, CronJob, Job

Configure kubernetes configuration entry in kiali CR in control plane

namespace to change the workloads exclusion list so Kiali will recognize the
desired workload types to monitor.

59
V0000000
 Sidecar Injection (1/3)
CONFIDENTIAL designator

The upstream Istio community installation

automatically injects the sidecar into pods within the projects you have labeled.

60
V0000000
 Sidecar Injection (2/3)
CONFIDENTIAL designator

Red Hat OpenShift Service Mesh does not automatically inject the sidecar to any pods,
but requires you to opt in to injection using an annotation without labeling projects.

This method requires fewer privileges and does not conflict with other OpenShift capabilities such as builder pods.

To enable automatic injection you specify the sidecar.istio.io/inject annotation in the pod template spec.

61
V0000000
 Sidecar Injection (3/3)
CONFIDENTIAL designator

If you don’t inject Envoy proxy sidecar container, your pod will not be meshed. It means it will not be managed by the
control plane.

Traffic to / from this non meshed pod will still be controlled by namespace network policies.

62
V0000000
 CONFIDENTIAL designator

Traffic
Management

63
V0000000
 CONFIDENTIAL designator

Traffic Management

Traffic management decouples traffic flow and infrastructure scaling. This flexibility allows you to use Istio to specify
which rules to apply for traffic management between pods. Envoy manage which pods receive traffic.

makes call to

v1
c
affi
f tr
95%o

frontend-app backend-app (Java)

MySQL Database
5%
of t
raffi
c
app.example.com
makes call to

v2

frontend-app backend-app (Go)

64
V0000000
 Gateway (1/2)
CONFIDENTIAL designator

A Gateway CR will allow to receive incoming HTTP/TCP connections into the data plane.

You must a configure a Gateway to allow external traffic in your data plane.

Traffic must use the ingressgateway service in the

control plane.
Hosts will define the FQDN routes from which the
Gateway will receive the traffic.
Configure the ports and protocols lists the gateway will
listen. Note this should be only ports managed by
Ingress Controller routers.

65
V0000000
 Gateway (2/2)
CONFIDENTIAL designator

In control plane namespace, a Route will be created for every configured hosts in the YAML

Note you will also have to configure at least a VirtualService CR to route the traffic from the gateway to the desired
services in your data plane.

66
V0000000
 Virtual Services CONFIDENTIAL designator

A VirtualService defines a set of traffic routing rules to apply when a host is addressed.
Each routing rule defines matching criteria for traffic of a specific protocol.
If the traffic is matched, then it is sent to a named destination service (or subset/version of it) defined in the registry.

Mention the gateway(s) from where the traffic comes. If the

traffic does not come from a gateway, leave this empty

You can narrow the hosts to filter the traffic for this Virtual
Services. Hosts can be:
- Services (if traffic come from data plane services)
- Routes FQDN (if traffic come from gateway(s))

Routing rules based on some criteria:

● URL matches or starts a URI
● URL contains some headers
● Etc.
Destination host is a service with its port number

67
V0000000
 CONFIDENTIAL designator

DestinationRule
Virtual services route traffic to a destination. Destination rules configure what happens to traffic at that destination.

Load Balancing algorithm inside each subsets defined

below:
- ROUND_ROBIN: Default configuration, each
instance gets a request in turn
- RANDOM: Requests are forwarded at random to
instances in the pool.
- WEIGHTED: Requests are forwarded to instances in
the pool according to a specific percentage.
- LEAST_REQUESTS: Requests are forwarded to
instances with the least number of requests.

List of subsets.
A subset is a service endpoint for pods with a unique label
to distinguish it from others endpoints of the same
68 service.
Load Balancing can be overridden for a particular
V0000000
subset.
 ServiceEntry - Hardening the egress traffic
CONFIDENTIAL designator

By default, the Istio proxy lets calls to unknown services pass through the egress gateway.
If you need to control access to external services you have to configure proxy.networking.trafficControl.outbound.policy
in ServiceMeshControlPlane CR.

ALLOW_ANY: All call to unknown services are pass through

REGISTRY_ONLY : Only allow calls to services whose service entry is
defined within the mesh

You have to create ServiceEntry CRs to lets the traffic pass when hardening to registered service only.

69
V0000000
 CONFIDENTIAL designator

ServiceEntry - Define external services

A ServiceEntry CR adds an entry to the service registry that Red Hat OpenShift Service Mesh maintains internally.
After you add the service entry, the Envoy proxies can send traffic to the service as if it was a service in your mesh.

Add the external host ext-svc.example.com to your

OpenShift Service Mesh

You can configure virtual services and destination rules to control traffic to a service entry in the same way you
configure traffic for any other service in the mesh.
70
V0000000
 CONFIDENTIAL designator

ServiceEntry - Unmanaged outbound traffic

When outgoing traffic is not managed by a ServiceEntry Kiali displays this traffic detected by the Envoy proxies.

Unmanaged traffic is blocked if REGISTRY_ONLY policy

hardening was set. Outgoing traffic attempts are
displayed going to a BlackHoleCluster icon.

Unmanaged traffic passed if ALLOW_ANY policy

hardening was set. Outgoing traffic attempts are
displayed going to a PassthroughCluster icon.

71
V0000000
 Egress Gateway CONFIDENTIAL designator

By default egress traffic exit directly from the istio proxy of the workload calling the external service.
To enforce the egress traffic to exit from the egress gateway, you have to configure the appropriate resources for the egress
gateway.

Gateway CR apply to
Egress Gateway

Both Gateway and VirtualService have external host

configured.

VirtualService will route the

outgoing traffic from mesh to
the egress gateway service then
the traffic coming to the egress
gateway will be routed directly to
the external service.

72
V0000000
Define a subset for each external host (or external hosts group) in the
DestinationRule applying to the egress gateway service.
 CONFIDENTIAL designator

Security

73
V0000000
 Istio Security CONFIDENTIAL designator

Security in Istio involves multiple components:

- A Certificate Authority (CA) for key and

certificate management
- A configuration API server to distribute
to the proxies:
- Authentication policies
- Authorization policies
- Secure naming information
- Sidecar and perimeter proxies as Policy
Enforcement Points (PEPs) to secure
communication between clients and
servers
- A set of Envoy proxy extensions to
manage telemetry and auditing
74
V0000000
 Identity and Certificate management CONFIDENTIAL designator

Istiod offers a gRPC service to take certificate signing requests (CSRs)

1 A developer deploys two new pods with separate

backend and frontend service accounts.

2 OpenShift Service Mesh injects the Envoy proxy

sidecar container into each pod.

3 The Envoy proxy requests a new certificate-key

pair by using SDS

4
Istiod uses the internal certificate authority to sign
and return the identity.

Envoy proxy uses the certificates for security

5
features, such as mutual TLS.

Secret Discovery Service (SDS) enables Service Mesh to

rotate certificates at runtime without restarting the pods.
75
V0000000
Authentication - Auto mutual TLS (1/2) CONFIDENTIAL designator

By default, Istio tracks the server workloads migrated to Istio proxies, and configures client proxies to send mutual TLS
traffic to those workloads automatically, and to send plain text traffic to workloads without sidecars.

1 The application sends plain text traffic.

2 The Envoy proxy intercepts the traffic.

3 The origin Envoy proxy encrypts the traffic.

4 The destination Envoy proxy intercepts TLS traffic.

The destination Envoy proxy decrypts TLS traffic

5
into plain text traffic.

6 The destination service receives and responds via

plain text traffic.

Thus, all traffic between workloads with proxies uses mutual TLS, without you doing anything.

V0000000
 Authentication - Auto mutual TLS (2/2) CONFIDENTIAL designator

While Istio automatically upgrades all traffic between the proxies and the workloads to mutual
TLS between, workloads can still receive plain text traffic.

If this behavior is not desired, you have to configure

spec.security.dataPlane.mtls setting on ServiceMeshControlPlane CR
to enforce mutual TLS mode.

Secure connections are always used when proxies communicate with the
control plane regardless of spec.security.controlPlane.mtls setting.

This setting is only used when configuring connections for Legacy and deprecated Mixer telemetry or
policy on Service Mesh 1.x.

77
V0000000
 PeerAuthentication - Mesh Level (1/2) CONFIDENTIAL designator

Out-of-the box when creating the control plane, 3 PeerAuthentication CR are defined in the control plane namespace.

The global policy for the mesh

Policy to relax mTLS for Jaeger collector
Policy to relax mTLS for Grafana

These PeerAuthentication CR are controlled by the ServiceMeshControlPlane resource, you are not supposed to alter
them directly.

78
V0000000
 PeerAuthentication - Mesh Level (2/2) CONFIDENTIAL designator

PeerAuthentication CR stored in control plane namespace affects the entire mesh.

When Data plane Security is disabled, mTLS mode is set to PERMISSIVE to

allow both secure and plain text traffic in the mesh

When Data plane Security is enabled, mTLS mode is set to STRICT to allow
only secure traffic in the mesh

79
V0000000
 PeerAuthentication - Namespace & Workload Level (1/2) CONFIDENTIAL designator

If you want to customize for a particular workload in a namespace in your mesh,

you configure a PeerAuthentication CR stored in the target namespace with
the selector for your workload.

A corresponding DestinationRule is required to allow traffic if the workload

policy differs from the namespace/global policy.

In this example policy is set to STRICT for a workload which differs from
PERMISSIVE namespace policy level so we must set the Destination to
ISTIO_MUTUAL to allow only secure traffic.

80
V0000000
 PeerAuthentication - Namespace & Workload Level (2/2) CONFIDENTIAL designator

When a service uses an unknown protocol, the Envoy proxy treats the traffic as an opaque TCP connection and
wraps the connection for mTLS.

In some cases, you might need to disable specific ports from mTLS wrapping.

Disabling mTLS wrapping can be useful, for example, when:

- A protocol already uses some form of encryption.

- An application is location aware in terms of routing and cannot use Envoy

proxy routing.

81
V0000000
Security display in Kiali (1/2) CONFIDENTIAL designator

At the right side, Kiali shows a lock when the mesh has strictly enabled mTLS for the whole service mesh.

All the communications in the mesh Either the mesh is configured in

uses mTLS. PERMISSIVE mode or there is a
misconfiguration in the
mesh-wide mTLS configuration

V0000000
Security display in Kiali (2/2) CONFIDENTIAL designator

A lock when strict mTLS is enabled or a When an edge shows a lock icon it means at least
hollow lock when permissive one request with mTLS enabled is present.

Edges that use only mTLS won’t show any lock.

Kiali shows also the mTLS status at

In case the mesh-wide mTLS is enabled, the graph
namespace-wide. will show broken locks for only those edges
containing non-mTLS requests.
V0000000
 CONFIDENTIAL designator

Transport authentication - Ingress Gateways - Legacy method (1/2)

The Istio ingress gateway pod reads the ingress gateway certificates from the secret istio-ingressgateway-certs in
control plane namespace.
Create this secret in control plane namespace with the certificates and restart the istio-ingressgateway pod.

$ oc create secret tls istio-ingressgateway-certs --cert tls.crt --key tls.key -n istio-system

By default, the volumes are automatically added to the

x deployment by the Service Mesh Operator.
You don’t have to alter the istio-ingressgateway
Deployment!
84
V0000000
 CONFIDENTIAL designator

Transport authentication - Ingress Gateways - Legacy method (2/2)

Configure the TLS setting with certificates mounted by the secret istio-ingressgateway-certs.

This gateway configuration lets HTTPS traffic

from hostname.example.com into the mesh on
port 443

85
V0000000
 Transport authentication - Ingress Gateways - SDS method CONFIDENTIAL designator

You can configure a TLS ingress gateway to fetch credentials from the ingress gateway agent via secret discovery service
(SDS).
The ingress gateway agent runs in the same pod as the ingress gateway and watches the credentials created in the same
namespace as the ingress gateway.

Enabling SDS at ingress gateways brings the following

benefits:

● The ingress gateway can dynamically add, delete, or

update its key/certificate pairs and its root certificate.
You do not have to restart the ingress gateway
(contrary to the legacy method).
TLS Secret for hosts
httpbin.example.com
● No secret volume mount is needed. Once you
create a kubernetes secret, that secret is captured by
the gateway agent and sent to ingress gateway as
key/certificate or root certificate.
TLS Secret for hosts
● The gateway agent can watch multiple helloworld-v1.example.com
key/certificate pairs. You only need to create
secrets for multiple hosts and update the gateway
86 definitions.
V0000000
 Authorization CONFIDENTIAL designator

AuthorizationPolicy CR enables access control on workloads in the mesh. It supports both allow and deny policies.

ALLOW : Allow a request only if it matches the rules

(default)
DENY: Deny a request if it matches any of the rules.

Specify the source of the request. Can be

namespace, a service account, etc.

Operations of the request

Additional conditions of a
87
request V0000000
Authorization - Istio native authorization with JWT CONFIDENTIAL designator

PROS:
● Natively supported by Istio
● Simple to set up (2 CRDs: RequestAuthentication
and AuthorizationPolicy)
● Fine-grained policy based on JWT token
attributes

CONS:
● Token retrieval and embedding left to the user

CODE:
servicemesh-2-rhsso-examples - approach 1

V0000000
Authorization - OIDC workflow using oauth2-proxy CONFIDENTIAL designator

PROS:
● Authentication enforced at the ingress gateway
level
● Automated OIDC workflow to authenticate the
user

CONS:
● Coarse-grained authorization (authenticated ==
authorized)
● Complex, non-native setup (involves ressources
patches)

CODE:
servicemesh-2-rhsso-examples - approach 2

V0000000
Authorization - Combining OIDC workflow with JWT CONFIDENTIAL designator

PROS:
● Authentication enforced at the ingress gateway
level
● Automated OIDC workflow to authenticate the
user
● Can be combined with fine-grained JWT-based
authorization policies

CONS:
● Complex, non-native setup (involves ressources
patches)

CODE:
servicemesh-2-rhsso-examples - approach 3

V0000000
 CONFIDENTIAL designator

Observability

91
V0000000
 CONFIDENTIAL designator

Observability

A System is observable if current state can be understood from outside.

Observability is an important characteristic of cloud-native distributed systems that helps you understand, operate, maintain
and evolve the system

92
V0000000
 CONFIDENTIAL designator

Observability
There are 3 critical pieces that the Operation team should be aware of:

● Distributed tracing: What happens to each request as it comes in, traversing a set of microservices and goes back as a
response?
○ Which service is causing latency?
○ How much time is take to process each service?
■ Identify the bottleneck and improve the architecture.
● Metrics:
○ How many hits the application has received?
■ Based on metrics we can do capacity planning.
● Visualization:
○ We should be able to visualize graphically, how service are interconnected.

93
V0000000
 CONFIDENTIAL designator

Red Hat Service Mesh observability components

Trace the path of a request as it travels across a complex system, discover the latency
Distributed Tracing
Jaeger
of the components along that path, and know which component in the path is creating
a bottleneck.

Monitoring All service-to-service communication goes through Envoy proxies, and the service
Prometheus mesh control plane is able to gather logs and metrics from these proxies.

Helps you define, validate, and observe the connections and microservices of the
Visualization
Kiali
service mesh. It visualizes the service mesh topology and provides visibility into
features such as request routing, circuit breakers, request rates, latency and more.

94
V0000000
 Distributed Tracing - Spans & Trace concepts (1/2)
CONFIDENTIAL designator

Represents an individual unit of work done in a distributed system which consists of a

Spans
named, timed operation representing a piece of the workflow.

A visualization of the life of a request as it moves through a distributed system

Trace
consisting of multiple related spans assembled together

95
V0000000
 Distributed Tracing - Spans & Trace concepts (2/2)
CONFIDENTIAL designator

To be able to effectively trace the ongoing of a request we need two primary things:
● ubiquitousness of the request (availability of tracing mechanism at each microservice)
● what rate sampling the request (tracing all request would put pressure on the resources VS losing request context)

The trace begins when the user sends the Total time of the parent span. Parent span supports a
request and ends when the user receives bunch of child spans.
the response.
Within the request there are spans. A
span denotes an amount of time taken to
complete a particular request.

The first span starts when the request

reaches the frontend service.
This span ends when the request
response leaves the frontend service.

The complexity arises as how to

propagate header information from one
span to another?

96
Each span has a unique id and track its direct parent id (pid)
V0000000
 Distributed Tracing - Trace Context Propagation (1/2)
CONFIDENTIAL designator

Red Hat OpenShift Service Mesh uses a standard set of HTTP headers for trace context propagation.

The Envoy proxy sidecar tracks these headers and forwards them to Jaeger for storage and analysis.

A service needs to collect and propagate the following headers from incoming request to any outgoing requests:
● x-request-id
● x-b3-traceid
● x-b3-spanid
● x-b3-parentspanid
● x-b3-sampled
● x-b3-flags
● x-ot-span-context

All the services have these identifiers injected which help us to know what is part of the request.

You must explicitly enable tracing in your applications to generate traces and spans and to propagate context
information

97
V0000000
 Distributed Tracing - Trace Context Propagation (2/2)
CONFIDENTIAL designator

The targeted service extract

All this trace context data
these data from the request
are injected in a HTTP
A client service consumes another request.
service. It creates a trace context with
the Trace Id, the Parent Span Id, the
current Span Id and the Sampling
decision.

This context is then

propagated across multiple
service calls in the same
trace.
98
V0000000
 Distributed Tracing - Enabling in Quarkus application example
CONFIDENTIAL designator

Quarkus supports tracing with minimum source code changes

You need to add the Jaeger client into the

microservice.
This is achieve by adding a dependency in
Maven pom.xml

Then you need to add configurations parameters to enable tracing:

1 A unique service name to identify traces and spans sent to Jaeger

Indicates the rate at which traces and spans should be collected

const: collect all samples
2 probabilistic: random sampling
ratelimiting: collect samples at configurable rate per seconds
remote: control sampling from a central Jaeger backend
Indicates the percentage of requests for which spans should be
3
collected.
4 URL of the Jaeger collector.
5 Indicates Jaeger to propagate x-b3-* related headers.
Log span information for all incoming and outgoing traffic from the
99 6
application.
V0000000
 Distributed Tracing - Enabling in Node.js application example (1/3)
CONFIDENTIAL designator

To enable distributing tracing for Node.js based applications, you must user the jaeger-client and opentracing NPM
packages:

Import the Jaeger client library:

Configure and initialize Jaeger tracer:

100
V0000000
 Distributed Tracing - Enabling in Node.js application example (2/3)
CONFIDENTIAL designator

Start a new span:

Add contextual information to the span using the setTag() method (you can add multiple tags to a span):
You can pass any relevant object as an argument,
which will help you to troubleshoot issues as the
request flows accross services in the mesh.
Once the function finished executing, invoke the span.finish() function to end the span:

101
V0000000
 Distributed Tracing - Enabling in Node.js application example (3/3)
CONFIDENTIAL designator

When you create a new span using the startSpan() method, it creates a new root span by default.
A request can call multiples services in a certain order.
You must link these different calls using parent child relationships so that Jaeger web console can display the
appropriate service call graph and help you trace the call flow.

To declare a span as a child of another span add the childOf property with a value of the parend span to the startSpan() method

To inject contextual information using HTTP headers to outgoing traffic:

For incoming traffic, to extract contextual information, use the tracer.extract() method:

102
V0000000
 CONFIDENTIAL designator

Distributed Tracing - Jaeger (1/3)

$ oc get route jaeger -o template --template={{.spec.host}} -n istio-system

Insert queries in the Jaeger UI to trace your services

103
V0000000
 CONFIDENTIAL designator

Distributed Tracing - Jaeger (2/3)

Select a service

104
Trace you services V0000000
 Distributed Tracing - Jaeger (3/3)
CONFIDENTIAL designator

Click on a trace to
see trace details

105
V0000000
 Metrics CONFIDENTIAL designator

- envoy-stats: The different envoy

proxies.
- istio-mesh: Service mesh metrics
- istiod: The metrics exposed by the
control plane components about
themselves.
- kubernetes-service-endpoints:
Service endpoints that do not
necessarily belong to the mesh.

106
V0000000
 Identifying PassthroughCluster / BlackHoleCluster traffic endpoints (1/2) CONFIDENTIAL designator

If configuring Mixer for Telemetry in ServiceMeshControlPlane CR, you have to add

destination ip metric to be able to identify ip endpoint for unmanaged external traffic
(through PassthroughCluster / BlackHoleCluster icons)

Add destination_ip dimension entry to tcpconnectionsclosed

Add destination_ip label name entry (in tcpconnectionsclosed metric
instance CR located in control plane namespace:
instance) to prometheus handler CR located in control plane
destination_ip: destination.ip | ip(“0.0.0.0”)
namespace.

107
V0000000
Identifying PassthroughCluster / BlackHoleCluster traffic endpoints (2/2) CONFIDENTIAL designator

In Prometheus mesh instance, you can monitoring the traffic hits by viewing istio_tcp_connections_closed_total metric.

Unmanaged traffic is marked as

destination_ip tag shows the BlackHoleCluster /
destination IP for the connection. PassthroughCluster destination
service name.
108
V0000000
 CONFIDENTIAL designator

Visualisation - Kiali console (1/2)

$ oc get route kiali -o template --template={{.spec.host}} -n istio-system

Namespaces that form the Service Mesh

109
V0000000
 CONFIDENTIAL designator

Visualisation - Kiali (2/2)

View topology graph of the service mesh

based on real-time traffic. The graph
changes when traffic hits the services

110
V0000000
 CONFIDENTIAL designator

Advanced
Configuration

111
V0000000
 CONFIDENTIAL designator

Advanced Configuration - Control Plane

When you deploy Service Mesh you create a default ServiceMeshControlPlane CR that you can modify to change
your mesh parameters.

You can configure several Service Mesh components in the CR:

- Gateways
- Prometheus
- Grafana
- Kiali
- Jaeger
- Elasticsearch
- Telemetry

112
V0000000
 CONFIDENTIAL designator

Advanced Configuration - Control Plane

1 Install Grafana. Default: true

2 Indicate if Jaeger will store in memory or in ElasticSearch the

transactions. Default: Memory
1 Install Kiali. Default: true
3

4 Install Prometheus. Default: true

2
Indicates if policy checking / telemetry should be managed by Istiod or
3 Mixer (deprecated). Set to None if policy checking / telemetry is
5
4 disabled. Default: Istiod

6 Name of the ServiceMeshControlPlane profile to use for default

5 values. Default: default
6 Sampling rate in percentage (base 10000) Ex: 5000 -> 50% : Default:
7 10000
5
7 If Jaeger will be used for tracing or None.
8 8 Default: Jaeger

113
V0000000
 CONFIDENTIAL designator

Configuring Service Mesh - Control Plane templates

You can create reusable configurations with ServiceMeshControlPlane templates. To add custom templates you must
create a ConfigMap named smcp-templates in the openshift-operators project

Templates Configmap Inherits Config Marketing Dev

ServiceMeshControlPlane
Development
Template
Inherits Config Accounting Dev
ServiceMeshControlPlane
Production
Template
Inherits Config Accounting Prod
ServiceMeshControlPlane

Replace <templates-directory> with the location of the ServiceMeshControlPlane files on your local disk.

$ oc create configmap --from-file=<templates-directory> smcp-templates -n openshift-operators

114
V0000000
 CONFIDENTIAL designator

Configuring Service Mesh - Control Plane templates

You must mount the ConfigMap in the Operator container at /usr/local/share/istio-operator/templates.

Locate the Operator ClusterServiceVersion name.

$ oc get clusterserviceversion -n openshift-operators | grep 'Service Mesh'
servicemeshoperator.v1.1.5. Red Hat OpenShift Service Mesh 1.0.0 Succeeded

Edit the Operator ClusterServiceVersion

to mount the ConfigMap in the correct
location.

115
V0000000
 CONFIDENTIAL designator

Limitations

116
V0000000
 Limitations - Init Containers
CONFIDENTIAL designator

Because Red Hat Service Mesh use Istio CNI, initcontainers cannot makes network call before Envoy sidecar proxy is
deployed.

If they make network calls, it won’t

work! They can potentially fail or
stuck the pod deployment.

See Istio doc: Compatibility with application init containers

Use one or both of the following settings
● Set the traffic.sidecar.istio.io/excludeOutboundIPRanges annotation to disable redirecting traffic to any
CIDRs the init containers communicate with.
● Set the traffic.sidecar.istio.io/excludeOutboundPorts annotation to disable redirecting traffic to the specific
117
outbound ports the init containers use.
V0000000
Limitations
CONFIDENTIAL designator

● Control Plane workload on infrastructure node: https://issues.redhat.com/browse/OSSM-459

● Persistence storage in istio-prometheus: https://issues.redhat.com/browse/OSSM-316

● Cert Manager Integrations (OCP 4.10, GA in 4.11): https://issues.redhat.com/browse/OSSM-568

V0000000
 CONFIDENTIAL designator

Any Questions?

V0000000
 CONFIDENTIAL designator

Thank you
Red Hat is the world’s leading provider of enterprise

open source software solutions. Award-winning support,

training, and consulting services make Red Hat a trusted

adviser to the Fortune 500.

linkedin.com/company/red-hat facebook.com/redhatinc

youtube.com/user/RedHatVideos twitter.com/RedHat

120
V0000000