PROGRESSIVE

DELIVERY
WITH GITOPS
A Pocket Guide
 2
▼
PROGRESSIVE DELIVERY WITH GITOPS

With the push to deliver cutting edge features faster than ever before, customer experience has
become a top priority for software delivery teams. These teams are torn between maintaining
outstanding customer experience and building and releasing features faster than before. To find
a way to balance both these priorities, the most successful organizations like Google, Amazon,
Netflix, and Uber have adopted an innovative approach that not only allows them to release
faster, but to maintain very high customer experience ratings. This approach has been termed
Progressive Delivery.

What is Progressive Delivery?

Progressive delivery is the process of pushing changes to a product
iteratively, first to a small audience and then to increasingly larger audiences
to maintain quality control.” — TechTarget

Organizations can massively reduce the risk of failures in deployments by adopting progressive
delivery approaches. We say ‘approaches’ because there are many ways to implement progressive
delivery. Some approaches are by adding flag management to test code, and progressive rollouts
of applications through techniques like blue-green and canary deployments. In essence, you can
decide which user group gets the release and how much to limit the risk exposure.

There are quite a few elements involved in streamlined progressive delivery.

Let’s take a look at each one of them and understand their role.

Everyone

Staged Rollouts

Beta

Team

PROGRESSIVE DE LIVE RY

weave.works
 ▼ 3

PROGRESSIVE DELIVERY WITH GITOPS

5 Key Elements of Progressive Delivery
1 Manage code via a version control system
Using a version control system like Git will allow you to easily track, correct and revert
code from a single point. With Git-based hosting platforms to store your code and
implement the philosophy of a single source of truth, your team can concurrently work
on different versions of code.

Efficient source code management allows you to push, test, and roll back feature
developments seamlessly.

2 Feature flag management

Feature flagging or feature toggling is the process of turning particular parts of your
code ‘on’ or ‘off,’ which eliminates the need to deploy them. Feature toggling enables
you to control the execution of code remotely. One of the many applications of
feature flagging is to mitigate risk while introducing a new capability. You can do this
by switching on the toggle associated with the new code to a small subset of users.
In case of any negative impact or performance lags, you can easily roll it back by
switching the toggle ‘off.’

3 Feature rollout strategies

Feature rollout is the mechanism of releasing a newly added feature to a set of users
for preliminary review. It follows the philosophy of limiting the risk and improving
release efficiency. Earlier, organizations followed the practice of bundling all new
features and releasing them together across the entire user base. This, however, took
much longer and when failures happened, it required them to sometimes take down
the entire application to investigate issues.

Below are the top three commonly used feature release strategies:
A. Blue-Green Deployment
B. A/B Testing
C. Canary Deployment

With more frequent releases, the chances of negatively affecting application

reliability or customer experience can increase. Learn why effective deployment
strategies help teams minimize risk.

DEEP DIVE INTO KUBERNETES DEPLOYMENT STRATEGIES

weave.works
 4
▼
PROGRESSIVE DELIVERY WITH GITOPS

4 Observability
Implementing progressive delivery is essentially observing how changes in your
application are affecting your customer’s experience with the software. It leverages
information, metrics, and logging capabilities to ensure that the new version of the
software is safe for release. These metrics need to be reported and monitored in real-time.

Observability versus Monitoring

HERE ARE THE DIFFERENCES

5 Service Mesh
A service mesh facilitates service-to-service communication and adds capabilities like
observability, traffic management, and user segmentation. It plays a key role in facilitating
communication between the various microservices that makeup the application. It gives
the Network Administrator superpowers over what happens at the networking layer of
cloud-native applications. As traffic routing is integral to progressive delivery, service
meshes play a key role in enabling progressive delivery.

As you can tell, progressive delivery is a collection of a number of tactics and tools that
make up an entire strategy. You will not use all these tools and tactics together at any
given time, but which you choose will depend on your end goals.

How Does Progressive Delivery Work?

Progressive delivery is gaining wide acceptance as it solves the problem of introducing new
features without compromising on performance, stability, and the end-user experience. You can
utilize one or more feature release strategies to ensure an optimized application release. Once you
collect test data from the users who were exposed to the new features, you will be in a better place
to decide whether they’re ready for prime time.

PRODUCTION

Light Medium High

Canary Pilot
Test Canary load load load
launch regions regions
regions

PROGRESSIVE DELIVERY WORKFLOW

weave.works
 ▼ 5

PROGRESSIVE DELIVERY WITH GITOPS

Once you introduce the progressive delivery practice, your SDLC will run in a loop as below:

1. Your team writes code, commits it to Git, and initiates its release.
2. The released update is rolled out to a very specific and a small section of your users.
3. You gather intel on the feature’s performance, detect errors, and performance
bottlenecks, if any.
4. Use the insights to plan and implement follow-up improvements.

Progressive Delivery vs Continuous Delivery

Though progressive delivery is built on the principles of continuous delivery, there are some
fundamental distinctions between the two in terms of testing.

Continuous delivery is a philosophy of accelerating software production by keeping code in a

deployable state at all times. With greater emphasis on pre-production testing, it allows you to
move at a faster pace accommodating multiple and frequent releases.

Progressive delivery, on the other hand, is an upgrade to the rules of continuous delivery where
greater emphasis is laid on developer control and software reliability. It enforces delivery speed
without compromising on quality by going granular i.e., to the feature level.

Here’s a quick table of differences between the two:

Continuous Delivery Progressive Delivery

Focuses on application releases Focuses on feature releases

Trusts pre-production testing Enforces production testing

Emphasizes deployment-ready code Emphasizes developer control

Cannot protect customers from Implements mechanism to contain

code issues the damage from bad code

Entire codebase should be rolled out Features can be rolled out or rolled
or rolled back back independently

weave.works
 6
▼
PROGRESSIVE DELIVERY WITH GITOPS

Benefits of Progressive Delivery

Progressive delivery is gaining prominence among successful organizations as it puts developers in
control of the roll-outs. Below are some of the key benefits of this delivery practice:

1. Greater control over quality:

It enables you to intentionally improve the quality of software releases by fixing code errors and
bugs in the features being deployed.

2. Ability to predict failures:

Through observability and feature release strategies like canary releasing, you gather data on
performance spikes and errors, and can predict any potential issues early on.

3. Faster feedback implementation:

By releasing new features to a small subset of users, you can quickly understand how a release is
performing and implement the learnings to improve the application.

4. Built to deal with failures:

Progressive delivery as a philosophy is developed to deal with failures. With mechanisms like the
feature flag, you can quickly roll back features in case of failures.

5. Keep downtime to a minimum:

By gradually transferring traffic to newer versions of the code, you can keep downtime to a
minimum. Even in cases of failures, the blast radius is very small.

Challenges with Progressive Delivery

Despite the many benefits, progressive delivery comes with a few challenges. The manual effort
that the deployment process demands can be overwhelming. The enabling and disabling of
new features, monitoring for latencies, changing traffic routing patterns, and tracking errors and
saturations are just some of the critical tasks you will have to undertake.

In addition to this, the deployment strategies bring in their own complexities. For example, in blue-
green mirroring, if a customer calls for customer support, you will first have to verify which version
group the user belongs to.

Additionally, many progressive delivery solutions that are available lock organizations into a
particular product or platform as they’re often vendor-specific solutions. Such a predicament forces
organizations to seek new tools that are more open. One such approach is GitOps.

weave.works
 ▼ 7

PROGRESSIVE DELIVERY WITH GITOPS

A quick reminder on GitOps
GitOps is a set of principles that manages infrastructure and application configuration using a
version control system like Git. GitOps, similar to progressive delivery, is designed as a developer-
centric framework and is an extension of DevOps practices. GitOps adoption is growing
exponentially as it delivers high productivity, reliability, and security for cloud-native applications.

The GitOps practice is built on the four foundational principles, which are critical for anyone to
implement a successful GitOps methodology.

1. Declarative
2. Versioned and Immutable
3. Pulled Automatically
4. Continuously Reconciled

You can learn more about these at https://opengitops.dev/.

The GitOps Pipeline

Building a GitOps pipeline is very different from a CI/CD pipeline as the former features an
ecosystem of could-native tools. Below, we list the main stages of a GitOps pipeline along with
the ideal tools.

Source Code Management

A git-based hosting service to store and manage source code and infrastructure code is a vital
aspect of GitOps. A Git repository enables your team to write code, collaborate with your team,
manage code changes, document version history, and push code changes to be implemented.
The popular Git repositories are GitHub, GitLab, and BitBucket.

Packaging Code with Container Images

Once you have the code in Git, the next step is to package the code using containers. You create
container images and store them in a container registry. The container images contain information
like application executables and dependencies. They are used as templates to scale software
deployment at speed. Therefore they need to be stored in a secure registry. Docker Hub, GitHub
container registry, and AWS Elastic Container Registry (ECR) are popular container registries.

weave.works
 8
▼
PROGRESSIVE DELIVERY WITH GITOPS

Automated Deployment with GitOps

In this part of the pipeline Flux plays a key role in automatically detecting the new container image.
When this happens Flux updates the configuration in Git either directly or via a pull request. The
merged configuration change is pulled automatically and continuous reconciliation updates the
Kubernetes cluster. Flux uses Kubernetes solutions - Helm and Kustomize to efficiently implement
the automated workflow.

Helm is a package manager for Kubernetes,

​​ and it uses a format called charts. A chart contains a
set of templates for a deployment. Charts can have dependencies, they can include other charts,
which in turn, can include other charts. At deployment time values are applied to the templates.
Therefore a single chart may deploy multiple applications to multiple environments like staging, or
production by using different values.

Kustomize is another Kubernetes tool that automates configuration changes across clusters.
Keeping a base layer of a resource like a Helm chart, Kustomize adds several patch layers (like
staging, or prod) to customize the base.

Progressive Delivery with GitOps

GitOps employs progressive delivery strategies enabling traffic routing in a phased manner to keep
negative ramifications at a bare minimum. As a leading progressive delivery tool, Flagger is tightly
integrated with Flux and service meshes like Linkerd and Istio to automate canary deployments.
We discuss Flagger in greater detail below.

Monitoring

Monitoring and observability are critical parts of the GitOps pipeline. Once the deployments begin,
the system constantly monitors the pipeline for errors, latencies, and security risks. Prometheus is
the leading monitoring tool for Kubernetes
​​ systems and is the default monitoring tool used by Flux
and Flagger. Prometheus or other O11y tools such as New Relic, Datadog or similar collect real-time
performance metrics and integrates with Grafana to visualize this data.

The GitOps Approach to Progressive Delivery

As much as progressive delivery improves code quality and reduces risks, it comes with quite a
few challenges that you might face when implementing it. For effective execution of progressive
delivery, you must identify the ideal set of users, rollout frequency and volume, and measures to
take in case of failures. One way of ensuring all the best practices are taken into account, is by
manually monitoring metrics in real-time to assess how well the deployment is going. This, however,
is a challenging task and can’t be done efficiently by a human who would need to be glued to the
metrics every second.

weave.works
 ▼ 9

PROGRESSIVE DELIVERY WITH GITOPS

GitOps has the ideal development philosophy to manage progressive delivery, simplifying various
complex parts of the delivery process. As discussed earlier, with GitOps’ declarative approach to
infrastructure, application code and infrastructure configuration are stored within Git as the single
source of truth. Using tools like Flux, you can ensure that the current state of the system stays in line
with the declared state.

Flagger - The Leading Progressive Delivery Tool

Flagger is a progressive delivery operator for Kubernetes that automates the release of applications
securely. It mitigates the underlying risk factors of introducing new software versions in production
through deployment strategies like canary releases, blue-green mirroring, and A/B testing). It
gradually shifts traffic to the new version using service mesh tools like Istio, Linkerd, and App Mesh
or an ingress controller like Skipper, NGINX, and Contour. Flagger can be integrated with GitOps
tools like Flux since it is declarative and made for Kubernetes.

Code repo CI Docker Registry

Dev Commit Build Push

ve
Wea Flux
Cluster repo

Operator Commit Sync

pull

apply

analise
ry
Dep

na

10% oy a
Ingress Fla
g g er
l

m ent C
e
ot
om

query
pr

User HTTPS

90%
ar y
Dep

oy
m

i ro
m e n t Pr m etheu
s
l

Source: https://github.com/stefanprodan/gitops-progressive-delivery

weave.works
10
▼
PROGRESSIVE DELIVERY WITH GITOPS

Flagger also comes with capabilities to automate the analysis and testing of code changes to
ensure their stability before releasing them across the whole cluster. It shifts traffic to the new
version while keeping the old one as a backup. You can also integrate Flagger with tools like Slack
and Microsoft Teams to send out timely updates across your team.

Flagger triggers a canary release process anytime it observes changes in any

of the following objects:

Deployment PodSec ConfigMaps Secrets

A Podspec contains the A ConfigMap is an API object Containers in a pod use
version of a Pod library with containing non-sensitive data Secrets mounted on data
information on container that Pods use as environment volumes or utilized as
images, commands, ports, variables, command-line environment variables.
environments, and other arguments, or as config files
metadata. in a volume

Flagger employs the below strategies to automate the end-to-end process of application analysis,
testing, and rollback:

Canary releases for progressive traffic shifting

Flagger implements a canary release deployment strategy in the form of a control loop gradually
shifting traffic to the new software version. The loop comes to an end once the entire traffic
is shifted to the new version. It constantly measures the health of the canary by monitoring
performance metrics like HTTP requests success rate and requests average duration. If the KPIs do
not meet the criteria, the traffic routing is aborted and the release will be rolled back.

1 2 3
v1 v1 v1 v2

5% v2 10...50%

v1 v1 v1 v2

4 5 6
v2 v2 v2 v2 v2 v2

50% 0%

v1 v2 v2 v2 v2 v2

Image Source: https://docs.flagger.app/usage/deployment-strategies

weave.works
 ▼ 11

PROGRESSIVE DELIVERY WITH GITOPS

You can implement an emergency traffic shift by skipping the analysis stage. Flagger checks the
health of the canary deployment and releases the code changes.

Under normal circumstances, the canary analysis is run at regular intervals until the entire traffic is
moved to the new version or the failed checks threshold is met.

A/B testing for HTTP headers & cookies-based traffic routing

With Flagger, you run A/B testing by implementing match conditions based on HTTP headers or
cookies to ensure that your subset of users is routed to the respective canary for the whole duration
of the analysis. This Flagger configuration is particularly important for frontend applications that
need session affinity.

1 2 3
v1 v1 v2 v1 v2

conformance tests insiders

v1 v1 v2 v1 v2

4 5 6
v1 v2 v1 v2 v2 v2

collect metrics validate SLOs

v1 v2 v1 v2 v2 v2

Image Source: https://docs.flagger.app/usage/deployment-strategies

Blue-green deployment for gradual traffic switching

When orchestrating a blue-green deployment strategy, Flagger quickly runs tests to ensure that the
canary pods are stable. If the analysis turns out well, the traffic is gradually shifted from the primary
version (blue) to the canary (green). However, if the failure threshold is met, Flagger will abort the
traffic routing.

weave.works
12
▼
PROGRESSIVE DELIVERY WITH GITOPS

v1 v1 v2 v1 v2

conformance tests load tests

v1 v1 v2 v1 v2
1 2 3

4 5 6
v1 v2 v1 v2 v2 v2

collect metrics validate SLOs

v1 v2 v1 v2 v2 v2

Image Source: https://docs.flagger.app/usage/deployment-strategies

Once the traffic is routed to the canary (green), the primary (blue) is updated with the canary spec.
Following this, the traffic is routed back to the primary with the updated code.

Blue-green mirroring for traffic shadowing

Unlike a blue-green deployment, blue-green mirroring involves a pre-deployment test phase.

In this strategy, Flagger creates a mirror copy of all the incoming traffic requests to the primary and
sends it to the canary. The response from the primary version is sent to the user, while the canary
response is discarded.

The monitoring mechanism, meanwhile, collects metrics from both versions and evaluates factors
such as request success rate and request duration. If the canary metrics are healthy, the gradual
traffic routing to the canary begins.

Once 100% of the traffic is routed to the canary, the primary version is updated and the traffic
is sent back to the main version once primary pods are checked out.

weave.works
 ▼ 13

PROGRESSIVE DELIVERY WITH GITOPS

Quick summary of the 4 core benefits

1. Safer feature releases

Flagger consistently measures key metrics like HTTP headers and gRPC request success
rates and latency before pumping in the full traffic to the new version of the software. This
ensures that your software release is measured and risk-free.

2. Flexible traffic routing

You can employ a number of strategies to move traffic from one version to another while
continuously observing the process. Using a service mesh like Istio, Linkerd, and Kuma, or
an Ingress controller like Contour, Gloo, and NGINX to manage the traffic routing.

3. High integration capabilities

Besides compatibility with every tool built within the Kubernetes ecosystem, Flagger
can be integrated with a number of platforms to implement an effective alerting system.
Link Flagger with your internal collaboration tools like Slack and Microsoft Teams to get
automated updates of the workflow.

4. Greater observability
Flagger offers great observability capabilities to validate service level objectives (SLOs)
based on application-specific metrics. It monitors and measures objects like availability, error
rate percentage, and average response time. The traffic routing or shifting is decided based
on the analysis of these factors to ensure minimal impact on the users.

Weave GitOps - The best way to operationalize Flagger

Weave GitOps enables you to run cloud-native applications on Kubernetes. With this platform you
don’t need any expertise to run apps in clusters. Weave GitOps is powered bu Flux and built in
tandem with Helm, Flagger, Prometheus, and Grafana to form a seamless GitOps pipeline including
progressive delivery. It enables you to simplify automated deployments to implement successful
progressive delivery.

weave.works
14
▼
PROGRESSIVE DELIVERY WITH GITOPS

Progressive Delivery Across Multi Cloud

Weave GitOps is a great choice especially for organizations introducing progressive delivery across
multi cloud set ups.

Consider a scenario where the development cluster and production cluster are running on two
different cloud platforms. Progressive delivery won’t be a problem here because you initiate a
change from the Git repository through a pull request. This is approved through a merge. With a
Git-centric workflow, you don’t need to communicate with the clusters directly. The deployments are
executed right from Git.

Executing a similar process manually will be a hassle with constant traffic re-balancing and a
number of manual deployments. With GitOps and Flagger, teams can automate the workflow
through a set of rules. The deployment will take care of itself, all you need to do is to monitor the
live implementation of progressive delivery and fine-tune it whenever necessary. In case of a failure,
the rollback process is simple thorough revoking the change within Git. Other benefits of Git are
code versioning, code reviews, change history, and security.

weave.works
▼ 15

PROGRESSIVE DELIVERY WITH GITOPS

Key Takeaways

1 Progressive delivery streamlines the process of adding new features to your

software and adds focus on stability, reliability, and end-user experience.

2 Progressive delivery implements code changes quickly using methodologies like

feature flag management and various deployment strategies - canary, green/blue,
A/B testing, and so on.

3 GitOps is an ideal workflow to implement progressive delivery as it drives

software development through automation and pull requests using Git as the single
source of truth.

4 Both GitOps and progressive delivery aim at giving developers greater

control over the deployment of new features by mentioning the system state
in a declarative manner. Flagger couples with a service mesh to implement the
declaratively defined instructions.

5 Progressive delivery tools like Flagger empower developers with capabilities to

automate the analysis and testing of new code changes. This approach makes sure
that the system is stable before releasing them across the whole cluster..

6 With Weave GitOps Enterprise, which includes Flux and Flagger, deployments run in
an automated mode, as you monitor the live implementation of progressive delivery
and make necessary changes whenever needed. Weave GitOps is the easiest and
most controlled way to implement progressive delivery at scale.

CONTACT US FOR A DEMO OF WEAVE GITOPS TODAY

[email protected] weave.works

weave.works