? GitLab - A Python Script Calculating DORA Metrics - DEV Community
? GitLab - A Python Script Calculating DORA Metrics - DEV Community
10 1 2 2 2
Initial thoughts
Considered alternate solutions
GitLab Value Stream Analytics (official solution)
LinearB (SaaS solution with free tier)
Four Keys (open source based on GCP)
DORA Metrics and calculations insights
Metric A: Lead Time for Changes
GitLab official calculation
Our calculation
Metric B: Deployment Frequency
GitLab official calculation
Our calculation
Metric C: Change Failure Rate
GitLab official calculation
Our alternative metric calculation: Ratio of Deployments Needing Hotfix(es)
Metric D: Time to Restore Service
GitLab official calculation
Create account
Our alternative metric calculation: Last Hotfix Median Delay
The Python script
Output example
Pre-requisites
Source code
Wrapping up
Further reading
Initial thoughts
The DevOps Research and Assessment (DORA) team has identified four crucial
metrics for measuring DevOps performance. Employing these metrics not only
enhances DevOps efficiency but also effectively communicates performance to
business stakeholders, thereby accelerating business results.
In his insightful article, DORA Metrics: What are they, and what's new in 2024?, Justin
Reock provides a comprehensive overview of these metrics. By the way, you should
also check out the follow-up article, Developer Experience is Dead: Long Live
Developer Experience! 🤓
This script has been executed on a couple of projects and groups across different
organizations, yielding interesting results and with little to no effort on the targeted
codebase.
Let's now look into the available solutions for GitLab projects, how GitLab officially
Create account
compute these metrics, and understand the specific metrics our script calculates.
Regrettably, it is exclusively accessible with the Ultimate license level priced at $99
per developer per month. While it certainly brings value, it may be difficult to
convince managers to upgrade to this level.
LinearB is a SaaS solution that retrieves metrics overtime, some of them being used
to calculate DORA Metrics. They also have a Youtube channel that advocate for DORA
Metrics and more.
The DORA segment is free, and you should certainly explore it while evaluating
solutions in this field.
This is a complete solution that needs a complex set of GCP resources to store and
query the data.
Create account
But we deemed the initial cost too high for initiating DORA Metrics calculation.
We'll dig into the thinking behind our choice, examining the details of GitLab's official
metrics calculations. We'll point out any quirks or limitations and introduce our
alternative methods.
We'll break down the four key DORA metrics—Lead Time for Changes, Deployment
Frequency, Change Failure Rate, and Time to Restore Service. For each one, we'll
compare GitLab's way with ours, making it easy for you to grasp how to get these
metrics practically.
GitLab calculates lead time for changes based on the number of seconds to
successfully deliver a commit into production: from merge request merge time
(when the merge button is clicked) to code successfully running in production,
without adding the coding_time to the calculation. Data is aggregated right after
the deployment is finished, with a slight delay.
OK, so if a commit has been pushed 2 weeks ago, and we merged an hour ago, and
then we deploy to production, the commit age is one hour ? It does not seem quite
right.
By default, lead time for changes supports measuring only one branch operation
with multiple deployment jobs (for example, from development to staging to
production on the default branch). When a merge request gets merged on
staging, and then on production, GitLab interprets them as two deployed merge
requests, not one.
This is hard to understand, so what is the LTfC for this commit ? Does it contribute 2
times to the average ? And what about the branch name ? Is it configurable ? The
verified branch is or ? main production
Our calculation
Our calculation is fairly simple: the average age of commits deployed to production,
created after the last successful deployment to production. Excluding merge
commits.
The branch names checked are and by default, configurable with a regex. main master
This makes sense. But it takes into account bug fixes as valid deployments. If we
deploy one feature a week and then deploy fixes everyday until the feature works, are
we deploying once a day ? It is debatable, but we do not think so.
The calculation takes into account the production environment tier or the
environments named production/prod. The environment must be part of the
production deployment tier for its deployment information to appear on the
graphs.
The environment tier is a nice generic solution. / environment names alternatives are
simple and efficient, but will not fit every projects without some
change. production prod
Our calculation
We chose to discard deployments of hotfixes. For now this is simple, even simplistic:
if the last commit message starts with , it is considered a hotfix, then it is not a
feature deployment. Later versions of the script could involve regular
expression. Merge branch 'hotfix
For environment names, tier is not taken into account (yet), but a regular expression
is used to accommodate to most situations without impacting legacy projects.
Default regular expression is every environment starting with "prod", within a
subfolder or not (). We have to be careful not to include environments.
(|.*\/)prod.* preprod
Again, a smart solution for a problem involving something beyond the code:
detecting an incident. This presupposes several factors; however, achieving precise
accuracy is challenging.
Instead, we compute the average Ratio of Deployments Needing Hotfix. If a hotfix has
been performed after a deployment, we consider the deployment resulted in
degraded service. While this calculation is not the complete metric, it offers a
technical and easily measurable insight.
As for another metric, if the last commit message starts with , it is considered a hotfix
on the previously non-hotfix deployment. Merge branch 'hotfix
In GitLab, time to restore service is measured as the median time an incident was
open for on a production environment. GitLab calculates the number of seconds
an incident was open on a production environment in the given time period. This
assumes:
Instead, we compute the Last Hotfix Median Delay. If there is a hotfix one week after
the last successful non-hotfix deployment, it is considered something needed from
day one, as a fair approximation. This is not the full compute of the metric, hence not
a DORA metric per se, but something technical and easily mesurable.
As for another metric, if the last commit message starts with , it is considered a hotfix
on the previously non-hotfix deployment. Merge branch 'hotfix
Output example
python compute-dora-metrics.py --token $GITLAB_TOKEN --group-id 10000 --days 180
Create account
Pre-requisites
Some Python packages installed
pip install requests ansicolors
An access to all the projects in the group
Hotfixes are performed by branches whose name starts with 'hotfix', and merges
are performed with a merge commit.
Deployments are performed using the environment feature, and the production
environments are distinguished by a common regex.
Source code
"""
GitLab Dora Metrics Calculator for a project or all projects in a group
This script is designed to calculate key DORA (DevOps Research and Assessment) metri
Prerequisites:
- pip install requests ansicolors
- An access to all the projects in the group
Create
- Hotfixes are performed by branches whose name starts with 'hotfix', and account
merges are
- Deployments are performed using the environment feature, and the production enviro
Usage:
python compute-dora-metrics.py --token <token> --days <nb_days> --group-id <group_id
python compute-dora-metrics.py --token <token> --days <nb_days> --project-ids <proje
Arguments:
--token Your GitLab personal access token.
--gitlab-host Your GitLab host (default: gitlab.com).
--group-id The ID of the GitLab group to analyze.
--project-ids Comma-separated list of GitLab project IDs to analyze.
--days The duration of analysis in days (default: 90).
--branch The main branch (default: main).
--env The environment name for production deployments (default: 'prod*'
--debug Whether to print more logs (default: false)
import requests
import argparse
import re
from datetime import datetime, timedelta
from colors import * # COLORS = ('black', 'red', 'green', 'yellow', 'blue', 'magenta
from statistics import median
def get_available_environments_names(project_id):
environments_names = []
params = {
'per_page': 100,
'states': 'available',
}
Create account
any_environment_response = requests.get(f'https://{args.gitlab_host}/api/v4/proj
if any_environment_response.status_code == 200:
environments = any_environment_response.json()
if environments:
for environment in environments:
environments_names.append(environment['name'])
return environments_names
params = {
'environment': env_name,
'per_page': 100,
'order_by': 'updated_at',
'sort': 'desc', # Sort in descending order (oldest first)
'status': 'success',
'updated_after': updated_after,
}
response = requests.get(f'https://{args.gitlab_host}/api/v4/projects/{project_id
if response.status_code == 200:
return response.json()
else:
if args.debug: print(red(response.json()))
return []
last_deployment_was_a_hotfix = True
else:
print(black(f"Deployment ref={ref} launched_at={launched_at}"))
nb_standard_deployments += 1
if last_commit_date_of_previous_deployment:
params = {
'since': last_commit_date_of_previous_deployment,
'until': launched_at,
'ref_name': branch,
'per_page': 100,
}
commits = get_commits(project_id, params)
if args.debug: print("=> New Commits Since Previous Deployment:")
if commits:
for commit in commits:
if not is_merge_commit(commit):
time_diff = get_time_delta(launched_at, commit['created_
if args.debug: print(f"Commit: {commit['id']}, Commit Da
all_time_differences.append(time_diff)
last_deployment_was_a_hotfix = False
if all_time_differences:
median_time_difference = calculate_median_delta(all_time_differences)
print(green(f"=> Median commit age before '{env_name}' deployment for {proje
if days > 0:
return f"{days} days, {hours} hours, {minutes} minutes"
elif hours > 0:
return f"{hours} hours, {minutes} minutes"
elif minutes > 0:
return f"{minutes} minutes"
else:
return f"{seconds} seconds"
durations = []
queued_duration: []
coverage: []
page = 1 Create account
params = {
'per_page': 100,
'updated_after': updated_after,
'status': "success",
'source': "merge_request_event",
'page': page
}
# get latest pipeline id list for given project_id
while True:
response = requests.get(f'https://{args.gitlab_host}/api/v4/projects/{projec
if response.status_code == 200:
page_pipelines = response.json()
# print(red(response.json()))
if not page_pipelines:
break # No more pages, exit the loop
for pipeline in page_pipelines:
pipeline_response = requests.get(f'https://{args.gitlab_host}/api/v4
pipeline_detail = pipeline_response.json()
# print(red(pipeline_response.json()))
durations.append(timedelta(seconds=pipeline_detail["duration"]))
page += 1
params['page'] = page
else:
print(f"Error: {response.status_code} {response.reason}")
break
return median_duration
else:
return None
while True:
response = requests.get(f'https://{args.gitlab_host}/api/v4/projects/{projec
Create account
if response.status_code == 200:
page_commits = response.json()
if not page_commits:
break # No more pages, exit the loop
commits.extend(page_commits)
page += 1
params['page'] = page
else:
print(red(response.json()))
break # Handle errors or stop on non-200 response
return commits
# Function to retrieve project IDs for a group and its subgroups recursively
def get_project_ids_with_available_environments_for_group(group_id, full_path):
nb_projects_found = 0
project_ids_with_envs = []
response = requests.get(f'https://{args.gitlab_host}/api/v4/groups/{group_id
if response.status_code == 200:
projects = response.json()
for project in projects:
if full_path in project['path_with_namespace']:
nb_projects_found += 1
print(f" 🗒️ found project {cyan(project['path_with_namespace'])}
environments_names = get_available_environments_names(project['i
if environments_names: Create account
print(f", environments: {magenta(environments_names)}")
project_ids_with_envs.append(project['id'])
else:
print("")
# weird bug: the API return projects not in the group O_o
else:
print(red(response.json()))
return nb_projects_found
if ttlh_values:
average_ttlh = calculate_median_delta(ttlh_values)
print(green(f"=> Median Time To Last Hotfix for project {project_id}: {forma
return average_ttlh
else:
return None
if args.group_id:
response = requests.get(f'https://{args.gitlab_host}/api/v4/groups/{args.gro
if response.status_code == 200:
nb_projects_found, project_ids_with_env = get_project_ids_with_available
else:
print(red("Group does not exist. Did you provide a project ID ?"))
exit(1)
if not project_ids_with_env:
print("No projects with environments found in the group")
elif args.project_ids:
project_ids_with_env = [project_id.strip() for project_id in args.project_id
nb_projects_found = len(args.project_ids.split(','))
else:
print(red("You must provide either --group-id or --project-ids"))
exit(1)
projects_without_prod_env = set()
projects_without_enough_deployment = set()
projects_with_analyzed_deployments = set()
all_projects_time_differences = []
all_projects_ttlh = []
all_deployments_dates = set() # Collect deployment dates across all projects
nb_total_deployments_without_hotfix = 0
nb_total_deployments = 0
response = requests.get(f'https://{args.gitlab_host}/api/v4/projects/{projec
if response.status_code != 200:
print(red(f"Project {project_id} is not reachable (HTTP code {response.s
if args.debug: print(red(response.json()))
continue
project_info = response.json()
project_name = project_info.get('path_with_namespace', 'Unknown Project')
title = f' Processing project {project_name} (ID {project_id}) '
print(f"{title:-^100}")
environments_names = get_available_environments_names(project_id)
print(f"Environments: {magenta(environments_names)}")
branch = args.branch.split('|')[0] Create account
alternative_branch = args.branch.split('|')[1] or "none"
all_deployments_dates.update(project_deployment_dates)
nb_total_deployments += nb_project_deployments
nb_total_deployments_without_hotfix += nb_project_deployments_without_ho
if has_prod_env == False:
projects_without_prod_env.add(project_name)
print("")
print(f"{nb_projects_found} total project(s) analyzed")
print(f"{black(len(projects_without_prod_env))} project(s) have environments but
print(f"{blue(len(projects_without_enough_deployment))} project(s) have a '{args
print(f"{green(len(projects_with_analyzed_deployments))} project(s) contributed
print(f"Checked deployments after {updated_after} (last {args.days} days)")
if all_deployments_dates:
num_days_with_deployments = len(all_deployments_dates) Create account
deployment_frequency_days = args.days / num_days_with_deployments
print(f"Overall, there has been {green(num_days_with_deployments)} days with
deployment_frequency_human_readable = format_time_difference(timedelta(days=
print("")
print(green(f"DORA Deployment Frequency:".ljust(30) + f"{deployment_frequenc
if all_projects_time_differences:
overall_median_time_difference = calculate_median_delta(all_projects_time_di
print(green(f"DORA Lead Time for Changes:".ljust(30) + f"{format_time_days(o
if nb_total_deployments > 0:
deployments_with_bug_ratio = 1 - ( nb_total_deployments_without_hotfix / nb_
print(green(f"Deployments Needing Hotfix:".ljust(30) + f"{deployments_with_b
if all_projects_ttlh:
overall_ttlh = calculate_median_delta(all_projects_ttlh)
print(green(f"Time to Last Hotfix:".ljust(30) + f"{format_time_days(overall_
print("")
Wrapping up
In this article, we explored DevOps Research and Assessment (DORA) metrics,
comparing various solutions and calculating key metrics like Lead Time, Deployment
Frequency, Change Failure Rate, and Time to Restore Service. We contrasted GitLab's
official calculations with our simpler alternatives, introduced a Python script for
GitLab metric computation.
Whether using official solutions or our alternatives, the aim is clear: cultivate
efficiency, reliability, and swift software delivery in the dynamic DevOps landscape.
Check out the Python script for practical metric calculations! And feel free to provide
feedback or suggestions in the comments below! 🚀
Create account
Further reading
Faster Pipelines
Benoit COUETIL 💫 for Zenika ・ Nov 6 '23
#gitlab #devops #pipeline #cicd
This article was enhanced with the assistance of an AI language model to ensure clarity
and accuracy in the content, as English is not my native language.
👋 Before you go
Do your career a favor. Join DEV. (The website you're on right now)
It takes one minute, it's free, and is worth it for your career.
Get started Create account
Zenika
☸️ Why Managed Kubernetes is a Viable Solution for even Modest but Actively Developed
Applications
#kubernetes #architecture #devops #webdev