Stealth operations: The evolution

of GitLab's Red Team

We discuss how GitLab's Red Team has matured
over the years, evolving from opportunistic
hacking to stealth adversary emulation.

At GitLab, our Red Team conducts security exercises that emulate

real-world threats. When the team was first formed, these exercises
were opportunistic and done in plain sight. As the GitLab Security
organization matured, so did our Red Team.

We now perform a majority of our operations in stealth, meaning that

only a small group of team members are aware of the details.

This blog dives into the steps we took as we matured and lessons we
learned along the way. We also share highlights of a recent stealth
operation and the value it provided our organization.

If you're building an offensive security practice, or looking to mature

an existing one, you may find some inspiration below.

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 1 of 11
:
 Where we started
Our Red Team was formed in July 2019 - about four years ago. We
started off as three engineers and one manager spread across the
U.S., Australia, and Europe.

Back then, GitLab's security maturity was at an earlier stage. Some

of the more advanced capabilities we have in place today were still
being planned or improved.

As newly hired hackers, it was tempting to jump right into emulating

advanced threat actors in top-secret operations. But we weren't just
hackers - we were a Red Team with a mission to help make our
organization more secure. It wasn't just about attacking all the
things, it was about identifying and addressing realistic threats.

Getting to know GitLab

Before we started hacking, we did the following:

Wrote down what we were doing, why were doing it, and what
rules we would stick to. This was critical to our success,
especially as a team that worked asynchronously across time
zones.
Met with our counterparts in Security Incident Response (SIRT)
to understand how they could benefit from an offensive security
practice.
Met with our counterparts in Engineering and IT to build
relationships and help them understand our overall goals and
approach.
Read. A lot. Documentation, runbooks, architecture diagrams.
Whatever we could find to understand GitLab's environment
and attack surface.

Getting to work
https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 2 of 11
:
 Finally, it was time to hack.

We started out doing what we called "open-scope" work, which was

similar to a penetration test but without the bureaucracy and
boundaries of a typical time-based engagement. We wrote
enumeration scripts, scanned publicly exposed cloud resources, and
hunted for leaked secrets.

When we found something that could be hacked, we hacked it and

reported it in an issue to prevent it from happening again.

As we noticed patterns emerging, we developed automation to more

efficiently find and report them.

This was great - it reduced risk at GitLab and gave our team a
chance to better understand our environment and its risks.

But it wasn't quite Red Teaming.

We were finding, exploiting, and reporting vulnerabilities, but we

weren't providing GitLab with an opportunity to practice detecting
and responding to real-life attackers.

How we planned to mature

Over time, we found systemic solutions to more and more of the
opportunistic findings. A new Vulnerability Management group was
formed, taking ownership of our custom scanners and implementing
more robust and permanent solutions. Visibility and control over
endpoints increased as did the ability to monitor and alert across our
entire organization.

As GitLab's defensive capabilities matured, it became important for

the Red Team to do the same. We needed to emulate more advanced
attackers and provide more realistic opportunities to detect and
respond to these attacks.

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 3 of 11
:
 We needed a plan.

We created a maturity model with unique stages showing where we

started, where we were, and where we were headed. Each stage had
a list of behaviors the team strived to demonstrate, or states we
hoped to achieve.

This gave us a broad roadmap that we could work towards for the
next two-to-three years. Looking back, it was worth the effort. We
use the roadmap extensively, leveraging it to guide tricky decisions
and to plan quarterly goals that moved us further on our journey.

The inspiration for our model came from many places, including:

The general-purpose Capabilities Maturity Model

The Red Team maturity model from Jordan Potti, Noah Potti,
and Trevin Edgeworth
The Red Team maturity model from Brent Harrell and Garet
Stroup

We used a GitLab issue board to build the model.

You can read about the logistics and benefits of using an issue board
in our handbook.

This is what our model looks like:

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 4 of 11
:
 Key milestones along the way
When we first wrote our maturity model, we were sitting somewhere
in the second column. Moving beyond that would require a big shift -
from opportunistically finding and exploiting vulnerabilities to
emulating adversaries and providing opportunities for detection and
response.

For us, that path started with Purple Teaming and then moved on to
stealth operations.

We used GitLab epics to make high-level plans for each of these

stages. Epics allow you to group individual issues, breaking down
long-term projects into actionable tasks.

Implementing Purple Teaming

Purple Teaming was a pathway to stealth operations. It would give us

an opportunity to build and practice our processes transparently and
in collaboration with our Blue Team.

We made a plan to develop these processes and to test them out by

conducting a small-scale Purple Team operation. This was done in
the context of an OKR (Objectives and Key Results), and took us
about three months to complete.

Here is the description from the epic we opened to get started:

OKR: Purple Team Foundations & Initial Run

Our SIRT team continues to grow and implement more robust

detection and response capabilities. Recently, they have begun to
adopt the MITRE ATT&CK framework for classifying attack
techniques.

These strategies are highly aligned with our own, and build an

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 5 of 11
:
 excellent framework for a more collaborative approach in
planning, designing, and executing attack emulations. When both
teams are involved in all stages of a campaign, we are more likely
to produce an outcome that is actionable and beneficial to the
organization.

This OKR will allow us to focus on ensuring all of the

foundational/logistical pieces are there, and then to execute a
smaller controlled operation to make sure we got it right.

At a high-level, the OKR contained the following tasks:

Meet with various teams at GitLab to discuss what we were

trying to accomplish, how we would work together across
timezones, what rules we should put in place, etc.
Plan for specific changes/additions to our handbook to capture
the results of those discussions.
Collaborate across teams to plan and execute a small operation
using these new processes.

When the quarter was complete, we had the following to show for it:

Purple Teaming at GitLab: A handbook page describing our

methodology
Red Team issue templates: Public, re-usable templates for
planning, executing, and reporting on operations
"How we run Red Team operations remotely": A blog talking
about the how and why we do this work asynchronously across
time zones

We then used those processes and issue templates to plan and

execute a small Purple Team operation. The brainstorming stage
allowed us to work with our friends in SIRT, identifying recurring
security themes and selecting attack techniques that would allow
them to improve their detection and response capabilities.

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 6 of 11
:
 We replicated a token leak where an attacker leveraged legitimate
credentials to establish persistence and move laterally within the
GitLab.com environment. This provided an opportunity to test
existing security information and event management (SIEM) alerts,
validate the ability to locate all malicious activity in log files, and to
implement earlier detection and prevention capabilities.

We made changes to our Purple Teaming processes based on

lessons learned. In following quarters, we moved on to full-scale
emulation of relevant adversaries using a Purple Team process that
was developed and tested in collaboration with groups across our
organization.

Implementing stealth operations

Shifting to stealth was a natural evolution from Purple Teaming. We

continued to work from our maturity model, operating from the plan
that was already established and communicated across the
organization.

Just as we did with Purple Teaming, we created an epic to shift to

stealth operations by default and aligned it with our quarterly OKR.

This epic was opened with the following description:

OKR: Improve the maturity of the Red Team by shifting to

stealth operations by default

As part of our general team roadmap, we are focusing on

maturing the Red Team's processes and procedures this year.
This quarter, we will complete various tasks allowing us to shift to
a "stealth by default" way of performing operations.

This will provide the organization a better opportunity to practice

detecting and responding to the most relevant and realistic
threats.
https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 7 of 11
:
 We will do this by:

Refreshing the Red Team Rules of Engagement by

collaborating with SIRT and agreeing on processes and
procedures.
Researching, documenting, and automating architecture
requirements for stealth operations.

We ended up breaking those two bullet points into separate child

epics, as there was a lot of work to do in each.

The first child epic, around processes, resulted in output that is

mostly public. Some examples are:

A short summary of Stealth Operations in our general handbook

page
A new Stealth Operations section in our rules of engagement
Example Stealth Operation Techniques section, also in the rules
of engagement
Iterations to our Red Team issue templates

We were very transparent with all of these changes. Each change

was a merge request, which was visible to everyone at GitLab. We
opened a dedicated issue to discuss any concerns and used an all-
company Slack channel to invite everyone to provide feedback.

As an internal Red Team, building relationships across the

organization is key to our success. We find that transparency about
how we operate helps us maintain this trust.
https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 8 of 11
:
 The second child epic, around technical research, resulted in outputs
that are mostly not public and involved things like:

Using "Attacker VMs" with Parallels on our corporate laptops.

This provides us a space without security monitoring, where we
can use commercial VPNs to appear as separate entities when
emulating remote attackers.
Working with our IT department to acquire our own AWS
accounts with exceptions to standard security monitoring. This
gives us a space to install our C2 infrastructure, phishing sites,
etc.
Testing various command and control (C2) frameworks, agents,
and redirectors. Designing automation to deploy these
environments from scratch with each new operation.
Establishing private communication channels and a wiki for Red
Team engineers and trusted participants.
Testing encrypted secret management tools for temporary
storage during operations.

Results from a recent stealth operation

With our new tools and processes in place, stealth operations
became our default.

One recent operation began with selecting an attack group that had
been in the news for targeting organizations similar to ours. This
operation spanned three months - the majority of which was spent
on researching the adversary and developing capabilities to emulate
them.

We started with a volunteer from a non-security team at GitLab.

They were one of our "trusted participants", meaning they were
briefed on the operation. We had them visit a website we created
which mimicked the download page of a popular open-source
desktop utility. They downloaded the utility and followed the on-
https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 9 of 11
:
 screen instructions to install and authorize it.

The application was a modified fork of the legitimate tool, created

just for this operation. It contained an embedded script which
downloaded our command and control (C2) agent and provided the
Red Team access to the laptop. This scenario mirrored the adversary
we were emulating, who would deploy malware to engineers'
laptops.

Using an insider to launch the initial payload is a common Red Team

technique called an "assumed breach." This allows the Red Team to
focus their efforts on emulating post-exploitation activities, where
there is more value in practicing detection and response.

With remote access achieved, the Red Team conducted various

attack techniques locally on the laptop to steal web browser cookies
and impersonate their active sessions.
From there, we pursued further objectives similar to those of our
emulated adversary.

These techniques triggered an alert from our SIEM system. This

created an incident with our SIRT team, who immediately took action
to contain and investigate the incident.

Select members of security leadership were included as trusted

participants in the operation. We were all closely monitoring the
investigation from a Slack room set up for this purpose. This allowed
the SIRT engineers to experience responding to a very realistic
attack while preventing the incident from escalating too far.

At some point during the investigation, it was revealed that the

attacker was in fact the Red Team. SIRT had performed a thorough
investigation, collaborating across the team to trace the attack back
to our initial access vector.

This operation helped us validate some existing detection

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 10 of 11
:
 capabilities, recommend improvements for more, and give the team
a chance to work together to solve an interesting challenge in a safe
and controlled environment. This type of experience only comes
from conducting attack operations in stealth, which is exactly why
we have an internal Red Team at GitLab.

What we learned
At GitLab, we believe that performing Red Team operations in stealth
provides the most realistic opportunity to practice detecting and
responding to real-life attacks.

We also realize that every organization is different, and your security

evolution may follow a different path.

We learned that having a plan defined early on and shared

transparently across the organization was key to success.
Here are the things that helped us the most:

Defining a maturity model and using it as a roadmap.

Committing to broad goals defined in GitLab epics, and breaking
them down into manageable tasks inside GitLab issues.
Thoroughly documenting processes in our handbook and in
GitLab issue templates.

We would love to hear your thoughts on Red Teaming and how

you've managed your own security evolution. If there are any specific
topics you'd like our team to write about in the future, please let us
know. Feel free to comment below or to open issues or merge
requests in any of our public projects.

https://about.gitlab.com/blog/2023/11/20/stealth-operations-the-evolution-of-gitlabs-red-team/ 11/21/23, 8 48 PM
Page 11 of 11
: