OpenTIDE

Threat Informed Detection Modelling and Engineering as-Code

1 Foreword by Anton Chuvakin

Detection engineering is a critical yet often overlooked aspect of cybersecurity. It is the process of
developing, maintaining and retiring content and systems that can detect malicious activity. While
detection engineering can be a complex and challenging process, it is essential for any organization that
wants to protect itself from the top-tier cyber threats .

This work is relentlessly challenging. Threats evolve faster than many teams can adapt. Mountains of data
obscure the faint signals of intrusion. And skilled engineers, those who understand the intricacies of both
security and analysis, are in desperately short supply.

One challenge is the ever-changing nature of cyber threats. New threats are constantly emerging, and it
can be difficult to keep up with the latest developments. Another challenge is the volume of data that
needs to be analyzed. Security teams are often faced with terabytes of data, and it can be difficult to
identify the data that is relevant to threats. A third challenge is the lack of skilled detection engineers.
There is a shortage of qualified detection engineers, and this can make it difficult for organizations to find
the talent they need.

Despite the challenges, detection engineering is a critical process that can help organizations protect
themselves from a wide range of cyber threats. By investing in detection engineering, organizations can
reduce the damage and detect faster and better

Cybersecurity is a war of attrition, and the odds often favor our adversaries. Yet, in detection engineering,
we find the potential to shift the balance. It's the unglamorous work of building the digital tripwires that
alert us to danger – the unsexy but absolutely critical act of knowing when we're under attack.

This paper provides a comprehensive overview of detection engineering. It covers the key concepts of
detection engineering, the challenges associated with detection engineering, and the best practices for
developing and maintaining effective detection systems.
Let this paper be the start of your journey to transform detection from a reactive scramble into a proactive
strength. Because in this digital arms race, the best way to prevent tomorrow's breach is to detect the
signs of it today.

2 The Missing Link in Detection Engineering: DetectionOps

2.1 Origin of Detection Engineering

Detection Engineering is the dedicated discipline to create data systems and analytics content in order
to detect Cyber Threats. It is a recent concept – only a few years ago, the vast majority of Security
Operations Center were multi-tiered, with new creation detection assigned to the more senior
personnel, and mostly alongside other duties. In recent years, a transition has been done to focus on
streamlining SOC as a service organization, where Detection Engineering is, increasingly, a key capability.

Modern DE teams face several challenges, but through several interviews with peers to exchange on
SOC management practices, one which comes back frequently is the lack of common actionable
frameworks to build from. This leads, according to recurrent feedback, having to spend effort and deal
with the complexity of building in-house frameworks, with varying level of investment due to the effort
required, and often the immediate need to address key detection requirements. In general, a lot of
effort of DEs are spent on meta-tasks before starting analysis: writing knowledge articles, aligning
stakeholders, gathering intelligence from different sources and parsing it, etc. Furthermore, throughout
scaling, knowledge gets spread across different systems and people, is stored in different forms, and
overall makes it increasingly harder to get the data under control.

The European Commission CSOC, through internal reorganization, defined a new service vertical
dedicated to Threat Hunting and Detection Engineering, with emphasis on being Threat-Driven. The new
team, CATCH (Cyber Analytics, Trending, Correlations and Hunting), was in the right environment to
develop a top-down framework, leveraging as-code and DevOps principles, expanding automation
capabilities and emphasizing threat and detection modelling. This framework and platform grew to be
OpenTIDE, and after 2 years of continuous adoption and further development, we now propose this
framework to the open SOC and Cyber communities as the current only open blueprint to structure
Detection Engineering and create a consistent system of exchangeable and actionable data.

2.2 DetectionOps
OpenTIDE is best thought of as a Detection Operations engine. It contains a data schema, automations,
concepts and principles that aim to be deployed in a detection engineering team as a central platform.
The core idea of the framework, is that a Detection Engineering team can express all its artifacts as-
code, following common, extensible data schemas, in a continuous flow from threat to detection –
defining the steps to create a detection, and avoiding manual overhead.

Similar to how DevOps teams leverage Infrastructure-as-Code to manage complex deployments,

Detection Engineering team adopted Detection-as-Code. OpenTIDE goes a step beyond, and has an
everything-as-code approach. The key philosophy, thus, being that all the deliverables and workflow in
 on den al i ited

DE can be codified into structured YAML (a well-known and human-first alternative to JSON) files called
models, increasing automation opportunities, and exposing actionable data, such as more immediate
coverage metrics. This loop, from threat analysis to detection release, and new intelligence ingestion, is
DetectionOps.

2.3 Existing frameworks that could support DE practices

For DE, there is considerable open-source information in the form of blog posts, articles, and social
media exchanges which often lead to inspiration. Other structured approaches often either tend to
study or reverse engineer the repositories of a particular vendor’s rules, or Sigma rules. This approach is
what OpenTIDE considers to be bottom-up, which is starting from technical capabilities instead of
threats. Whilst initially more efficient, during the development of the framework we have found
bottom-up to be less scalable past the initial few dozen detections, as there is less knowledge and
visibility over relevant, technical threats.

A more top-down approach is MITRE ATT&CK, which is the largest and most useful taxonomy of known
cyber threats. Whilst MITRE ATT&CK provides lots of insights on threats, it also normalizes discrete
procedures into more generalist (sub-)techniques. For DE, the procedural level, or an even more
technical intel level is often required and ATT&CK can provide a general direction but can also fall short
of the more granular technical needs of a DE team.

Thus, for DE purposes, existing frameworks of methodologies are often insufficient, and engineers
supplement this gap with deep dives into particular TTPs or by creating accompanying frameworks like
the Atomic Red Team tool chain. OpenTIDE makes this process structured, as intel can be broken into
suitably-sized chuncks and ingested as threat vector models, indexed and metadata rich. Still connected
to ATT&CK, OpenTIDE aims at being a bridge between DE needs and other forms of intelligence, or
taxonomies which are initially limiting. ayer of abstraction coming from existing frameworks and
directly provide DE with a low-level, inter-related and actionable technical data format.

Framework Summary Strengths Weaknesses Fit for DE teams

MITRE ATT&CK MITRE Active community 3-6 month Globally the framework
ATT&CK® is a release cycle, contributed hugely to
globally- behind latest improving
 accessible High quality threats to allow conversations as the
knowledge content. time to shared vocabulary
base of normalize as helped everyone.
adversary Comprehensive techniques. Useful as a reference to
tactics and overview of real Techniques map existing detection
techniques world threat abstract related content as a rough
based on real- landscape. implementation measurement.
world s/ Can give directions in
observations. Innovative at the procedures, but what areas should be
The ATT&CK abstraction layer, they are high improved in detection
knowledge advanced infosec level and often coverage
base is used as as a whole in and difficult to
a foundation by itself. action as DE
for the teams need
development understanding
of specific on a lower level
threat models of granularity,
and even if higher
methodologies level detections
in the private are eventually
sector, in (also) built.
government,
and in the
cybersecurity
product and
service
community
CAPEC CAPEC, short Offers a technical Less pleasant to Not targeted to DE
for Common knowledge base read and parse usage more or less at
Attack Pattern which helps than OWASP or all.
Enumerations understanding ATT&CK
and exploitation resources.
Classifications, paths. Limited usability
is a MITRE Particularly useful for detections,
project which as a framework application
goal is to when looking at exploitation
provide a application threat focus only.
dictionary of modelling. Lack of
exploitation Good at description for
techniques describing non-exploitation
related to mitigation related
application strategy. adversarial
exploitation. It activities, hence
is often used in why ATT&CK
other takes over with
information the technique
sharing concept.
frameworks, Lack OS internal
such as with exploitation
STIX in its descriptions
Attack Pattern (lolbins,
Object, to
 standardize the WindowsInterna
way attack ls...).
events are
described –
especially
around CVE
exploitation,
and the earlier
part of the
malware
lifecycle when
it is first
delivered
MITRE D3FEND D3FEND is a MITRE-driven. Is an early-stage D3FEND maps to
knowledge Solid project project focusing MITRE ATT&CK and
base, but more start, big on a level of offers a
specifically a community. Maps abstraction for glossary/dictionary of
knowledge to ATT&CK detection types terms that describes a
graph, of through within the knowledge graph of
cybersecurity inferences against detect tactic. No countermeasures,
countermeasur a large artifact automation. including detection
e techniques. ontology Lack of high level techniques,
In the simplest structured way which is the tactic
sense, it is a to expand ‘Detect’. The non-
catalogue of abstracted Detect elements of
defensive detection types D3FEND are not of
cybersecurity into required interest.
techniques and detections. Not
their comprehensive The strengths and
relationships to within detection weaknesses identified
offensive/adve types/categories are consistent with the
rsary , as example no analysis from the
techniques. cloud native MITRE D3FEND white
The primary capabilities. Not paper(1)
goal of the driven by
initial D3FEND detection
release is to modelling or
help detection
standardize the engineering
vocabulary requirements.
used to Driven by a
describe need to test
defensive vendor solutions
cybersecurity for coverage.
technology
functionality.
D3FEND is an
early-stage
initiative and
needs

(1 https://d3fend.mitre.org/resources/D3FEND.pdf)
 community
input to grow

DeTTECT DeTTECT was YAML format. No concept of Detect maps a

built as a detection vocabulary of data
focused short- GitOps process to objectives or a sources to MITRE
term project to maintain. structured ATT&CK and allows a
show approach for sort of ‘visibility ap’
Rabobank Some defining where you can see how
management automation, links detection many MITRE ATT&CK
an attempt at between threat objectives. techniques you have
defining actors and data sources for, and
detection detections Lack of which data sources
coverage and is through ATT&CK automation map to how many
at least mapping, with engine and ATT&CK techniques. It
partially data sources detection-as- also allows the building
successful at attempted code of threat actor heat
giving an integrated. deployment. maps based on
attempt at a techniques mapped to
mapping and Actively Data source threat actors (used by)
knowledge expanded and integration only then visualized using
graph. updated. partially the ATT&CK navigator
completed. layer.
The DeTTECT
framework DeTTECT also adds
adds together some capabilities to
threat actors map meta data for
with ATT&CK detection rules:
techniques. It 1. Type of
also adds data system
sources and applied to
detection 2. Where
rules, putting it detection
together in a resides
manageable 3. Comment
way through a 4. Date added
repository of Detection score
YAML files. following
https://github.com/rab
obank-
cdc/DeTTECT/wiki/Det
ection-scoring
Atomic Red Team A library of Granularity of The atomic red team is
simple, focused tests. a collection of tests
tests mapped created to enable
to the MITRE Maps against detection testing of
ATT&CK® MITRE ATT&CK. granular detections and
matrix. Each is mapped back against
test runs in five One of the first MITRE ATT&CK.
minutes or ever projects
less, and many launched to
 tests come support the
with easy-to- needs of
use detection
configuration engineering at
and clean-up levels of
commands (2). granularity below
MITRE ATT&CK
procedures.

Maps to domains.

Active community
and project
evolution.

Can integrate to
TIDEMEC.

Each atomic test

can test a specific
detection.

Actual innovation
within the space
of cyber security.

Drives the need

to understand the
challenge of
reaching
detection
coverage.
STIX The STIX First-ever Threat focused, No SDOs on making
framework attempt to define not detection supporting Detection
builds out shared focused. No Engineering, limited
shared vocabularies and attack path SDOs on making CTI
vocabularies data modelling actionable for DE
and data models/format aspect. Threats teams
models/format for describing CTI are not broken
s for describing data in a structed down into the
anything that is global way. SDOs level of
recognized as a cover multiple granularity
STIX Data categories of required for
Object. STIX data. Has useful detection
achieves elements for modelling and
harmonization detection engineering.
of CTI efforts engineering Limited
between teams efforts, such as applicability for
on a global artifacts including detection
scale. STIX windows services

(2https://d3fend.mitre.org/resources/D3FEND.pdf )
 enables CTI used or created, modelling and
teams to track malware artifacts, engineering.
threat actors and similar
and their work artifacts that can
in a mostly inform detection
structured way modelling and
focused from engineering
the threat efforts. Great
perspective. documentation of
threat actors,
intrusion sets and
campaigns and
meta data around
these.
Relationships can
be modelled
between SDOs.

MaGMa Management is MAGMA is Limitations in MAGMA was (In the

done using an supportive for the framework humble opinion of the
Excel detection use model slow authors) the leading
spreadsheet. case down and framework for DE work
Detection development. prevent a prior to OpenTIDE.
objectives (DOs The value mainly proper
– Level 3) are arises from the TOP/down
defined in one application of a approach to
sentence (one structured detection
cell). framework. modelling, often
DOs are linked MAGMA resulting in
to ATT&CK implements a shortcuts taken
(sub)technique detection to go directly to
s. (1::1 in lifecycle view. detection rules
MAGMA) MAGMA adds a 3- (below L3).
There is no tier structure, Going directly to
attempt to link where for the detection rules
to MITRE first-time makes the work
ATT&CK detection non-repeatable.
‘Procedures’. objectives are Following
The procedures mapped to MAGMA is
are not a business intensive
formal spec or objectives, drives manual work
consistently and stakeholders. and very
applied in a MAGMA also difficult to keep
structured adds the first updated.
way, although elements of MAGMA
the granularity detection definitely works
of ‘procedures’ lifecycle as a decent
approaches management and initial starting
usability for a structured point for a
detection maturity CSOC, but the
modelling and approach using value is limited
engineering CMMI. MAGMA for a scalable
 more than the adds the first- detection
usability of sub ever attempt at coverage
techniques. creating a maturity effort.
There are no structured
precise detection
guidelines to coverage
extend framework via L1
MAGMA Level models mapped
2 Use Cases to L2 and then to
(over-arching L3.
categories
which drive
Level 3 ones)
once
intelligence
outgrow the
initial library.

Palantir ADS An ADS Strong focus on Fully depends Strong and capable
framework is a enabling 1st level on ATT&CK to focus on providing first
set of responder prioritize level analysts the
documentation actions/determin coverage gaps. information/knowledge
templates, ation. needed to make alerts
processes, and No focus on actionable. We’re
conventions Includes alert making CTI working on either
concerning the testing actionable. eventually bringing this
design, (validation). level of capabilities into
implementatio Not created to OpenTIDE or
n, and roll-out Analyzes blind structure DE documenting how we
of ADS. Prior to spots and false processes or achieve the
agreeing on positive support with actionability for first
such a expectations. automation. level responders using
framework, we other processes and
faced major tooling.
challenges with
the
implementatio
n of alerting
strategies: The
lack of rigor,
documentation
, peer-review,
and an overall
quality bar
allowed the
deployment of
low-quality
alerts to
production
systems. This
frequently led
to alerting
 apathy or
additional
engineering
time and
resources to fix
and update
alerts.
SpectreOps blog series ‘On The blog series Argues that No A must for anyone
detection’ introduces MITRE ATT&CK implementation doing CTI, red teaming,
(mostly, we lacks granularity follows/accomp purple teaming, blue
guess partially (possibly only anies the teaming and of course
supports also) indirectly) for DE structured detection engineering
arguments purposes. thinking.
around the
complexities of First-ever
the science of attempt to
detection describe the
engineering levels of
and the need complexity that
for granularity are not handled
of threat by any framework
descriptions today. Only the
atomic red team
framework
reaches the
functional level
similar to the blog
series. Defines 2
levels below the
procedural, which
drives detection
coverage
discussion
progress and
enables future
detection
coverage
maturity efforts.

Argues that
visualizing threats
and how
they/systems
work is an
advantage

In general just
fantastic reading
Indicators of Behavior -
OCA
(opencybersecurityallian
ce.org)
2.3.1 Part-conclusion on ontologies/topologies and detection coverage framework topology
All of the listed frameworks aim to reduce the complexity of the global threat landscape in order to
inform the efforts of cyber security teams. There is a need to improve the capabilities of defensive
teams, or attackers will forever remain at an advantage (3). ATT&CK contributed the first-ever structured
effort to reduce the complexity of the threat landscape with a 4-tiered pyramidical structure, and
D3FEND adds countermeasures to that including detection types/categories.

DeTTECT adds the first attempt at mapping data sources into a graph containing detection
implementations and threat actors.

Atomic red team adds unit tests for detection implementations at the functional level.

The SpectreOps blog series adds 2 more levels to the MITRE ATT&CK pyramid, the levels of functional
and literal. It also attempts to provide argumentation for where and how to implement detections at the
right level of granularity of detection modelling and engineering.

MAGMA adds a 3-tier structure, where for the first time detection objectives are mapped to business
objectives, drives and stakeholders. MAGMA also adds the first elements of detection lifecycle
management and a structured maturity approach using CMMI (4). MAGMA adds meta data to measure
detection coverage, effectiveness, weight, and potential improvement value. Additionally, MAGMA adds
the first-ever attempt at creating a structured detection coverage framework via L1 models mapped to
L2,then to L3 with L4 being the detection rule layer.

Of all the frameworks listed and evaluated, MAGMA is by far the most capable for a detection
engineering team, but still falls short of the target.

2.3.1.1 Other research ingested and analysed

The following papers were reviewed:

1. Architecting threat hunting system based on the DODAF framework (5)

a. Builds a functional threat hunting framework that maps to MITRE ATT&CK and D3FEND
based on DODAF, not detection modelling focused.
2. Toward a Knowledge Graph of Cybersecurity Countermeasures (6)
a. The MITRE D3FEND paper explaining D3FEND and its purpose as a shared vocabulary
and knowledge graph. Content is similar to that on D3FEND pages but expressed as a
research paper and explaining background and research done.

(3)https://github.com/JohnLaTwC/Shared/blob/master/Defenders%20think%20in%20lists.%20Attackers%20think%
20in%20graphs.%20As%20long%20as%20this%20is%20true%2C%20attackers%20win.md
(4)https://www.betaalvereniging.nl/wp-content/uploads/FI-ISAC-Use-Case-Framework-Full-Documentation.pdf
(5)https://link.springer.com/content/pdf/10.1007/s11227-022-04808-6.pdf?pdf=button
(6) https://d3fend.mitre.org/resources/D3FEND.pdf
 3. A semantic approach to improving machine readability of a large-scale attack graph (7)
a. The paper cites previous research on modelling attack graphs and explains the need to
understand attack graphs and paths including attacker state transitions. Proposes an
ontology to make attack graphs understandable.
4. SANS 2022 ATT&CK® and D3FEND™ Report: Incorporating Frameworks
into Your Analysis and Intelligence (8)
a. The paper describes the value of D3FEND in informing blue teams about ways to
itigate threats by using case studies and specifically for the ‘detect’ part of D3fend
shows an exa ple of counting i ple ented detections per D3FEND ‘detect’ category
5. How To Improve Security Monitoring With Detection Engineering Program (9) by Oracle SCS
team

Important points from this article, which is very on-point for the challenges presented in this
white paper:

a. The three main components of the SCS program which drive success are 1) having a
formalized, detection creation process, 2) using a joint, cross-functional, security team,
and 3) using a shared repository to store detection information.
b. We manage the Oracle SaaS Detection Engineering as a joint team, within SCS. It
includes Security Operations, Incident Response, Threat Intelligence, Software
Engineering, technology-specific subject matter experts, and business representatives.
The joint team concept improves collaboration, common goal setting, and, most
importantly, exposes multiple teams to the process and gives them a way to add to the
threat detection requirements. The goal of the team is to develop and manage
technology-solution, agnostic, detection capabilities based on proactive threat research,
and reactively, based on security incidents.
c. For each use case, we track data points such as: MITRE ATT&CK alignment, Detection
technology (SIEM, EDR), Log sources, Functionality description, Deployment status,
Sigma rule
d. Input and Concept Phase
In this phase, which fits into the DevSecOps planning phase, we gather and prioritize use
case requests. Use cases can be inspired from many sources, such as internal research,
forensic investigations, or open source data sets which contain detection rules, such as
the Sigma GitHub repository owned by Florian Roth. We use the MaGMA Framework to
help provide control over the security monitoring process and align security monitoring
to business and compliance needs. In this phase, use cases describe follow-up actions
(incident response) and are tied with business drivers to show how security monitoring
reduces risk in the organization.

(7) https://link.springer.com/content/pdf/10.1007/s11227-018-2394-6.pdf?pdf=button
(8)https://fs.hubspotusercontent00.net/hubfs/2617658/Gated%20Assets/Q123%20SANS%20Mitre%20D3FEND%2
0and%20ATT%26CK%20Report.pdf
(9) https://blogs.oracle.com/cloudsecurity/post/how-to-improve-security-monitoring-with-detection-engineering-
program
 e. Deployment Phase
The third phase in our use case lifecycle encompasses the release and deploy phases
found in the DevSecOps framework. When a use case is deployed, the team adds it to
the Cyber Analytics Repository (CAR). Within CAR, we use either Sigma or YARA
languages, and we map all the content to the MITRE ATT&CK matrix and assess any
gaps in detection capability. The DeTT&CT framework is an open source project that
"aims to assist blue teams using ATT&CK to score and compare data log source quality,
visibility coverage, detection coverage, and threat actor behaviours" (GitHub, n.d.).
6. Capturing Detection Ideas to Improve Their Impact by Florian Roth (10)

Salient elements from this article, which is also very on-point for this white paper:

a. In times of an increased flood of information, it becomes increasingly important to

structure and automatically process that information. A textual description in the form
of a blog post requires the receiver of that information to process the data and turn it
into applicable information. Every professional processes that information differently.
Digital forensics analysts add the idea to their notepad or mind maps, and SOC analysts
create a query and add the result set to a dashboard. Network detection engineers write
a Snort rule and apply it to a node in the testing network. Wouldn’t it be great if we
could extract, store, and share that information in a generic and machine-readable form
so that ten thousand users of detection rule feeds would immediately benefit from the
once-derived infor ation? Wouldn’t it be cool if only a few analysts extracted and
edited a set of rules and made it available for everyone else?
b. Coverage: When we start sharing detection rules with other analysts through extensive
databases, we achieve a much higher overall coverage of threats compared to analysts
working on detections on their own. While a single analyst may miss many threat
reports and detection ideas, the group misses much less of them. By sharing and
i proving each other’s rules, we achieve a uch higher coverage of the whole threat
landscape.
c. Permanence: By transforming the detection idea into a rule, the core idea is stored in a
standardized form in a database of rules that grows over time. This way, even detection
ideas shared long ago are stored and kept for newcomers, be it an analyst of a SOC, an
organization that starts monitoring, or a tool developer that began to support one of the
rule for ats. They don’t have to search and process old reports or tweets any ore.
They can immediately benefit from the condensed knowledge of such a detection rule
database.
7. Building the Threat Detection Ecosystem at Brex (11)

Salient points: Detections should:

a. Be driven by a possible scenario of an external threat.

b. Focus on capturing attacker techniques rather than indicators.

(10) https://cyb3rops.medium.com/capturing-detection-ideas-to-improve-their-impact-311cf4e1c7a8
(11) https://medium.com/brexeng/building-the-threat-detection-ecosystem-at-brex-215e98b2f1bc
 c. Include a prioritization score from the risk of the system(s) and the likelihood of the
attack scenario.
d. Produce low alert noise and be tuned quickly if creating alert fatigue for the on-call
team.
e. Trigger automation workflows.
f. Be unit tested and regularly reviewed by the team.

3 From Loose Data to Structured Objects

OpenTIDE has developed the concept of DetectionOps as a practice to create as-code objects,
representing the entire Detection Engineering process, currently often flaky, in a repeatable and
codified workflow

3.1 Focusing work in a Detection Engineering Loop

During the development of the framework, we noticed that DE time is most often spent away from
detection related work, and takes on a more bureaucratic role, carefully documenting new intel and
defining future improvements. Much of this work is due because of a lack of a common way to express
and work with DE related data. It is also explained by the general difficulty a DE has to get the right data
– intel comes in many forms and not always the one that makes sense when creating detections (too
high level, too hyper specific, untechnical or not focusing on uncovered areas…), and this intel is
documented in whatever ends up being available to the DE, which is often an internal wiki, or notebook.

OpenTIDE’s approach is to break down the DE lifecycle as a threat-driven loop, and aims at delivering
artifacts at every milestone. Threats are analyzed into objects called Threat Vector Models, Detection
Objectives related to those TVMs as Cyber Detection Models, and detection rules fulfilling those
objectives as Managed Detection Rules. Doing so, a DE team is always assigned to meaningful R&D, with
a clear outcome, and less complexity in finding the right form of deliverable; instead, the framework
provides a straightforward path.

3.2 Centralized Data Repository

Key to its DetectionOps concept, OpenTIDE leverages Git and thus all objects are stored in a central
repository, alongside automation workflows. With version tracking, workflow safety with merge
reviews, visibility and collaboration, teams can use a refer and expand a single source of truth, powering
data driven decision making when creating new objects and allowing further analysis. This improves on
the current situation where DE teams often have to keep detection documentation in sync across
multiple places (doc pages, tickets, and detection platforms).

on den al i ited

3.3 Collaborative creation of DE-actionable detection backlog

Using the SOC-as-code principles, GitOps/GitFlow principles are applied to the detection modelling and
detection engineering disciplines, which contain the following phases in a lifecycle of a detection:

1. Data Ingestion
2. Backlog Prioritization
3. Object Creation
4. Object Review
5. Object Approval
6. Object Merging

In OpenTIDE, work is focused on creating artifacts supporting detection modelling by representing

threats and related detection objectives as Threat Vector Models (TVM)and Cyber Detection Models
(CDM). Due to the varying responsibilities of the different teams in any given organization, threat
identification and detection modelling could be implemented using a shared, distributed or centralized
responsibility model, which OPENTIDE natively supports with a central repository of data, and a Merge
Request process where different participants can contribute to in different workflows suiting the team
structure and composition best.

As an example implementation:

a. TVM creation: Stakeholder such as CTI, RT, DFIR

b. TVM review, approval and merging: Detection
c. CDM creation and approval: Involved stakeholder, or CTI centrally
d. CDM review and merging: Detection

4 Threat Intel Ingestion in a Threat Informed Pipeline

The optimal situation for a DE team is to have an up-to-date and accurate representation of its detection
capabilities against defined and prioritized threats. This is called detection coverage in the industry, and
a DE team aims at keeping up its coverage as new threats and actors emerge, adapting to changes to the
threat landscape.

However, the reality of DE team is that they often do not have a sufficient visibility of their detection
coverage, nor of the threat landscape they should cover. Improving situational awareness is complex, as
it requires keeping a tab on many offensive and defensive topics, and often DE team do not have the
procedures or tooling to take rapid efficient decisions. OpenTIDE embraces and developes the concept
of a prioritized detection backlog, where threats are now stored as objects, they can be linked to project
management procedures and prioritization based on the data they contain.

Measuring threat coverage against detection capabilities is key in OpenTIDE DetectionOps loop,
as it drives every prioritization procedure which happen in a top-down manner, starting from
documented threats and trickling down to detection work.

4.1 Backlog and Prioritization

OpenTIDE posits that the critical path to create new detections starts with a threat-driven approach,
powered by ingesting intelligence sources from the start of the process, alongside internal requests
from red teams, incident response and other CSOC stakeholders.

Prioritization is done by analyzing the incoming intelligence, weighing the intelligence against known
organization key targets and vulnerabilities, and comparing against the current detection coverage. The
goal of a threat backlog is to efficiently and continuously assign work capacity to the most critical
detections to engineer or refine, in other words that will have the most impact to the detection
coverage.

4.2 Threat vectors and determining vector atomicity

OPENTIDEOpenTIDE central object to describe threats in the TVM : Threat Vector Model. A Threat
Vector being simply a representation of a threat, which could sit at different level of atomicity, with an
inclination for lower level, technical descriptions. A TVM could be describing an adversarial procedure, a
TTP, a malicious tool execution and is generalized enough to flexibly adapt to new unseen threats.
Unlike an ATT&CK Technique, or other taxonomies, it concerns itself less with trying to group related
procedure into common concepts, but rather to be able to ingest and describe new intel, breaking down
complex threats into smaller, digestible elements that support DE.

4.3 Threat Vector Chaining

OpenTIDEOpenTIDEDuring the adoption of OpenTIDE, the team grew its practice in modelling incoming
threats as objects. It was noticed that for complex threats, the different subtleties and variance in how
threats could be performed by an adversary led to challenges, where it was necessary to either create a
single, very large TVMs, or many highly specific ones, which were hard to write and maintain. The issue
was narrowed down to the fact that the TVM was a very “vertical” concept, it had to be completely self-
encapsulating; in other words a TVM had to be a standalone representation of a threat vector. To break
this boundary, the concept of chaining was introduced, where TVMs could be related to one another in
a more lateral manner, and allow to represent attack paths much more efficiently. Threat vector
concepts can be related more powerfully, and allow to represent their full context by describing
precisely how they interact.

Threat vector Context: Analysis of a chain of event an adversary took to execute malicious commands
over SMB.

Chaining allows to have clearly defined and reusable concepts, but still reconstruct a realistic attack path
without ending up with a very complex TVM, which would be hard to consume and effectively only be
useful once.

Threat vector Context: A legitimate DLL (zipfldr.dll) executed by rundll32 to perform malicious
operation.

Without chaining, similar TVMs would have to be created per exploitable DLL. If the attacker uses
something different than rundll32, the entire exercise has to be redone to be either generalized or
duplicated for the new executable. With chaining, the key concepts can be extracted then related.

4.4 Identifying detection objectives & complex detection objectives

The Cyber Detection Model (CDM) object type is at the root a detection idea aka objectives (“What to
detect?”), accompanied with various pieces of information to help research the idea and implement it
plus various meta data. Not all detection ideas will work for everyone and not all detection ideas will be
implantable as functioning detections. Even so, documenting detection ideas as CDM detection
objectives is inherently an advantage as the universe of detection ideas has natural limitations and since
an idea that may not work for one entity may indeed work for another.

Coupled with the concept of vector chaining you inherently also intuitively reach the point of combining
various chained threat vectors into more complex detection ideas. Coupling this one step further with
parallel threat vectors, you end up with detection objectives that can look something like a AND b and
NOT c but with d OR e OR f.

4.5 OpenTIDE Modularity

OpenTIDE is extensible from a data model perspective. New models can be created and connected to
other OpenTIDE objects, keeping a consistent data model graph and allowing the tool to support DE
processes further. Existing model types can be supplemented with new fields, vocabularies can be
added or extended. This architecture was developed to allow sufficient fluidity in the future, as DE
practices mature and the required metadata grows.

4.6 Data Driven Detection Engineering

When the data to understand what good detection coverage means becomes available as the number of
TLP:CLEAR models in OpenTIDE increases, this could contribute towards increased transparency of
detection capabilities in the market (squeezing a market of lemons?12). Sharing data makes data
available for everyone to consume, which means that the foundation for the possibility for the
community to iterate on the work of each other is created. Reducing the level of complexity required to
do detection engineering, making the work more repeatable, and enabling the mapping of detection
coverage at a more atomic level all three support a data driven DE process where the outcome improves
defender capabilities in detecting adversary activity.

In the optimal situation, all the threat vector data already available globally is catalogued and associated
with detection objectives, turning DE efforts into a simpler process of weighing threat vectors and attack
paths against each other and focusing DE efforts where the most value to be gained is.

Threat Graph approach to support DE practicesIn the last few years, researchers in the global infosec
community13 have attempted to move away from the thinking in ‘lists’ described by John Lambert of
Microsoft in his seminal post14. OpenTIDE enables defenders to think in graphs , and with OpenTIDE,
each object type need only be described as a TLP: CLEAR object once for the entire global community to
have access to it.

OpenTIDE enables researchers and vendors, to share their work (public or private) in a common,
normalized format which is directly actionable by receiving parties. For example, an intelligence report
describing new threats by an actor for example, could be broken down in TVMs, detection ideas as

12
https://www.youtube.com/watch?v=I0pbNlzZQtQ
13
As example https://center-for-threat-informed-defense.github.io/attack-flow/introduction/
(14)
https://github.com/JohnLaTwC/Shared/blob/master/Defenders%20think%20in%20lists.%20Attackers%20t
hink%20in%20graphs.%20As%20long%20as%20this%20is%20true%2C%20attackers%20win.md
linked CDMs, and some example rules are MDRs. This entire package can be ingested, reviewed and
prioritized without further work on the data itself.

Public repository of detection content could not only be directly deployable detection-as-code by
adopting the MDR format, but also fully mapped to threat data and detection objectives, providing rich
context and allowing DE team to expand into new objects, relevant for their organization.

5 Detection Deployment

OpenTIDE methodology and processes are meant to guide Detection Engineering, and once the relevant
objects have been created, there should be clear detection objectives, alongside well defined threats.
The last key object of the framework are Managed Detection Rules, or MDR, which are detection-as-
code objects containing the detection rule configurations.

5.1 Detection-as-Code Strategy

There are different detection-as-code schema published, such as Microsoft Sentinel, Elastic, or Splunk.
Generally, these are wrapping the platfor ’s API endpoint, and are used to publish detection content in
a public repository.

Due to this, there is little to no interoperability between platforms. This may still be functional for teams
with a single central SIEM product, but does not work as efficiently when there are many tools (IDS,
NDR, EDR, Cloud Detection, Container Security, Mobile Agents, Email Gateways Rules etc.). Most
detection team create an in-house schema, and there are no known standardization for detection-as-
code rules. Sigma is the closest equivalent, and even normalizes to the actual content of the query,
which can then be translated by different backends into actual queries. It has limitation both in its cap
abilities, since it needs to normalize the features it exposes across many platforms, so complex rules
with platfor specific techniques often can’t be used, and it cannot exploit the full potential of
platforms feature. While very efficient to share rules to the community, especially for well-defined
environment like windows, it can not fully scale to DE teams need and is as such rarely operationalized.

OpenTIDE strategy is to provide a common schema for the basic data, alongside platform-level
subschema. A single MDR can contain the detection rule for a single platform, or many. This is especially
useful when the same detection rule is implemented on different SIEMs, or for example when part of
the environment's endpoints is monitored by the SIEM, and another covered by EDR as it’s often the
case in large organizations. A layer of processing also takes place, to create an easier developer
experience – for example, instead of having to write a cron expression, allow the developer to have a
simple time expression ( 3h ) and do the conversion in the backend.

Each of those subschemas contain rich configurations options, so the full platform features can be used.
Queries are written in the natural language for the platform.
5.2 Complex Platform Deployments
We emphasize functionality and expressivity in the MDR Deployer Schemas – platforms are
implemented with the full capability of what their API exposes; codifying different features and
configuration combination into conditional schema where required, or highly validated and if possible
autocompleted. On open-ended platforms such as Splunk, where there can be a large number of
actions, we take a progressive approach – we fully implement the base functionality displayed in the
Saved Search editing GUI, adding the popular notable and risk framework actions on top. Actions can be
further conditionally enabled or disabled at the configuration file level, and even be conditionally
enabled based on the status of the MDR.

Status based configuration allow advanced users to override parameters based on the status of the MDR
(more precisely, configuration status as MDR can have multiple configuration – that will trigger multiple
deployers). This is particularly useful for enabling staging-like deployment : in SecOps, there is rarely a
concept of Staging for detection toolings. Staging SIEM for example are often technically impossible as
the data backend and detection engines are often coupled, or would require data sampling into another
instance which would make it unusable for detection engineering. Similarly, most teams have one EDR
installation, one NDR/IDS console etc. Detection-as-code is missing a staging deployment concept, and
by introducing conditional staging parameters, we can enable new types of workflow – for example, by
changing the severity, disabling notable events outside of the PRODUCTION status etc.

By creating tailored system schemas, and tying them with a backend deployment logic, we can expose to
Detection Engineers highly expressive and productive ways to build detections, while ensuring continuous
validation.

5.3 MDR modular architecture

The data architecture of the MDR object was done in such a way that new platforms can be added, by
creating a new sub-schema, a new deployment backend and a configuration entry. The platform
identifier is used to correctly find the right subschema, deployer at deployment time.
There is a rich variance in how platforms expose their APIs. For some of them, they provide a very well
abstracted SDK. Some others, provide minimal bindings with a SDK, and others only expose their API. To
avoid hardcoding many configuration idiosyncrasies in the deployment backend, and to allow developer
to have a nicer experience decoupled from the API request layout, several utilities work together to
fetch the correct API name for a given parameter, with support for more complicated structures (for
example nested dictionaries). In this manner, the schema can look friendly, and expressive, and the
backend work with the subschema to reconstruct the correct API request.

5.4 Staging and Production Concepts

During research, it was identified that many in-house detection-as-code solutions did not have sufficient
support for non-production detection rules. OpenTIDE contains in its tooling and systems a concrete
Staging and Production understanding : for MDR objects, every system they contain can be assigned a
status. Certain statuses are considered Production-grade, while the others are considered Staging.

During a Merge Request, the pipeline will identify the MDRs to deploy, and then check whether these
statuses are Staging-grade – if so, it will proceed. If not, then it will block deployment. Once merged on
main, and automated job will promote the MDR and assign it a default Production-grade status before
deploying it to production.

This approach allows a Detection Engineering team to fully work as code, including during the earlier
stages of development, and reduce the chances of disrupting production – several manual steps would
have been needed, whereas a bad commit to the wrong file in a Merge Request will have been blocked
unless the status was downgraded.

Doing so also avoids the confusion with Staging and Production concepts in IT compared to SecOps – we
rarely have spare SIEM instances considered Staging – instead we deploy our Staging or Production rules
to the same production instance – Staging is here considered under the Incident Response lens, i.e.
alerts we can ignore.

5.5 Deployment Plan Identification

There are roughly 3 different strategies to approach GitOps within a detection-as-code architecture:

1. Pull : The platform is responsible for fetching the current state of the repository, then rectifying
its current state against the target. This is a pattern used by Kubernetes for example, with the
Git Operator.
2. Push (Full) : Every commit to a dedicated branch (for example, main) triggers a complete re-
deployment or re-alignment from a CI/CD pipeline automation, the state of the branch is the
state of the platform. Different branches can represent different environments.
3. Push (Incremental) : Every commit to a branch trigger a localized deployment. This is adequate
when units of deployment can be compartmentalized, with little to no interaction across them.
This strategy allows for smaller deployment units across various branches. This is the approach
that OpenTIDE uses.

Detection Engineering teams leverage SIEM, EDR, NDR, XDR, IDS and Raw Telemetry Database Engines.
The data load on these systems is so high, that a staging/development instance would not make sense –
it would not be able to replicate the data, and most of the tools do not allow a decoupling of the data
backend and the components that would perform the threat detections. As a result, SecOps team do not
have two separate SIEM (or other tools) instances that are functional for detection testing. They develop
and “release” production-grade detection on the same instance most of the time. A few SIEM support
forms of data replay, or sampling, but during detection development it is strongly beneficial to have
access to as much data as possible, so these features are rarely every used in the context of Detection
Engineering.

This means that the target or Staging and Production deployments, in the context of detection-as-code,
is actually the same infrastructure; we deploy Staging and Production rules onto the same SIEM
instance, EDR, etc. This creates a conflict with the Trunk-based develop ent philosophy, as we can’t
deploy the entire branch into a staging or production environment without conflicting with changes
being done parallelly in many Merge Requests – we need to only deploy the MDR that has been created
or modified, without impacting other ongoing development, or overwriting data that is on the main
branch. To resolve this technical challenge, our solution was to allow every commit on any MR to trigger
a CI/CD pipeline, which would contain a job to calculate a git diff, filter it to isolate the MDR that can be
deployed, and then pass them onto the deployment engines. The git diff is performed differently on
main, and on feature branches – on main, the calculation is between the commit the pipeline is running
on and the previous. On feature branches, the root of the branch is found by iterating on every commit
until we find two parents – which is the start of the branch. This allows developers to keep pushing
changes to their branch and MR, and the deployment to deploy the entire change contents with all
cumulated changes.

6 OpenTIDE Technical Data Architecture

6.1 Metaschemas
A specialized YAML format has been developed to allow targeting a dynamic generation of JSON
Schemas and YAML templates from a single source of truth. The structure of the metaschema format
strictly follows the JSON Schema syntax (json-schema.org), and adds supplementary namespaced keys
that are used in the target schemas generation, to support automating certain data transformation
(such as fetching a list of possible values from other files, named vocabularies, or importing smaller
common schemas called definitions) or formatting (for example, hiding keys in the template to avoid
clutter but keeping them in the schema).

Metaschema build on top of JSON Schema with custom tooling and functionality

To further support the MDR plugin architecture, each system is represented by a standalone subschema
file, which gets dynamically inserted into the final JSON Schema if the system is enabled at the instance
configuration level.

System Subschemas encapsulate the data modelling part of detection rule deployment algorithms, and
make OpenTIDE modular. New supports can be built, and plugged into the framework.

6.2 Templates, JSON Schemas, Snippets

Metschemas are not useful by themselves – they are a higher order definition of the target schema, and
require a layer to transpiling and compilation. Automation translates the metaschema in a JSON
Schema, which is usable by automation and IDEs to provide validation and autocompletion, and also
compiles a template – a skeleton that can be directly used by a Detection Engineer, with rich formatting
( for example, if the key expects a list as a value, the template will represent it with an empty single item
list ), and auto-commenting keys which are not required. This template can be used by itself, but for
ease of use (see: 7.2 IDE Support), it is also turned into a VSCode snippet so DEs can access it directly
from their IDE.
 Ready to use templates

Schemas are the base of OpenTIDE – they define all the allowed data structure, and ensure a high
standard of consistency and quality

6.3 CI/CD Pipeline

OpenTIDE is enabled by strong automation, which is orchestrated using a CI/CD pipeline. All automation
modules are ultimately triggered by the pipeline, which contains the higher-order logic to know in which
condition the module should be triggered, what environment variable it has access to etc. Every commit
creates a pipeline, but depending on whether the pipeline is triggered from a commit to main, or from a
merge request, different behaviours will be triggered. Validation, Documentation, Deployment,
Framework Generation, Mutation are example of pipeline stages created by OpenTIDE. To simplify the
orchestration, and reduce compute times (as jobs can take a while to spin-up), scripts called
orchestrators are in charge of calling the relevant modules for their stage in the right order, but this
behaviour is ultimately open and modules can also be executed as a script.
6.4 Knowledge Graph
Because all objects can reference others in a structured manner, dictated by the schema for the object
in question, a directed relational graph emerges, with the edges being the references and the node the
objects themselves. The overall graph was initially designed to be vertical: threats reference actors and
are referenced by detection models, which are then referenced by detection rules. This initial hierarchy
was challenged when tackling complex modelling exercises, and the threat chaining concept emerged to
inter-relate threat vectors – this made the global data model much more horizontal, and opened the
door to an even richer knowledge graph. Future development on the platform will align on opening the
door to more relations between models.

6.5 Vocabularies
OpenTIDE objects often use lists of allowed values as part of
their schema. It would unmaintainable to hardcode those lists
in the schemas themselves, and would prevent reusability. To
solve this issue, dedicated YAML files with a specific schema,
called vocabularies, contain such data. Vocabularies support
several features used to enrich the schemas, for example by
assigning entries against a stage, and being able from the
metaschema to only point to certain stages. During the JSON
Schema generation, the algorithm dynamically seeks, retrieve,
and pivots vocabularies before inserting them in the final
schema.

6.6 Indexing Concepts

A single OpenTIDE instance can contain hundreds, or thousands of objects over time. With sharing, this
can become a larger pressure on the code side, as accessing files on disc can become increasingly slow.
Files in the instance would need to have hundreds of thousands of accesses to resolve different
dependencies (for example, finding the threat vector referenced by a detection model), and those
accesses would be very repetitive if they have to keep iterating the same files.

Indexing was introduced in OpenTIDE to resolve those performance issues, and facilitate the overall data
management and access when developing. The entire repository gets indexed only once, including
vocabularies, schemas etc. stored in the injected CoreTIDE repo, and is stored in an object in memory.
This object is optimized for access speed by using object IDs as keys, and other approaches.

A JSON representation can also be produced, and it is for example leveraged when creating the wiki on
Gitlab Wiki – the index is generated from the main repo, then injected into the wiki repo. Then,
automation picks up the JSON and generates the documentation without more involved steps.

The JSON Index allows complex data operations to be executed in seconds, even on very large
repositories

6.7 Configs
Configuration files are written in TOML, to differentiate more from the large YAML usage. They contain
variables commonly used across automation modules, and allow central management of features. There
is two layers of configuration, which are identical – core-side, and instance-side. When a setting is
modified on the instance-side, it takes precedence from the core-side, allowing to retain default
configurations on the core and to have full customization from the OpenTIDE instance. This setup is
especially useful when introducing new features and related configurations without having to update
manually on the instance-side. On the technical side, this is accomplished during the indexation time,
using a deep merge resolution. This is encapsulated, and can be called by automation modules who
need access to resolved configuration before indexation can occur.

OpenTIDE configuration files allow a large amount of customization and expressivity, pushing Detection-
as-Code further
Secrets are supported with a specific syntax, if a value starts with a dollar sign ($) a function will search
this key in the environment variables. When using CI systems, environment variables are used to inject
secrets in the CI run environment, allowing to manipulate secrets without hardcoding them in the
repository files.

7 Detection (and Platform) Developer Experience using OpenTIDE

OpenTIDE was developed under the lens of an active SOC environment, where not all Detection
contributors will have a background in software development and may struggle to adopt the key
concepts around Git versioning, file-based configurations, DevOps workflows, distributed development
strategy and other powerful, but often hard to adopt, as-code concepts. Thanks to a strong operational
feedback loop, the framework features were aimed at facilitating as-code adoption as much as possible,
where one DevOps champion is enough to support the rest of the team to migrate to the new way of
working, with tooling in the forefront to provide quality-of-life features.

7.1 Open Source Tooling

The platform inherits a strong open source DNA and mindset, and uses open tools and standard as much
as possible. Only a few dependencies are strictly required to leverage the platform efficiently : Visual
Studio Code, Git, any Git VSC like Gitlab, Github or BitBucket, a CI build pipeline, and a python execution
environment for the CI jobs. The CI workflow themselves may need tweaking to adapt to environment
specificities, and by default OpenTIDE comes with Gitlab support – but all the automation are python-
based and platform independent.

7.2 IDE Support

When the OpenTIDE instance repository is opened with VSCode, it will automatically recognize
important configurations allowing auto-completion and live validation, alongside prompting the user to
download the relevant extensions.

A crucial feature of OpenTIDE is the IDE experience: during the framework generation job, part of the
platform automation pipeline, templates are generated and turned into VSCode snippets. Developers
can quickly use those snippets to start working, while getting full file validation with JSON Schema,
alongside autocompletion. Greatly reducing onboarding time, auto completion and validation was
crucial for the platform to be adopted in production scenarios. Specific features developed allow to
search a vocabulary containing only IDs by their full name, for example when searching in the ATT&CK
vocabulary, or when trying to reference a model.
 Rich auto-completion and documentation

Self-documenting schemas

Fuzzy searches even for large vocabularies

7.3 Pipelines
When a Detection Engineer commits to their development branch in a Merge Request, or when the MR
is merged to main and creates a merge commit, an automation pipeline is triggered. DEs only need to
observe whether the pipeline passes – if validation jobs fail, they must review the log output which has
been developed to allow immediate mental parsing of what went wrong.
 Rich purposed designed logs to give Detection Engineers immediate feedback

Pipelines are used to support many other functionalities, and logs contain a lot of clear information over
what happened.

7.4 Automated Documentation

Another automation pipeline feature, documentation is self-generated from the OpenTIDE objects. This
completely removes the need to document content, which is an immense toll on Detection Engineers
when working without structured objects, and allows to focus on the detection R&D complexities. It also
provides a great knowledge base, also to incident responders who may want to view the context around
why an alert triggered.

The instance wiki, containing all the documentation, provides rich pages with links to related objects and
visualization of the knowledge graph, allowing to identify detection coverage gaps and better grasp the
current detection posture.
 Complex documentation exposes the rich internal graph-like data structure into powerful visualization

7.5 IndexTIDE, DataTide, PluginTide Interfaces

IndexTide is the interface to access data indexing measures (see: 6.6 Indexing Concepts), with DataTide
as the interface over the cached index. DataTide is a dataclass, its only purpose is to store data and it
doesn’t need to be initialized. Further nested dataclasses in DataTide allow developers to access many
aspects of the platform and the content in a natural way, for example to access a specific TVM data,
DataTide.Models.tvm[“TVM0123”] will return the object content. This approach facilitates the very large
amount of data accesses and manipulations required.

PluginTide is part of the plugin architecture used by OpenTIDE to address Detection-as-Code complexity.
Most DaC frameworks are hardcoded to only support a handful of platform, generally those used by the
SecOps team who developed it. OpenTIDE ambition is to be the de facto platform for DE teams, and as
such the DaC mechanisms were thought from the start to be modular. New deployment modules can be
added, and if a TOML configuration file, and subschema is present, the platform correlate it all and
correctly orchestrate the deployment. PluginTide is the interface exposing active plugins on the platform
- DeployTide.mdr[“splunk”].deploy(deployment=list_of_mdr_uuids_to_deploy) will for example trigger
the deployment routing for splunk across the selected set of MDRs.

DataTide exposes all the required hookups to the data of the OpenTIDE repo. When called, it will index
the repo only once and directly return the data on all subsequent calls.

The Plugin Architecture of OpenTIDE allows to import modules, initialize them, and expose them to the
codebase for more complex operations.

7.6 CoreTIDE and Core Injection

CoreTIDE is a repo containing all the automation and relevant schemas. It separates the content
management on OpenTIDE instances from the core platform development, which resolves many issues
that appear when clubbing the two together. During a pipeline, the instance, which only contains a basic
CI definition, clones CoreTIDE in the working directory. CoreTIDE has specialized measures to resolve
paths, and be able to access files in the directory above. In Gitlab, the basic CI definition in the instance
can then call another entrypoint file, itself stored in CoreTIDE. This entrypoint definition can in turn call
more definitions, allowing a single reference, but complex pipelines maintainable away from the
content.

Core Injection is very similar to git submodules, with the difference

that to a Detection Engineer, it is fully transparent: they don’t have to
deal with submodules in their IDE, can’t be confuse by inco ing
commits directed towards the submodule, and they can’t ake
manual mistakes or need to resolve anything with git, and keep a very
simple view on their repository. It is also very flexible : an opentide
instance can choose where and how the CoreTIDE repo is located, by
default StartTide references the public code.europa.eu CoreTIDE repo,
but a local one can be created to allow custom development.

7.7 Debugging Support

DebugEnvironment is a class which identifies if the run environment
can be considered a development environment, and if so different parts of the automation can respond
differently to the execution (for example, to print DEBUG log messages only in this state).
DebugEnvironment reads environment variables, to see if it is executing in VSCode, or if DEBUG is
present in environment variables.

In Deployment modules, a developer can then point to a test files, and keep rerunning the deployment
against them locally to check if everything works instead of having to trigger the pipeline by pushing to
the repository in a Merge Request. Secrets are then often required, and to avoid having to touch the
configuration files, or manually setting the environment variables, the Engines/modules/local_secret.py
file can be used to add those variables, and in DEBUG mode they will be injected. Missing environment
variables will be ignored when resolving the configuration files. local_secret.py is part of the .gitignore
file, so it won’t be pushed to the repo, but local secrets should always be handled with caution.

OpenTIDE custom tooling allows to take in account many specific of Detection-as-Code problematics
8 Collaborative detection engineering: OpenTIDE as a the first enabler
of Detection Engineering sharing

OpenTIDE artifacts can only be shared manually at the time of release. Pending later releases, or
community contributions conforming with the intended spec, it will become possible to automate the
sharing of DE artifacts.

8.1 Artifact sharing over MISP

An integration with MISP is possible, where your OpenTIDE instance would automate the upload the
relevant artifacts to MISP entities corresponding to the configuration settings on your own OpenTIDE
instance. Public artifacts to public MISP entities, more limited artifacts would go only to where these
would be allowed to go. From there, other SOCs would be able to consume OpenTIDE artifacts similar to
how they consume MISP today, except into a OpenTIDE instance.

8.2 Relation with STIX

The STIX data format normalizes CTI data which improves sharing of data in a knowledge sharing
community and enables easier integration of the data into tooling.

As of Stix 2.1, the objects that exist to capture data in STIX format do not support representation of CTI
data in a way that’s targeted towards consu ption by DE tea s. OpenTIDE allows CSOC teams to obtain
the benefits of data normalization, OpenTIDE objects could be translated as STIX SDO to construct a new
layer of DE friendly data, sitting between traditional CTI data and detections.

8.3 TIDeX
The sharing of OpenTIDE artifacts could also be automated by implementation of a dedicated OpenTIDE
exchange network of servers. Similar to the MISP implementation, the TLP assignment to models would
drive which artifacts an entity would exchange with another.

8.4 Sharing outcomes

OpenTIDE enables collaborative cross-CSOC detection engineering within either open or closed
knowledge sharing communities. Imagine 2 governmental institutions, where today they may both be
engaged in understanding 2 new threats, whereas using OpenTIDE they can divide the work and share
the results.