2022 REPORT

THE STATE OF SIEM

DETECTION RISK
Quantifying the gaps in
MITRE ATT&CK coverage
for production SIEMs

X X X X X
X X X X X
X X X X X
 Table of Contents
1.0 EXECUTIVE SUMMARY ..................................................................................................................... 3
METHODOLOGY .................................................................................................................................. 7
2.0 STATE OF DETECTION COVERAGE AS MEASURED BY MITRE ATT&CK ........................ 8
2.1 Coverage across all ATT&CK techniques ....................................................................................... 9
2.2 Coverage for top ATT&CK techniques used by adversaries in the wild ............................................. 9
2.3 % of broken/nonfunctioning SIEM rules ........................................................................................ 9
2.4 % of default vendor-supplied detection content that is disabled ....................................................... 10
2.5 Top log sources with no detections ............................................................................................................ 10
3.0 BEST PRACTICE RECOMMENDATIONS .......................................................................................... 11
3.1 Review current SIEM processes ................................................................................................................... 12
3.2 Become more intentional about how you develop
and manage detection content ................................................................................................................... 12
3.3 Build or refresh your use case management processes .......................................................................... 13
3.4 Measure and continuously improve ............................................................................................................ 13

4.0 HOW CARDINALOPS CAN HELP ................................................................................................................ 14

4.1 Visualize coverage based on MITRE ATT&CK heat map ......................................................................... 15
4.2 Recommend, test, and automatically deploy new rules to close
gaps in threat coverage ................................................................................................................................ 15
4.3 Identify, test, and automatically remediate broken or noisy rules ...................................................... 16
4.4 Enable risk-based prioritization based on organizational requirements ........................................... 16
4.5 Identify log sources not contributing to threat coverage ...................................................................... 17
4.6 Produce independent metrics for detection posture
and demonstrate continuous improvement ............................................................................................. 17
5.0 SCREENSHOT EXAMPLES ................................................................................................................................ 18
5.1 MITRE ATT&CK heat map ............................................................................................................................ 18
5.2 Recommended rules to close gaps in threat coverage ........................................................................... 19
5.3 Example of recommended best practice rule to close gaps in ATT&CK coverage ............................ 20
5.4 Description for recommended rule .............................................................................................................. 20
5.5 Example of broken rule with remediation ................................................................................................. 21
5.6 Log source types not contributing to detection coverage .................................................................... 21
5.7 Log source types not parsed correctly ....................................................................................................... 22
5.8 MITRE ATT&CK coverage score over time ................................................................................................ 22
6.0 RAPID, SECURE, API-DRIVEN DEPLOYMENT .................................................................................... 23
7.0 KEY INITIATIVES BENEFITING FROM DETECTION USE CASE MANAGEMENT ............. 24
7.1 Operationalize MITRE ATT&CK in your SIEM/XDR ................................................................................. 24
7.2 Securely accelerate cloud transformation initiatives .............................................................................. 25
7.3 Rapidly update detections for high-profile vulnerabilities and attacks ............................................. 25
7.4 Scale effectiveness of detection engineering teams by 10x ................................................................. 25
8.0 CUSTOMER EXAMPLES .................................................................................................................................... 26
ABOUT CARDINALOPS ..................................................................................................................................... 27
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0

EXECUTIVE SUMMARY
“Use cases are the core of security monitoring activities. A

“
structured process to identify, prioritize, implement, and maintain
use cases allows organizations to align monitoring efforts to
security strategy, choose the best solutions and maximize the
value obtained from security monitoring tools.”

- Dr. Anton Chuvakin

Blog post

SIEMs are foundational to the modern SOC, cloud monitoring tools, identity providers, etc.).
providing the essential role of helping security This data is then analyzed using predefined threat
teams rapidly detect and respond to cyberattacks detection rules and queries to identify suspicious or
before they can have a material impact on the unauthorized behavior.
business of the organization.
In this 2nd annual data-driven report, CardinalOps
In order to be effective, SIEMs now aggregate log set out to gain visibility into the current state of
and event data from an exponentially-growing threat detection coverage in enterprise SOCs.
number of data sources across the infrastructure
(applications, network and endpoint security tools,

www.cardinalops.com 3
Using the 190+ adversary techniques in MITRE ATT&CK as the baseline,
baseline we found that actual
CONTINUOUSLY MANAGE YOUR DETECTION POSTURE
detection coverage remains far below what most organizations expect and what SOCs are expected
to provide. In particular, we found that on average:

Enterprise SIEMs only address

5 14 of
the
top

ATT&CK techniques used by adversaries in the wild.

80%
Enterprise SIEMs are MISSING DETECTIONS for

of all MITRE ATT&CK techniques.

15%
of SIEM rules are broken and will never fire
due to common issues such as misconfigured
data sources and missing fields.

www.cardinalops.com 4
Organizations disable

75%
of generic out-of-the-box
vendor content due to noisiness
& customization challenges
(e.g., log source types, field names, etc.).

Only

25%
of organizations that forward identity
logs such as Active Directory and Okta
to their SIEM, actually use them in
their detection rules.

(Zero trust, anyone?)

 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

Worse, organizations are often unaware of the gap between the theoretical
security they assume they have and the actual security they get in practice,
creating a false impression of their detection posture.

What are the reasons for this disparity?

Complexity from a constantly increasing number of data sources, attack

vectors, and correlation rules. In fact, according to Ponemon, more than
80% of security professionals rate the complexity of their SOC as very high,
and less than 40% assess their SOC as highly effective.

Constant change in infrastructures, security tools, attack surfaces, adversary

techniques, and business priorities (e.g., cloud).

No “one-size-fits-all” — every enterprise is unique, making it impractical to

copy-and-paste generic content.

Manual and error-prone processes make it difficult to effectively scale and

maintain high-quality detections.

Lack of skilled personnel to understand and develop use cases across diverse
scenarios and log source types.

www.cardinalops.com 6
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

In section 2 of the report, we provide a series of Google Cloud and former Gartner Research Vice
best practice recommendations to help CISOs President and Distinguished Analyst.
and detection engineering teams address these
challenges, and be more intentional about how It is our goal to help the security community move
detection coverage is measured and continuously forward in recognizing the importance of bringing
improved over time. These recommendations are automated, repeatable, and consistent processes
based on the experience of our in-house security to detection engineering, and to provide
team and SIEM experts like Dr. Anton Chuvakin, independent benchmarks enabling CISOs and SOC
Head of Security Solution Strategy at leaders to answer the question “How prepared
are we to detect the highest priority threats?”

METHODOLOGY
Rather than rely on subjective survey-based data, CardinalOps analyzed
configuration data from real-world production SIEM instances to gain visibility
into the current state of threat detection coverage in modern SOCs.

We examined aggregated and anonymized data across:

■ Diverse verticals including financial services, manufacturing, media,

pharmaceuticals, freight logistics, and MSSP/MDR services
■ Diverse SIEM solutions including Splunk, Microsoft Sentinel, IBM QRadar,
and Sumo Logic

■ 14,000+ log sources

■ Thousands of detection rules
■ Hundreds of log source types

These organizations represent multibillion dollar, multinational corporations – making

this one of the largest recorded samples of actual SIEM data analyzed to date.

www.cardinalops.com 7
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0

2.0 3.0 4.0 5.0 6.0 7.0 8.0

STATE OF
DETECTION COVERAGE
AS MEASURED BY
MITRE ATT&CK
“Organizations need to become more intentional about
detection in their SOCs. What should we detect? Do we have
use cases for those scenarios? Do they actually work? Do they
help my SOC analysts effectively triage and respond?”

- Dr. Anton Chuvakin

Head of Security Solution Strategy, Google Cloud
SANS webinar on “The Future of SIEM”

www.cardinalops.com 8
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

Here are the insights we developed after analyzing a broad cross-section of

real-world SIEM production instances.

2.1 Coverage across all ATT&CK techniques

On average, SIEMs only cover 20% of the 190+ adversary techniques described in MITRE
ATT&CK v10.1

2.2 Coverage for top ATT&CK techniques used by

adversaries in the wild

On average, SIEMs cover fewer than 5 of the top 14 techniques used by adversaries
in the wild, based on 2021 threat intelligence data from Recorded Future, Red Canary,
and Picus Security.

These consist of:2

- T1059 Command and Scripting Interpreter - T1036 Masquerading

- T1218 Signed Binary Proxy Execution - T1486 Data Encrypted for Impact
- T1543 Create and Modify System Process - T1082 System Information Discovery
- T1053 Scheduled Task / Job - T1497 Virtualization/Sandbox Evasion
- T1027 Obfuscated Files or Information - T1098 Account Manipulation
- T1105 Ingress Tool Transfer - T1219 Remote Access Tools
- T1569 System Services - T1018 Remote System Discovery

2.3 % of broken/nonfunctioning SIEM rules

15% of SIEM rules are broken and will never fire, primarily due to fields that are not
extracted correctly or log sources that are not sending the required data.

1V11 was released after this analysis was concluded.

2To be conservative in our analysis, we excluded two techniques that SIEMs 9

are ill-equipped to address: Process Injection and Credential Dumping.
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

2.4 % of default vendor-supplied detection content

that is disabled

On average, 75% of default out-of-the box (OOB) rules provided by SIEM vendors are
disabled, due to the difficulty of adapting generic rules to each organization’s unique
infrastructure, log sources, naming conventions, etc..

2.5 Top log sources with no detections

■ The top 3 log sources that are ingested by the SIEM – but not being used for any
detections – are identity sources; SaaS productivity suites such as Office 365 and G Suite;
and cloud infrastructure log sources.

■ In fact, 75% of organizations that forward identity log sources to their SIEM,
such as Active Directory and Okta, do not use them for any detection use cases.

■ This appears to be a major opportunity to enhance detection coverage for one

of the most critical log sources for strengthening zero-trust.

“Contrary to recent opinion, monitoring, detection and response

play a critical role in zero trust deployments. In many cases,
robust visibility controls over identities, access and endpoints are
essential to make zero trust a success.”

- Dr. Anton Chuvakin

Head of Security Solution Strategy, Google Cloud
“20 Years of SIEM – SANS Webinar Q&A”

www.cardinalops.com 10
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0

3.0 4.0 5.0 6.0 7.0 8.0

BEST PRACTICE
RECOMMENDATIONS

“
“MITRE is a really good method of categorizing, measuring,
and enhancing security operations. Those that are most
successful use MITRE to communicate with others in their
businesses, to measure processes and success.”

Blog by industry analyst

www.cardinalops.com 11
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

Here are a series of best practice recommendations for enhancing

detection coverage and detection quality in your SOC.

3.1 Review current SIEM processes

■ What is the approach for finding ■ How are detections developed today and
false negatives – and what adversary what is the process for turning threat
techniques, behaviors, and threats knowledge into detections?
are currently being missed?
■ How long does it typically take to develop
■ How are use cases managed and new detections?
prioritized? Typically, we find they’re
added to the backlog via an ad-hoc ■ Is there a systematic process to
process driven by a combination of: periodically identify detections that are
no longer functional due to infrastructure
- Threat analysts changes, changes in vendor log source
- Red teaming formats, etc.?
- Breach and attack
simulation (BAS) tools
- Manual pen testing
- News about the latest high-profile
attacks or vulnerabilities

3.2 Become more intentional about how you develop

and manage detection content

Focus on effectiveness, coverage, and improvements. Ask your team questions such as:

■ What do I need to detect based on our ■ Do I triage and respond correctly?

business priorities, crown jewel assets,
industry sector, etc.? ■ What is our current coverage compared
to adversary techniques most relevant
■ What do I detect today? to our organization?

■ Do I really detect it? ■ Are we missing data sources that

would improve our coverage in high-
■ Do I detect it well? priority areas?

www.cardinalops.com 12
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

3.3 Build or refresh your use case management processes

Choose 3-5 enhancements to address the questions from the last section, with an agreed-
upon timeline.

3.4 Measure and continuously improve

■ Detection engineering processes are on people, process, and technology - are

no different than other security and IT needed for consistent improvement
management processes. As IT modernizes
and uses DevOps and SRE approaches, ■ Set organizational goals around how to
so should the SOC. increase detection coverage and reduce
the time to detect non-functioning rules.
■ You can’t improve what you can’t
measure. Many SOC metrics - focused

“The kill chain framework was a very popular approach for

use case development until the ATT&CK framework came
and expanded upon it. Essentially, this builds on the kill chain
with more depth and more value for most use cases today.
I would probably focus my analysis on the ATT&CK
framework and not on the original kill chain model
(while keeping my awareness of it, of course).”

- Dr. Anton Chuvakin

Head of Security Solution Strategy, Google Cloud
“20 Years of SIEM – SANS Webinar Q&A”

www.cardinalops.com 13
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0 3.0

4.0 5.0 6.0 7.0 8.0

HOW CARDINALOPS
CAN HELP
“Network defenders do not need 100 percent accuracy in
our models to support risk decisions. We can strive to simply
reduce our uncertainty about ranges of possibilities. The concept
of measurement is not the elimination of uncertainty but the

“
abatement of it. If we can collect a metric that helps us reduce
that uncertainty, even if by just a little, then we have improved
our situation from not knowing anything to knowing something.”

- Douglas Hubbard
(as reviewed by Rick Howard, former CISO at Palo Alto Networks)
How to Measure Anything: Finding the Value of ‘Intangibles’ in Business

www.cardinalops.com 14
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

CardinalOps’ cloud-based platform content and metrics enabling your SOC team
continuously ensures your SIEM/XDR has to stay ahead of constant change in the
high-fidelity detections for the adversary attack surface and threat landscape – plus
techniques most relevant to your business continuously identify and remediate broken
priorities and infrastructure, mapped to the rules and misconfigured log sources – so you
MITRE ATT&CK framework. can close the riskiest detection gaps that
leave your organization exposed.
Leveraging proprietary analytics and AI-
powered, API-driven automation, the
platform continuously delivers new detection

After connecting to the SIEM/XDR via its API, the service will analyze all
rules, alerts, and log source types to:

4.1 Visualize coverage based on MITRE ATT&CK heat map

Provide a coverage heat map based on MITRE ATT&CK for all use cases in your production
SIEM/XDR

4.2 Recommend, test, and automatically deploy

new rules to close gaps in threat coverage

■ Recommend missing rules from a ■ Customize rules to the organization’s

knowledge base of best practice detections field mappings, thresholds, exclusions,
to improve threat detection coverage and and naming conventions
close gaps in threat coverage

■ Enable new rules to be reviewed, tested

(versus historical data), and automatically
deployed with a single click after approval
(or via the API for Git-based CI/CD
workflows)

www.cardinalops.com 15
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

4.3 Identify, test, and automatically remediate

broken or noisy rules

■ Identify all broken or misconfigured ■ Recommend remediations to broken or

rules (i.e., rules that will never fire due to sub-optimum rules
missing fields, misconfigured log sources,
etc.) ■ Enable remediations to be reviewed,
tested (versus historical data), and
■ Identify noisy rules that can be tuned automatically deployed after approval
to further reduce alert fatigue (e.g., by with a single click — or via the API if using
modifying thresholds and exclusions) Git-based CI/CD workflows

4.4 Enable risk-based prioritization based

on organizational requirements

Enable risk-based prioritization by:

Log source type Adversary group

(APT28, FIN7, etc.)

Platform type (cloud, Windows, High profile attacks

Linux, containers, etc.) (log4j, SolarWinds, etc.)

Increasing your MITRE ATT&CK Threat severity

coverage score

www.cardinalops.com 16
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

4.5 Identify log sources not contributing

to threat coverage

■ Identify log sources that are ingested but

not contributing to threat coverage.

■ This information can be used to reduce

ingestion costs and/or develop new rules
to expand threat coverage by incorporating
these log sources.

4.6 Produce independent metrics to measure detection

posture and demonstrate continuous improvement

■ Help CISOs answer the question “How

prepared are we to detect the highest
priority threats?”

■ Generate independent, board-level metrics

to measure SOC effectiveness

■ Track continuous improvement over time

www.cardinalops.com 17
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0 3.0

5.0 6.0 7.0 8.0

Screenshot Examples
You can also view a 2-minute demo here.

5.1 MITRE ATT&CK heat map

Map with all ATT&CK techniques color-coded according to health of detection rules.
Techniques shown with vertical cross-hatched lines are not covered by any detections.

www.cardinalops.com 18
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

Map with examples of sub-techniques and broken rules within the technique.

5.2 Recommended rules to close

gaps in threat coverage

www.cardinalops.com 19
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

5.3
Example of recommended best practice rule
to close gaps in ATT&CK coverage

5.4 Description for recommended rule

www.cardinalops.com 20
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

5.5 Example of broken rule with remediation

5.6
Log source types
not contributing to
detection coverage

www.cardinalops.com 21
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

5.7 Log source types not parsed correctly

5.8 MITRE ATT&CK coverage score over time

www.cardinalops.com 22
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0 3.0 4.0 5.0

6.0 7.0 8.0

Rapid, Secure,
API-Driven Deployment
Deployment is typically achieved in less than an hour.

■ The platform connects to your SIEM/XDR’s to run queries on that data. Raw events
API via provided credentials. never leave the SIEM. The platform only
requires access to SIEM/XDR configuration
■ Secure network connectivity is achieved information and metadata around
via your preferred process (Azure detection rules, connectors, data sources,
Lighthouse, onsite broker VMs, etc.). etc.

■ To get an initial health check, you can ■ Integration with Git-based CI/CD platforms
configure the platform with read-only is also enabled via our API.
credentials and only expand permissions
later to derive the full platform value ■ The CardinalOps platform is SOC-2
from automated provisioning of new and compliant for security and confidentiality,
remediated rules. ensuring your SIEM/XDR configuration
data is always protected using best
■ The platform never obtains or requires practices.
access to log data because it uses the SIEM

www.cardinalops.com 23
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0 3.0 4.0 5.0 6.0

7.0 8.0

Key initiatives benefiting

from detection use
case management
7.1 Operationalize MITRE ATT&CK in your SIEM/XDR

Most organizations track coverage of MITRE ATT&CK using static, manual approaches like
spreadsheets. Although some SIEM/XDR vendors have recently begun displaying a coverage
map in their platforms, true operationalization requires going beyond simple mapping to
include rule recommendations, identification of data quality issues, and customization of
detection content to your organizational priorities and infrastructure.

www.cardinalops.com 24
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

7.2 Securely accelerate cloud transformation initiatives

Organizations are increasingly moving business-critical applications to the cloud. Security

teams want to support the business by accelerating cloud initiatives – but without introducing
unnecessary risk.

Adding complexity, each cloud platform has its own monitoring tools, which need to be
onboarded as log sources to the SIEM/XDR. For example, AWS log sources include S3, VPC,
EC2, CloudWatch, and Cloud Trail, as well as both system- and application-layer log sources for
containers such as Kubernetes. Plus, new cloud security tools like CSPM and CIEM also require
onboarding of new log sources.

CardinalOps helps securely accelerate your cloud initiatives by providing pre-built, ready-to-use
detection content for these new log sources.

7.3 Rapidly update detections for high-

profile vulnerabilities and attacks

Community sharing, blog posts, and subscription-based content can be helpful with new
threats — but copying and pasting generic content is often insufficient for addressing complex
threats like log4shell and SolarWinds. CardinalOps can help with rich, high-fidelity detection
content that’s specifically adapted to your log sources and infrastructure.

7.4 Scale effectiveness of detection engineering teams by 10x

Security talent is hard to find and retain. With CardinalOps, you leverage AI-powered, API-
driven automation and analytics to replace tedious and mundane activities (like looking for
Regex typos and misconfigured collectors) – so that your best talent can focus on higher-value,
more strategic activities — such as researching new and novel attack techniques — thereby
boosting morale and retention.

In fact, customers tell us the platform has increased the detection content output of their
security engineering teams by a factor of 10 compared to their previous manual approaches.

www.cardinalops.com 25
 CONTINUOUSLY MANAGE YOUR DETECTION POSTURE

1.0 2.0 3.0 4.0 5.0 6.0 7.0

8.0
Customer Examples
The CardinalOps platform is currently in SOCs protecting some of
the world’s largest and most complex organizations, across diverse
verticals including financial services, manufacturing, hospitality, media,
transportation & logistics, law firms, and managed security services.

Chosen by Global Organizations

Top 10 CPG Manufacturer Top 10 Private Equity Firm Top 20 Retail Cosmetics firm

Top 10 Casino Company Top 10 Money Transfer Firm Top 10 US Law Firm

Top 15 MDR Prodivder $3B Fteight Logistics Firm Top 10 Cable Operator

www.cardinalops.com 26
 ABOUT
CardinalOps’ cloud-based platform continuously
ensures your SIEM/XDR has high-fidelity detections
for the adversary techniques most relevant to your
business priorities and infrastructure, as measured
by the MITRE ATT&CK framework.
Leveraging proprietary analytics and API-driven automation, the platform
continuously delivers automatically customized, deployment-ready
detection content and metrics enabling your SOC team to stay ahead
of constant change in the attack surface and threat landscape – plus
continuously identify and remediate broken rules and misconfigured log
sources – so you can close the riskiest detection gaps that leave your
organization exposed.

Founded in early 2020, CardinalOps is led by serial entrepreneurs whose

previous companies were acquired by Palo Alto Networks, HP, Microsoft
Security, IBM Security, and others. The company’s advisory board includes
Dr. Anton Chuvakin, recognized SIEM expert and former Gartner Research
VP (now Head of Security Solution Strategy at Google); Dan Burns,
former Optiv CEO and founder of Accuvant; and Randy Watkins, CTO of
Critical Start.

For more information, please visit www.cardinalops.com.

X X X X X X X X
X X X X X X X X
X X X X X X X X