Accelerate and secure

your SDLC
with DevSecOps
A phase-by- 03
Introduction

phase guide 05
DevSecOps at every phase
of the SDLC

The exploration phase 07

→

The coding phase 11

The building phase 14

The testing phase 15

The deploy and runtime phase 17

WRITTEN BY:

19
What’s next
Chas Clawson Colin Fallwell
Field CTO, Security Field CTO, Observability
Sumo Logic Sumo Logic

02
INTRODUCTION: Driving the adoption of DevSecOps is the need for teams to
maintain innovation at speed and establish a competitive

DevSecOps
differentiation that maintains and accelerates market share.
In the past security was often pushed to the end of the software
development life cycle (SDLC). As hacks have become more

is a term
prevalent, more costly, and more difficult to detect, security has
become of top-line importance to teams.

In short, when your team has to go from requirements to production

that’s grown in a day — and do it safely and securely — DevSecOps is the path
that gets you there.

in usage and
popularity.

Shared dashboards extend visibility across teams.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

03
Elite teams embracing DevSecOps
don’t conduct security code reviews

15%
as an afterthought.
They have a security-first, security-always approach and
integrate and prioritize security and observability at every step
of the SDLC. They embrace DevSecOps practices everywhere
of dev teams can
from developer laptops to pushing code into production — and
be considered
everything in between.
“elite”

In this guide, we’ll take a look at tactical steps you can take at
each phase of the SDLC to help transform your team into an elite
DevSecOps performer. Elite team performance

DEPLOYMENT FREQUENCY On demand

LEAD TIME FOR CHANGES Less than one day

TIME TO RESTORE SERVICE Less than hour

CHANGE FAILURE RATE 0-15%

Elite teams, as defined by the DORA (DevOps Research and

Assessment) metrics, deploy software multiple times per
day, have a lead time for change and a restore of service in
under an hour, and keep their change failure rate under 15%.

Source: Accelerate State of DeVOps 2021

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

04
THREE CONSIDERATIONS:

DevSecOps SOFTWARE
DEVELOPMENT
→ Lead time

at every phase Deployment

frequency

of the SDLC
SOFTWARE
DEPLOYMENT
→ Change fail

Time to restore

SERVICE
OPERATION
→ Reliability

DevSecOps can be viewed not only

from a maturity perspective but from
Performance metrics
a discrete capabilities perspective. * Key metrics
Source: “Is it ODD to shift left? Becoming elite DevSecOps performers”
Colin Falwell, 2022
That is, your team can add individual tools and practices one
at a time. Further, teams adopting DevSecOps may need to think
beyond the typical metrics used by development, security and
operations teams. GSA, the Government Services Administration
shares a range of high-value and supporting metrics and
capabilities to consider as teams collaborate across disciplines,
including deployment frequency and MTTR as well as frequency
of vulnerability patching and privilege auditing.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

05
DevSecOps has changed considerably Observability is foundational
with the advent of highly-capable to enabling DevSecOps.
open-source projects.
Building an observable system that provides telemetry to everyone
involved — SecOps, DevOps, IT, finance, business intelligence (BI)
Tools like Falco, kube-hunter, Prometheus, et al have become
and data science teams, and more — is critical.
mainstays in practice as enablers of innovation speed. Today,
enterprise security teams are adopting more open-source
Observability in DevSecOps is not just metrics and traces and logs
security solutions as they integrate with and compliment the
for dev teams. It’s embracing observability-driven development.
growth in environments running open-source, such as Kubernetes.
This means dev teams are highly capable and mature in observing
the internal processes that ultimately enable them to test code
directly in production. They are measuring not just deployed
services, but also CI/CD pipelines, telemetry pipelines, control
planes for automation, processes that govern software delivery,
the standards these processes employ and more. Having securi-
ty processes as part of the DevOps pipeline reduces friction and
increases adoption best practices.

Now let’s step through the five phases

of the SDLC and look at specific
and valuable actions you can take to
embrace DevSecOps.
ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS
06
 1 The
exploration
phase
Two distinct DevSecOps aspects you can
bring to the planning phase are creating a
standardized specification and defining
security inputs and outputs.

Creating a spec during the planning phase

ensures that the needs of development,
security and operations are all being
considered. The spec should not only
establish functional “what should the software
do” requirements, but it should also define
operational, performance and security
requirements. These specifications should
be defined for each supported language, and
all pipelines for that language type should be
standardized to the spec.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

07
 Operational and performance
impacts:

Defining the upper and lower bounds for Logging format and verbosity
critical metrics
A standardized logging format across the enterprise reduces
Golden Signals or any other metrics used to observe a system
churn. Define the metadata to include in the log message.
should have upper and lower specification limits that define
Invest time gathering input from stakeholders. Logged
a performance corridor that conveys normal operation. The
attributes directly impact the quality of dimensional analysis
metrics defined should answer common operational questions
in the automated machine learning (AutoML) that identifies
like “How does our environment behave during peak
root cause when canaries fall off their perches.
utilization?” “Can we identify the point when we need to
scale?” and “Do I have the telemetry to detect anomalous
behavior or malicious use of services?” Answering these
Open-source tooling decisions
questions in the spec makes it clear when you’re no longer Open-source software has become a foundation of
meeting the agreement between all stakeholders and when observability. Tools like Prometheus, Grafana, and Jaeger
security threats might unfold. create a common platform to share information
across multiple teams. Commercial solutions are exclusive
This is the basis for reliability management during the and proprietary, limiting what development teams can do
later phases of the SDLC. Your metrics should become the and ensuring vendor lock-in. In a culture of continuous
‘canaries in the coal-mine’, signaling with confidence when improvement owning the code and libraries to build
action is warranted. It is also critically important to observability is a necessity.
accomplish this with the least number of metrics possible.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

08
From a pure security perspective, Infrastructure security
a spec means defining security inputs is also imperative.
and outputs.
Making these decisions in a repeatable, measurable way during the
exploration and planning phase allows you to integrate additional
For example, to support supply chain security policies, the
tooling like kube-hunter for vulnerability discovery or kube-bench
development team may need to create a plan around developing
for compliance. Making these decisions in this exploration phase
a software bill of materials (SBOM) generation practice or need
allows you to integrate them into the overall deployment strategy
to expose the right metadata to support a downstream security
and increase the speed at which you find security problems.
information and event management (SIEM) software tool.
Recommendations from security tools should be actionable
As part of creating your spec, you may also consider starting
and incorporated into the specification.
your threat model. Threat models are planning documents that
consider and plan for various likely attack scenarios, the data
that could be exposed by the attacks, and possible solutions.
The threat model will evolve throughout your SDLC, but now is
a great time to start.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

09
 Log Log data is critical for monitoring, troubleshooting and
investigating reliability and security issues to get down to the root

data
cause of “why” that issue occurred. Log data is often the most
detailed information available about a company’s systems, so it
makes sense to put that data to work, pulling log files across the
organization into a single analytics platform for end-to-end visibility
and faster troubleshooting.
→

Log data is often the most detailed information

about a company’s systems.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

10
 2 The
coding
phase
DevSecOps in the coding phase means
building security into the code itself. In the
exploration phase, you defined the development
spec. In the coding phase, you’ll implement that
spec in your code. Two ways you might do this
are observability and security policies and
vulnerability prevention.
Sumo Logic Distribution for OpenTelemetry Collector

We live in a complex, multi-architecture, An easy-to-implement observability solution that

multi-microservice world that requires full considers logs, metrics, and traces is key. This is
observability across stacks and environments. especially critical in a microservice architecture and
In the context of DevSecOps, observability across all of your application footprints. Practically,
refers to making all phases of the SDLC visible this means leveraging a tool like Sumo Logic
to prevent security issues before they make Distribution for OpenTelemetry Collector.
it to production.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

11
In the coding lifecycle, you should
also implement software supply chain Common tool examples:
security policies and vulnerability
Static application security testing (SAST)
prevention (among other practices).
Tooling in this category often scans your code, helping
you to catch common issues like SQL injection, buffer

There are many approaches and overflows, and other well-known vulnerability catego-
ries. While not perfect, the outputs from SAST tooling
tools to making code secure — some provide data points that inform the overall health of
language-specific, some industry- the code base.

specific and you’ll need to research

to find the ones appropriate for Author verification
Validating that commits in your code come from a
your project. verifiable source is another recommended security
practice. In some cases, creating commits that are
GNU Privacy Guard (GPG)-signed ensures that the
author is verified. This also creates a bar for merging
code into the mainline: no code merges unless its
author is verified through a GPG signature. (GPG isn’t
the only option but it has some notable benefits over
other options, such as key revocation.)

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

12
 Reliability The coding phase is the root where you’ll implement capabilities
that support your reliability management framework.

management For example, you’ll need to consider how you’ll measure your
service level indicators (SLIs) against your service level objectives
(SLOs) and how your error budget informs how you prioritize work.
Exceeding your error budget will trigger more work to bring it back
→

in compliance with your service level agreements (SLAs).

Solve customer-impacting issues faster with visibility

to all your application data including logs, metrics and
traces across the entire development lifecycle.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

13
 3 The Real-world examples of CI/CD

building
practices you can implement:

Ensure that your process for creating

phase
container images is reliably reproducible and verifiable.

Pulling forward the concept from the coding phase,

only allow GPG-signed commits, thereby performing
The building phase should focus on builds for commits from verified authors.
reproducible and verifiable builds. Code that is
committed or checked into the repository should
Automate your pipeline to collect metrics
create a deployable artifact that is certified
and artifacts, keeping the feedback loop short and
reliable and secure.
consistent. For example, capturing and measuring build
time will inform the overall time it takes to release a fix
Many tools and practices can help with the above
to production.
goal, as CI/CD is a practice unto itself.

Generate and archive an SBOM for every build,

providing insight into the supply chain. Use this
alongside software composition analysis (SCA) tools
like OWASP Dependency-Check to find vulnerable
software in your dependencies.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

14
 4 The
testing
phase
You’ve integrated DevSecOps practices
and tools earlier into your SDLC. You’ve secured
your code, tightened up your build process, and
maxed observability. The next step is to test.

One of the keys to DevSecOps in testing is

automation. Running automated tests whenever
possible creates a predictable and consistent
testing practice. Run automated tests throughout
the testing (and other) phases of the SDLC.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

15
 Testing End-to-end Testing
Work in automated penetration testing that can then trigger

tools additional code iterations to fix vulnerabilities. Validating deeper

operational aspects like log emission and metric exposure can
also happen at this phase.
In addition to functional Dev/QA and non-functional
performance testing, the following tests are important
Dynamic application security testing (DAST)
during this phase:
“Black box” tests that have no knowledge of the inner workings
of the application and approach the application as a hacker
would, simulating real-world attacks.
→

Unit Tests Acceptance Testing

Verify that critical code paths have coverage and are Validate that the correct observability and security signals have
hardened against vulnerabilities discovered through SAST been captured. This phase will usually take place in a staging or
tooling and code linters. pre-production environment and is the time for activities that are
difficult to do manually, such as custom penetration testing and op-
erational or performance testing trigger additional code iterations
to fix vulnerabilities. Validating deeper operational aspects like log
emission and metric exposure can also happen at this phase.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

16
 5 The deploy
and runtime
phase
DEV SEC OPS

Once your application is live, you can’t just forget

about it. The operational/runtime phase is where
you rely on instrumentation to identify any
issues you missed. Humans can’t write perfect
DevSecOps delivers better quality
code, and you’ll never be aware of every possible and more secure software

problem. It’s imperative to have tools that

continuously scan and monitor for possible exploits By this point, your DevSecOps planning,
or performance problems. implementation, and testing metrics and artifacts
will coalesce into a single pane of glass accessible
to the development, issues and operations teams.
This solution should provide ubiquitous data
collection, easy analysis, alerting, and visualization.
This culmination of your DevSecOps practices
allows your security and operations teams to be
not just reactive but proactive.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

17
For example, you might use the
Kubernetes integration with Sumo Pulling forward the idea of reliability management, this

Logic, which can detect anomalous phase will inform what the next cycle of the SDLC looks like.
Burning through your error budget here will indicate that
container activity, such as a the next development cycle requires a focus on bug fixing
misbehaving container reading a and stability whereas a surplus of error budget could mean
there’s more room for innovation. Ultimately, the goal of the
sensitive file on the filesystem. reliability management framework is to balance the speed
of innovation with serving your customers.
This activity triggers a security alert rule that notifies your
security team, who might trace the deployment back to a git
commit with a configuration not captured by the test suite or
QA team. The security team can then notify the development
team to get a fix and gauge whether a deployment rollback is
necessary. The team can decide whether to wait for the fix to
deploy, basing their judgment on the captured build time and
deployment time metrics.

This is one of many potential scenarios where development,

security, and operations can have shared insight throughout
the SDLC and work together to improve their security posture
and operational efficiency using a single pane of glass.

The Kubernetes integration with Sumo Logic

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

18
CONCLUSION Sumo Logic is an all-in-one
observability and security platform

What’s next? that coalesces metrics, logging, and

tracing into a single pane of glass.

We’ve looked at the phases, tooling, and processes that It’s designed to work with all of the tools you already use and
embody the spirit of DevSecOps and how they can enable an provides additional utility such as the Sumo Logic OpenTelemetry
SDLC that results in quick and secure innovation. DevSecOps Distribution to collect your data in a non-proprietary manner and
comes down to integrating security and observability culture, deliver real-time insights.
processes and capabilities throughout your SDLC. By shifting
security and observability concerns to your entire team — and at Visit Sumo Logic to learn more about how our SaaS analytics
every step—your application will be more robust, cost-effective platform helps global industry leaders deliver reliable and secure
and secure. digital experiences.

ACCELERATE AND SECURE YOUR SLDC WITH DEVSECOPS

19
About Sumo Logic
Sumo Logic, Inc. (NASDAQ: SUMO) empowers the people who power
modern, digital business. Through its SaaS analytics platform, Sumo Logic
enables customers to deliver reliable and secure cloud-native applications.
The Sumo Logic Continuous Intelligence Platform™ helps practitioners and
developers ensure application reliability, secure and protect against modern
security threats, and gain insights into their cloud infrastructures. Customers
around the world rely on Sumo Logic to get powerful real-time analytics and
insights across observability and security solutions for their cloud-native
applications. For more information, visit: SUMOLOGIC.COM

©️ Copyright 2023 Sumo Logic, Inc. Sumo Logic is a trademark or registered trademark of Sumo Logic in the United States and in
foreign countries. All other company and product names may be trademarks or registered trademarks of their respective owners.