Training – DevSecOps

Overview
Familiarity with the DevSecOps process

​August, 2021
 Topics
Agenda 1 | What’s unique about DevOps?

2 | How can DevOps practices make Axis High Performing?

3 | How DevOps differs from traditional IT Operations

4 | Scaling from IT/Software/DevOps Engineer to a
DevSecOps Engineer
5 | Pre-requisites to adopt DevSecOps Model
6 | Axis Centralized Continuous Integration and Continuous
Deployment Platform
7 | Success Stories
8|Q&A
 What’s unique
​
about DevOps
 What is DevOps?

• DevOps is a set of practices that emphasizes the collaboration and communication of

both software developers and other information technology (IT) professionals, while automating the
process of software delivery and infrastructure changes, its implementation can include the definition of
the series of tools used at various stages of the lifecycle; Wikipedia
• …there is no one product that can be considered a single DevOps tool.
• Instead a collection of tools, potentially from a variety of vendors, are used in one or more stages of the
lifecycle.
• 7 stages of DevOps:
• Planning – code development and review, source code management tools, code merging
• Configuring – infrastructure configuration and management, infrastructure as code tools
• Creating – continuous integration tools and build status
• Verifying – continuous testing tools that provide quick and timely feedback on business risks
• Packaging – artifact repository, application pre-deployment staging
• Releasing – change management, release approvals, release automation
• Monitoring – applications performance monitoring, end-user experience

The infinity symbol depicts Continuous Improvement of the product/application in consideration by

improving the processes, skills and technologies
4
​

The history of DevOps and DevOps Days

Enormously frustrated with DevOps” had
the wall of separation officially landed
between Development and in the history
Operations books
Starts assessing IT
Gartner and Forrester
value chain Cameron Haight of reported DevOps as a
Inaugural
Gartner predicts Gene Kim released mainstream strategy for
Patrick Debois Agile System DevOpsDays
DevOps will hit the the Phoenix Enterprise Organizations
Project manager and event in Ghent,
Administrator Belgium
big time in 2015 project
agile practitioner from across enterprises
Belgium
Group
Oct 30-31

2013- 2016
2007 2008 2009 2010 2011 2012
15 onwards
June 23

DevOpsDays happened Industry leading Nicole Forsgren,

for the first time in software GeneKim and
Mountain View, vendors like others beign
California, on the heels of IBM, HP, CA publishing Annual
increased State of DevOps
the Velocity annual
Report, adoption
conference market presence
of DevOps to be
with Enterprise accelerating
class Devops
Andrew Shafer John Allspaw and Paul
tools
Hammod from Flickr
Agile Conference, presented “10+ Deploys
Toronto – “Agile per Day: Dev and Ops
Infrastructure” Cooperation at Flickr”
5
​

… so what DevOps is meant to be and not meant to be?

► DevOps is not a product or tool

► DevOps is not just a combination of

development & operations team
Culture Automation ► DevOps is not a separate team

Lean ► DevOps is not a one-size-fits-all strategy

Sharing Measurement ► DevOps is not only about automation

While the areas defines the success criteria for DevOps, these
also becomes the risks and challenges as well

6
 How can DevOps
​
practices make
Axis High
Performing
​

Key benefits of DevOps

Significant and Improved Software Reduced rework & better

Continuous improvement Performance bandwidth utilization contributing
of product quality for other meaningful work

Faster & Frequent Build & Consistent, faster and Reliable and Consistent
Deployment to promote reliable release contributes environments improving
experimentation & shorter time to market & system reliability
innovation change lead time

8
​

Who benefits from DevOps

Developers Test Engineers

• Fewer mundane, repetitive tasks because of • Availability of QA bandwidth to perform exploratory and
automation adhoc testing to determine system limits
• More time developing • Faster iteration of test cycles for every changes made
• Increased test iteration resulting potentially more defects to
be found

System Administrators
• 96x faster recovery from failures Operations
• 3x lower failure change rate • 22% less time spent on unplanned work and rework
• Less complexity to manage • Better code from developers because of increased
communication

Product Managers
• 46x more frequent software deployments
End Users
• Experience more consistent processes and
• 440x faster lead time to changes
applications
• Improved experience

9
 ​

Let’s revisit the traditional Software Development Life Cycle…

• Difficulty in integrating changes of other developers • Ops team’s intervention in all types of deployments creating
incurring frequent merge conflicts and eventual delay in bottleneck with frequent deployment requirements in lower
delivery environments
• Delayed and shorter testing cycle incurring • Increasing technical debt due to lack of review of the code
compromise on software quality changes and increased pressure for feature development
from business to meet customer need
• Increased application maintenance effort due to lack of
stability, reliability and performance • Continuous manual intervention leading to loss in
productivity and innovation

Development Team Operations Team

Testing Team

Plan Design Implementation Test Feedback Deployment Monitoring

Overall Impact: Increased Time to Market and change lead time, Increased code complexity,
Compromised code quality, inflexibility in responding to quick changes and modernization
10
​

..so how can we improve?

By automating the delivery process workflow

By redefining the delivery methodology
that supports on-demand delivery of Development team puts down a
plan keeping in mind of the
changing user needs application objectives Automate the
release readiness
process Deploy the changes
Instead of… Code the changes automatically across
and version control environments
The traditional waterfall Methodology

Ideate Plan Development Test Release

…adopt agile principles to iteratively

develop and deliver changes
Dev Ops
The agile delivery Methodology

Build the changes Validate the Continuously

automatically changes through monitor and act
automated test
Iteration 1 Iteration 2 Iteration 3 suites

The infinity symbol depicts Continuous Improvement of the product/application in

consideration by improving the processes, skills and technologies

11
​

Agile and DevOps complement each other while the former sets up the delivery
process and the later further refines and provides a way for implementation

Pre-game

1. Requirements
5b.
Development Business
5a. Daily Functionality
scrum
Scrum
Product
Testing
increment
2. Requirement 3. Backlog 5c. Refine life cycle
Validation construction product backlog
4. Sprint 5. Sprint
Check in code
planning to SCM
Invoke Continuous
Agile Integration/Deployment
Development pipeline
Product
Modern User Story backlog Check Code
Build
7. Sprint 6. Sprint Quality
retrospective review Deployment
Business Functionality Specification
Read Post
deployment Deploy to
Deployment Specification Deployment
specification UAT/Staging
Sanity
Testing Requirement Testing
9. Gather metrics
10. Post Automated
Infrastructure Requirement Product
usage, errors, 8. Deployment
Deploy to
testing in
feedback, performance Live production
feedback Prod Sanity UAT/Staging Testing
NFR (Continuous Monitoring)
Requirement
(Scalability/Availability/Performance)

Agile can not be fully implemented without DevOps 12

 Quiz Time
​
Quiz
​ Time

DevOps Means…
A. Developer taking over all operations tasks
B. Automating the process of software delivery and infrastructure changes
C. The collaboration and communication of both software developers and all other
information-technology (IT) professionals
D. The collaboration and communication of just software developers and operations
staff
Correct Answer: C

Which of the following statement is false about DevOps?

A. Less complex problems to fix
B. Stable operating environments
C. Faster resolution to problems
D. Continuous Software Delivery

Correct Answer: A

14
Quiz
​ Time
What benefit DevOps can bring within an organization?
A. Improved deployment frequency, which can lead to faster time to market
B. Lower failure rate of new releases
C. Shortened lead time between fixes
D. Faster mean time to recovery in the event of a new release crashing or otherwise
disabling the current system
E. All of the above
Correct Answer: E

Agile and DevOps are similar but differ in a few important aspect.
Which statement is correct?

A. Agile is a change of thinking whereas DevOps is actual

organization cultural change
B. Agile is actual organizational cultural change whereas
DevOps is a change of thinking.
C. Agile is process driven whereas DevOps is role driven.
D. Agile is role driven whereas DevOps is process driven.
Correct Answer: B
15
 How DevOps
​
differs from
Traditional IT
Operations
 ​

Traditional IT Operations
• Before the advent of DevOps, companies had separate “walled-off” teams of developers, testers, and
operations, All too often they had conflicting goals.
• Developers would spend 3-4 months building a ton of features and then try to merge their code. This process
was slow and tended to produce lots of errors.
• After a long integration, Developer would handover their code to QA.
• When testers discovered a ton of bugs, developers would often respond with the classic: IT WORKS on my
machine

• Once finger pointing is over and bugs are worked out, developers would pass the torch to Operations.
• Ops would then try to deploy the code to production. If deployment failed, they would blame Developers for
providing faulty artifacts.
• As working software was none of Developers’ business, Ops would have to swallow their pride, stock up on
coffee, and fix the mess.
17
 ​

DevOps Strategy
• DevOps emerged as a response to these issues.
• It’s a culture that aims to bridge different teams and eliminate communication bottlenecks. DevOps relies on
standardization of environments and extensive automation throughout the development pipeline.
• DevOps helps organizations to: Deploy more often (and with a higher success rate);
• Fix defects earlier and faster
• Improve product quality
• Reduce time-to-market
• Better adapt to market needs
• Boost user satisfaction
• Increase productivity
• Improve teamwork

18
 ​

DevOps Strategy
In order to embrace DevOps strategy, Organization has to abandon some of its old methods and gradually adopt
new practices.
Separation of environments
into development/test/staging/prod is the first requirement for DevOps transformation. Setting up
environments manually takes a lot of time and requires attention from the ops team.

Agile development
is another prerequisite. Focusing on rapid delivery and small cross-functional teams requires more
environments. The only way to stay agile is to increase automation and move past the ‘silo’ culture.

Microservices architecture
is an approach to building software that consists of a large number of independent services. Each
service represents a single feature and communicates with other services via APIs. However, the
increased number of releases requires more automation.

Automated regression testing

checks that updates to the code don’t break existing features. Ideally, regression testing should be fully
automated so that QA could focus on new functionality.

19
 ​

Continuous Integration (CI)

is one of the first steps in a DevOps strategy. CI allows developers to automatically build, test, and
integrate new code into a common repository. Version control helps to resolve merge conflicts. This
saves a lot of time and allows multiple Developers to collaborate on a single project.
Continuous Delivery (CD)
builds upon CI allowing you to deploy code to staging/production environment with little to no manual
intervention. Releasing as often as possible simplifies troubleshooting.
Continuous deployment (CD)
goes one step forward and automates the deployment process. The new code can be made available to
select groups of users. Automatically collected feedback helps companies respond to customer requests
in close to real time.

20
 ​

What is Continuous Integration

Continuous Integration is the software development practice in which frequent, isolated and small code
changes are integrated or added to the larger codebase on successful validation of the code changes in
an automated way
Result
Principle: “Work on short changes and Integrate frequently”

Continuous Integration Server

Check in
Depending on the size of the codebase,
code, Download Build Test Success/ the activity takes as low as couple of
frequently codebase Codebase Codebase Failure minutes to few tens of minutes which
is a significant time saving resulting in
Source Code faster feedback of the changes
Starts writing code
Repository (GitHub,
Bitbucket, svn etc.)

The workflow resembles a pipeline – hence called “Continuous Integration Pipeline”

Stages

Static Code Static Code Package as

Dependency Code
Quality Vulnerability Compile Unit Test deployable
Analysis Coverage
Review Review artefact

Break the pipeline on stage failure/threshold deviation to ensure only compliant code is integrated with the larger codebase thus less code break and improved code stability

“Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.”
21
Stages
​ of Continuous Integration

Improves Application Stability by optimizing

coding standard
Improves Developer Productivity Code Quality and • Static code quality review
Vulnerability review • Static code vulnerability review
• Repository creation strategy
• Branching Merging Strategy (SAST)
• Gated Check-in • Custom ruleset definition
• Peer Review/Parallel Programming Code Unit • Image scanning
• Repository, branch & data security Repository 02 testing and
Management code
overage Reduce the cost of defects by
01 03 finding it early in the cycle by
adopting Shift-left testing
Continuous • Unit test automation
Quick packaging for container
ready Integration • Unit test coverage analysis
• Secure Application Packaging Stages
• Automated versioning
• Artefact Repository Management Automated
06 04
• Container Image minification Automated
Packaging &
Build Multi Branch Build for continuous
Versioning
05 code integration
Automated resolution of dependent • Creation of build script and here
libraries Automated Dependency build manifest
• Software Composition Analysis Management • Multi-Branch and multi stage
• Open source vulnerability detection build
and resolution
• Central artefact repository

22
Tools
​ for Continuous Integration
There are several tools available in the market to help with Continuous Integration. But it is important for one to carefully select the right tool
that matches the needs of development and infrastructure.

23
 Quiz Time
​
Quiz
​ Time

What is Continuous Integration about?

A. Running build continuously in a loop

B. Increasing developer’s work by constantly providing
build feedback
C. Blaming the developer who put in a broken build
D. Integrating the code often so there are no last-
minute surprises
Correct Answer: D

What is not true about Continuous Integration?

A. Reduce risks
B. Reduce repetitive manual processes
C. Generate deployable software at any time and at
any place
D. Increases repetitive manual processes

Correct Answer: D
25
Quiz
​ Time

What are the success criteria for Continuous Integration?

A. Make the build self-testing

B. Keep the build fast
C. Everyone can see the results of the latest build
D. All of the above

Correct Answer: D

What is correct flow for the CI pipeline?

A. Build, version control, auto test, unit test

B. Version control, auto test, build, unit test
C. Version control, unit test, build, auto test
D. Version control, unit test, auto test, build

Correct Answer: C

26
 ​

What is Continuous Delivery and Deployment

Continuous Delivery is the software engineering practice where you build a refined version of the software
by continuously implementing the fixes and feedback until finally, you decide to push it out to production
Continuous Integration

Acceptance Deploy to Deploy to

Build Test Smoke Testing
Test staging production
Auto Auto Auto Manual Manual
Continuous Delivery

Continuous Deployment is the software engineering practice where every change goes through an
automated pipeline and a working version of the application is automatically pushed to production
Continuous Integration

Acceptance Deploy to Deploy to

Build Test Smoke Testing
Test staging production
Auto Auto Auto Auto Auto
Continuous Delivery

Continuous Deployment

Principle: “Reduce cost, time, and risk of delivering incremental changes to the business” 27
Tools
​ for Continuous Deployment
There are several tools available in the market to help with Continuous Deployment. But it is important for one to carefully select the right tool
that matches the needs of development and infrastructure.

28
 Quiz Time
​
Quiz
​ Time

What does Continuous Deployment help with?

1. Optimizing developer’s tasks
2. Building out automation test scripts
3. Getting to know Operations team
4. Effectively controlling the pipeline through automated process
Correct Answer: D

What is true about Continuous Deployment and

Delivery?
1. Both helps rolling out changes to the production
2. Continuous Deployment does not automate deployment in production
whereas Continuous Delivery automates production deployment
3. Continuous Delivery does not automate deployment in production
whereas Continuous Deployment automates production deployment
4. Neither of them automates production deployment

Correct Answer: C

30
Quiz
​ Time

Which of the following is not the right tool for deployment

automation?
A. goCD
B. Selenium
C. Jenkins
D. Azure DevOps
Correct Answer: B

Which of the following represents the correct fact?

A. Continuous integration can not be extended to continuous

delivery
B. Continuous integration can not be extended to continuous
deployment
C. Continuous deployment follows continuous integration pipeline
D. Continuous delivery follows continuous deployment pipeline
Correct Answer: C

31
 Scaling from an
​
IT/ Software/
DevOps Engineer
to a DevSecOps
Engineer
Only 21% of the [companies] believe that
their organization’s present culture and
practices support collaboration across
development, operations and security

~Freedom Dynamics
(IT Industry Analysis)

33
 ​

The SMARTER Approach…. DevSecOps

• DevOps and DevSecOps (Development, Security, Operations) are closely related and very much share the
same goal: to develop and build in the most efficient manner possible.
• The main difference is focus –
DevSecOps quite literally puts security at the center of the process. With the addition of security,
developing quality for production, considering the security at all times as we move down the pipeline.
Stages of DevSecOps:
• Source Composition Analysis (SCA) – With a modularized approach to development we do not write each
and every line of code and as such, it’s essential that we track the 3rd party libraries that we are using and
check all libraries for concurrency and vulnerabilities.
• Static Application Security Testing (SAST) – Along with SCA, this type of testing is performed early in the
development cycle and is not a one-time only process. All code that is deemed to be completed should be
assessed by SAST. There is no need for a built application, as SAST checks the code looking for the easily
identifiable vulnerabilities.
• Dynamic Application Security Testing (DAST) – The next logical step from a vulnerability testing standpoint
is DAST. It requires a deployed application and tests in the runtime environment and as such, has the
potential to identify issues that would not be detectable just by looking at the static code.
• Interactive Application Security Testing (IAST) - IAST is DAST with an instrumented
app/environment. If SAST is “white box” testing and DAST is “black box” testing, then IAST can be described
as ”grey box” testing.
• Infrastructure Scans – Ideally, within our pipeline we have infrastructure as code. Using scripts, APIs and
containerized technology we can build a variety of likely deployment scenarios to check for vulnerabilities.
34
 ​

Some common Security Integrations:

Software composition analysis (SCA)
• A code scanning tool that focuses exclusively on the third-party and open-source components you’re using to
build your application.
• SCA works by stepping through your source code to create a package bill of materials (BOM). BOM is a list
of packages that are used to help create your applications. Then, it identifies the security vulnerabilities that
are introduced due to the inclusion of those packages.
• SCA works best at the far left of the SDLC, and in many cases, it is bundled with SAST.
Static Application Security Testing (SAST)
• SAST tools examine code to find software flaws and weaknesses, such as the OWASP Top 10, duplicate
code, and hardcoded credentials.
• It analyze the application’s source code or binary. They never need to execute the application. Because of
this knowledge of an application’s underlying implementation, SAST tools are considered to be a white
box method.
• Because SAST tools only need the source code or binary, they can be used by developers as soon as they’re
ready to push their work to the main branch. This means the security-related issues can be identified early on
in the software development life cycle (SDLC) when the cost of fixing such issues is significantly reduced.
Dynamic Application Security Testing (DAST)
• DAST examines an application as it runs to find flaws and weaknesses that a malicious party could exploit.
DAST is considered to be a black-box method since the tools do not know how the application was
constructed.
• DAST tools do not need access to your application’s source code/binary; they just need to be able to run your
application.
• DAST is best performed on a running application in an environment as close to production as possible. 35
​

What is Continuous Security or SecOps

In regulated industries, security plays a critical role in the entire software development process and the
security touchpoints are interwoven with the CI/CD pipeline to make security as the underlying DNA of
the delivery pipeline

Acceptance Deploy to Deploy to

Build Test Smoke Testing
Test staging production

Static Application Security Testing Software Composition Package Signature

Package Scanning
(SAST) Analysis (SCA) Signing Verification

Container runtime
Security
DAST
Container Image Container Image Container Image
Plain
Code Security Dependency
Text License Risk
vulnerability Hotspots Analysis
Secret Binary non containerized Non Container Non Container
artefact (exe, war etc.) ized artefact ized artefact

36
Tools
​ for SecOps
There are several tools available in the market to help with SecOps. But it is important for one to carefully select the right tool that matches
the needs of development and infrastructure.

SAST

SCA

Container
Security
DAST

37
 ​

When DevOps moves ahead with Security…

In organizations with traditional delivery methodologies…

Organizations face declining

1 There is always a lack of 2 Developers lack the right 3 Ecosystem poisoning delivery speeds, and experiment
continuous security security insight proliferates and innovate less
testing
When Security is not integrated into DevOps,
Security testing is often conducted The lack of front-line security External/Internal repositories more and more defects are discovered by

The Impact
during the final stages of intelligence impedes developer's may lack the necessary monitoring activities during the CD cycles.
deployment, that does not always ability to address security issues controls to prevent bad Consequently, defect fixing takes precedence
detect the full scope of security and make effective decisions while components from being over feature building and technical debt rises
defects; as a result, organizational coding introduced or remaining in the which curtails innovation. Continuous Integration
pressure increases on developers ecosystem and being reused (CI) cycles become clogged with a growing
to fix their code and release on across the application portfolio. backlog of defects that take developers away
time. from building new products and features.

Shift Security left: DevSecOps as the Security DNA for DevOps

Shifting security and accountability “left” in the development cycle enables continuous compliance check of the underlying code
and real time insight of the threats that empowers business and IT to be always on top of the organizational security need and
external compliance requirements

38
 Pre-requisites to
​
adopt
DevSecOps
Model
 ​

Pre-requisites to adopt DevSecOps Model

Automate the process as Follow the DevOps Train to code securely Evaluation of Current Security
much as possible methodology measures and concluding what to
do to overcome problems

Integrate the security to Monitoring Continuous Analyse code and do a Mandatory Security at
DevSecOps and adopting Integration and vulnerability Assessment Every Stage
the right DevSecOps Continuous Delivery
toolset

40
​

The DevSecOps Big Picture

Program Layer Value Stream Management

App 3
App 2
App 1

Continuous Continuous Continuous Infrastructure Continuous Continuous

Planning Integration Deployment Management Testing Monitoring

Continuous Feedback

Continuous Security

Continuous Governance

Application Layer In Scope

41
​

The Art of Version Control & Branching-Merging

Principle: “Everyone commits to baseline everyday”

Unit Test and

Daily code Code
Local Copy CI Coverage Build
push
of the repo Pipeline
Raise Pull Request Auto update JIRA
to merge into User Story/Task
“develop_feature”

User1
Real time code quality Daily code Branch Protection(Min
Local Copy
scan in IDE push Gated Check-in approving review, deny stale
of the repo
PR, signed commits, privilege
merge)

User2 Peer Review of

Real time code quality Code along with
scan in IDE automated
review 42
Indicative
​ CI workflow | Containerized Applications
Source Code Mgmt.
The CI workflow works on the principle of “Shift-left”
Perform Dependency
Perform Static Code Perform Static Code
Developer IDE Secret Scan Analysis & License Perform Unit Test
Quality Review Vulnerability Review
Conflict Check
Pre-commit code
review through Perform repository scan Perform static code Perform vulnerability Perform dependency
review with a quality gate Invoke unit test suite
PR/IDE based for passwords, keys, scanning of the analysis & license conflict
tokens etc. configuration codebase check
plugins

Control gate to break

pipeline on secret scan Break pipeline on Quality Control Gate to break pipeline Control Gate to break pipeline Control Gate to break pipeline
failure Gate failure on Threshold Breach on SCA breach on Coverage %

Project Root
• App Feedback Channel
Control Gate to break
• Dockerfile Control Gate to break Control Gate to break
pipeline on image
pipeline on validation failure pipeline on build failure
vulnerability
• Jenkinsfile
• docker- Containerize Perform Dockerfile
Perform Image Sign Perform Image Scan Perform Build
compose/k8s Application validation
manifest files
for deployment Scan the application Build the codebase with
Sign the image Create application docker Validate Dockerfile best
image generated against appropriate build tool
generated to ensure image practices
base image & other integrated with the
image integrity
vulnerabilities pipeline

Push to Container Repository

Push signed image to the

container repository
43
​
Indicative CI workflow | Legacy Applications

Source Code Mgmt. The CI workflow works on the principle of “Shift-left”

Perform Static Code Perform Static Code Perform Dependency

Developer IDE Secret Scan Perform Unit Test
Quality Review Vulnerability Review Analysis

Pre-commit code
Perform repository scan Perform static code Perform vulnerability
review through Perform dependency Invoke unit test suite
for passwords, keys, review with a quality scanning of the
PR/IDE based analysis
tokens etc. gate configuration codebase
plugins

Control gate to break

pipeline on secret scan Break pipeline on Quality Control Gate to break pipeline Control Gate to break pipeline Control Gate to break pipeline
failure Gate failure on Threshold Breach on SCA breach on Coverage %

Project Root
• App Feedback Channel
Control Gate to break Control Gate to break
• Jenkinsfile
pipeline on validation failure pipeline on build failure

Build & artifact

Push to artifact repository Artifact signing
creation

Digital Build the codebase with

Push signed artefact to signing(signtool/jarsigner) appropriate build tool
artefact repository of the build artefact to integrated with the
ensure artefact integrity pipeline and create the
deployable artefact

44
Indicative
​ CD workflow | Containerized Applications
Container Registry

Post-Deployment
SCM Perform Pre-Deployment
Pull Image Validate Signature Deployment Validation with auto
Check
rollback

Project Root Perform pre-deployment Validate the deployment

Pull application Image Validate image Deploy the application
validation checklist (PSP, by checking the container
• Jenkinsfile from container registry signature image to the target
Network Policy, NS, health & perform rollback
• docker- cluster (Swarm/k8s)
Resource etc.) if deployment is not
compose/k8s stable
Break pipeline on Control Gate to break Control Gate to break
manifest files Control Gate to break
signature validation pipeline on deployment pipeline on deployment
for deployment failure
pipeline on validation failure
failure inconsistency

Feedback Channel

Control Gate to break CT Control Gate to break CT

pipeline on performance test pipeline on coverage % Control Gate to break
failure deviation pipeline on DAST
failure

Container Runtime Invoke Performance Test Invoke Test DAST

Security Suite Automation Suite

Invoke performance Invoke API/UI test Perform Dynamic

Monitor container
test suite automation suite Application Security
runtime behavior
Testing

Operate Pipeline CT Pipeline

45
Indicative
​ CD workflow | Legacy Applications

Container Registry

Post-Deployment
SCM Perform Pre-Deployment
Pull Artefact Validate Signature Deployment Validation with auto
Check
rollback

Project Root Perform pre-deployment Validate the deployment

Pull deployable artefact Validate artefact validation checklist Deploy the application by checking the
• Jenkinsfile from artefact registry signature (Server reachability, artefact to the target deployment health &
• Deploy script deployment mount point system perform rollback if
etc.) deployment is not stable
Break pipeline on Control Gate to break Control Gate to break
Control Gate to break pipeline on deployment
signature validation pipeline on deployment
pipeline on validation failure inconsistency
failure failure

Feedback Channel
Control Gate to break CT Control Gate to break CT
Control Gate to break
pipeline on performance test pipeline on coverage %
pipeline on DAST
failure deviation
failure

Invoke Performance Invoke Test DAST

Monitoring
Test Suite Automation Suite

Invoke performance Invoke API/UI test Perform Dynamic

Monitor Application
test suite automation suite Application Security
performance, log and
Testing
infrastructure

Operate Pipeline CT Pipeline

46
Continuous
​ Deployment Strategies
In an enterprise level, deployment and release follow multiple patterns and the DevOps platform should seamlessly support
the different deployment patterns

The best choice comes down to the needs and constraints of the business and the application owners and the most
critical considerations to be followed in choosing the deployment pattern are as follows -

• What degree of downtime is acceptable?

• What are the costs constraints imposed on the business?
• Does the team have the right skills to undertake complex rollout and rollback setups?

Frequently adopted deployment strategies

Recreate Blue-Green Canary Ramped Shadow A/B

47
With
​ multiple deployment patterns in place, a comparison across
business risk, duration, complexity and cost is often helpful
Business Rollout Parallel Rollback
Pattern Description Risk Cost Duration Complexity Operations Complexity
Recreate Fully scale down the existing application
before you scale up the new application

Blue-Green Perform two identical deployments of

your application, then the traffic is
switched to the new.

Canary Partially roll out an application version to

a group of “canary” users to evaluate its
performance

Ramped New application is slowly rolled out to

replace the existing version

Shadow Deploy and run a new application

version alongside the current version

A/B Test Route a subset of users to new

application version based on routing
rules

High to Low

48
 Axis Centralized
​
Continuous
Integration and
Continuous
Deployment
Platform
The
​ Axis centralized DevSecOps platform - HOI

Acunetix for
Checkmarx JFrog Twistlock for
Bitbucket for Dynamic
Jenkins for Continuous for Static Artifactory for container
code Application
Integration/Deployment Application artifact static and
versioning Security
Security repository runtime Testing
Testing security

50
The
​ Axis centralized DevSecOps platform architecture

JIRA Bitbucket Jfrog Application Cluster

Jenkins
(master)

EKS Cluster

Other
BOF Thanos Slaves
Application Specific Build & Deployment Agents

On-
Prem

On-Prem deployment agents On-Prem deployment targets SecOps Tools

51
Axis project archetypes and pipeline mapping
No CI possible, CD can be adopted
Both CI and CD can be adopted
for continuous deployment
with more control on code quality
1 2 3

​Code is developed in OEM ​Code is developed in OEM ​The code is developed, maintained
environment environment and deployed in Axis environment
Package(Image, ear/dll) uploaded to Code uploaded to Axis Code Code is committed to Axis Code
Axis artefact repository for deployment Repository Repository

Axis Artifact Registry Axis Code Repository Axis Code Repository

Microservice Legacy Microservice Legacy Microservice Legacy

1 Application 1 Open Source 1 1 1 1
Image Scan for Dependency & Microservices Legacy CI Microservices Legacy CI
vulnerabilities License Scan CI pipeline pipeline CI pipeline pipeline
triggered from triggered from
Artifactory Artifactory

2 2 2 2
2 2 Microservices Microservices Legacy CD
Legacy CD Legacy CD
Microservices CD pipeline CD pipeline pipeline
pipeline pipeline
CD pipeline

Axis HOI DevSecOps Platform

​HOI : Higher Order Infrastructure 52

Application On-Boarding Workflow
​

Assessment Design Implementation Rollout

Application team Application team
• Classify the application based on • Application team to perform
builds the pipeline takes the
Invitation to the application team pipeline archetype the production environment
based on the ownership of the
owner with the pre-assessment • Design the to be state based on the template shared specific configurations pipelines,
checklist target archetype prototype and build with support from Pipeline is replicated • Replicate the pipeline in DevOps CoE to
Share & sign-off the pipeline template DevOps CoE in for other services, if production provide support in
the assessment UAT applicable an on-demand
report /As Is report basis

Validation and if • Define the security scan DevOps team to review DevOps , Cloud and
found fit proceed thresholds the implementation and Infosec teams to provide
with the provide the handholding the required support per
assessment • Handover the pipeline
template and code required need basis
snippets

The onboarding process is expected to be completed in 10 – 12 weeks of time

53
 2 3

DevSecOps CI Pipeline for Microservices

Code Quality &
Vulnerability Scan Package Package Sign
Plan
• Secret Scan (Checkmarx) • Docker linting and minimization of
• Threat Modelling image size • Docker Image signing (Notary)
• Code Quality (SonarQube)
• Acceptance criteria defined • Use of authorized docker base
• SAST (Checkmarx)
with probable security images
• Open Source dependency vulnerability
vulnerabilities • Follow dockerfile best practices
check (Checkmarx OSA)
• Vertical slicing of US to
• Publish results
include infra level security

CI Pipeline

Secret Management
(Gemalto/HashiCorp Vault)
Code Build Push package to secure
registry & Scan
•
•
Branching strategy
Branch Protection (PR based
• Production Grade Build Package Scan
Configuration
merge instead of manual) • Configuration File Check • Push versioned application image to the
• Docker Image Scan secure container registry (Jfrog)
(Twistlock)
• Publish result

Must Have Good To Have Nice To Have Security touchpoints

 2 3

DevSecOps CI Pipeline for Legacy (Non containerized)

Code Quality &
Vulnerability Scan Package Push package to secure repository
Plan
• Secret Scan (Checkmarx) • Package the build artefacts into • Push versioned artefact to the secure artefact repository (Jfrog)
• Threat Modelling ear/war/dll/etc. following semantic
• Code Quality (SonarQube)
• Acceptance criteria defined versioning
• SAST (Checkmarx)
with probable security
• Open Source dependency vulnerability
vulnerabilities
check (Checkmarx OSA)
• Vertical slicing of US to
• Publish results
include infra level security

CI Pipeline

Secret Management
(Gemalto/HashiCorp Vault)
Code Build
Package Sign
• Branching strategy • Production Grade Build
• Branch Protection (PR based Configuration
merge instead of manual) • Configuration File Check • Sign the package with
jarsigner/signtool for
jar/ear/exe/dll
• Publish Result

Must Have Good To Have Nice To Have Security touchpoints

 1 2 3

DevSecOps CD Pipeline for Microservices

Download Automated Web Application

Package Deployment vulnerability Testing
• Automated deployment based on the
• Download package from strategy defined
• Dynamic Application Security Testing Operate
(Acunetix)
secure repository • Deployment health check
2 3 • Verify signature (Notary)
From CI
pipeline

CD Pipeline

Vulnerability
Scanning Secret Management
1 (Gemalto/HashiCorp
From external Vault)
partners Pre-Deployment Check Post-Deployment
(docker
Check Container
image)
• Verify Kubernetes namespace and system Runtime Security*
resources (CPU, RAM) are available for
• Monitor application deployment status
application.
• Rollback if deployment fails • Runtime visibility into
• Verify required pod security policy, network
• Clean up artefact if deployment failed containerized environments
policy, etc. are present on Kubernetes.
• Notification to relevant stakeholders (TwistLock)
• Verify RBAC is enabled for the cluster

Must Have Good To Have Security touchpoints

* In Production Environment
 1 2 3

DevSecOps CD Pipeline for Legacy (Non-containerized)

Automated Web Application

Download Package Deployment vulnerability Testing
• Stop existing application
• Download package from secure • Automated deployment (copying the • Dynamic Application Security Testing
repository artefact/app server CLI/build plugin) (Acunetix)
• Verify signature (jarsigner/signtool) • Deployment health check
2 3 for artefacts generated through CI
From CI pipeline
pipeline

CD Pipeline
Vulnerability
Scanning
Secret Management
1 (Gemalto/HashiCorp
From external Vault)
partners Pre-Deployment Check Post-Deployment
(jar/ear/exe/dll
/sln etc.) Check
• Server reachability
• Verify server resources (CPU, RAM, Diskspace,
• Monitor application deployment status
etc.) are available for application
• Rollback if deployment fails
• Deployment dependencies are met
• Clean up artefact if deployment failed
• Notification to the relevant
stakeholders

Must Have Good To Have Nice To Have Security touchpoints

57
 The Pipeline Topologies – Microservice/Legacy based applications
​
JFrog Multiple CD pipelines for
Bitbucket
separation of concern
MS 1 CD
#Design Option I

Webhook MS 1 CI Pipeline (UAT) Pipeline (UAT)

Application
MS 1 Repo MS 1 UAT
Image 1 MS 1 CD
Pipeline (Prod)
MS 2 Prod
MS 2 CI Pipeline (UAT)
MS 2 Repo Application MS 2 CD
Webhook Image 2 Pipeline (UAT)
MS 2 UAT
MS 2 CD
Pipeline (Prod)
MS 2 Prod
Microservice Legacy

Bitbucket JFrog
MS 1 CD
#Design Option II

Trigger Pipeline (UAT)

Application
MS 1 Repo MS 1 UAT
Image 1 MS 1 CD
Pipeline (Prod)
Single CI Pipeline MS 2 Prod
MS 2 Repo Application MS 2 CD
Trigger Image 2 Pipeline (UAT)
MS 2 UAT
MS 2 CD
Pipeline (Prod)
MS 2 Prod
Microservice

58
 Developer
To-Be CI Pipeline
Trigger Notification Channel
Code Commit

Bit Bucket

Break Pipeline Break Pipeline Break Pipeline

Break Pipeline

Push Docker
Code Pull Build Docker
Yes Yes Image to Non-
Yes Yes Image Prod Account
Perform
Build
Met
1 2 3 4
Met
SAST 6 7 Twistloc
k SLAs
9 10
SLAs?

Secret Software Unit Testing

Trigger Composition & Static Static Docker File Image Image
Scanner Code Linting
Scheduled Analysis Code Scanning Signing
Coverage Analysis
Pipeline Parameters
• Branch Junit and
• Environment Cobertura
• Vault URL and Path
• Vault Credentials
• Image Name

For SAST and

Secret Scanner Tool SCA Tool Docker File Linter
OSA scan
Axis Bank Document Classification | Confidential
SecOps Toolset
 Pipeline
1 Approval from
SM or PO
No

2 Bit Bucket
To-Be CD Pipeline for Prod
Environment
Pull Pipeline and
Pipeline Parameters deployment manifests
• Environment
• Vault URL and Path
• Vault Credentials
Break Pipeline 3 Pre-Deployment
Checklist
• Image Name and Fails
Tag Pull Docker Image from
4 Pull Docker Image from
Prod Account

UAT Account
Trigger Notification Channel

Image sign
5 Tag and Push Docker verification
Image to Prod Account Enabling RBAC
PSP Policies
Rollback Namespace Isolation Cluster
Deployment Pod Network policies Monitoring
Fails 6 Cluster Hardening
Trigger Fluentd Agent
Deployment CloudWatch Agent
Dynatrace Agent Logging APM
Prod
Cluster
Amazon
7 EKS
Key & Secret
Management

Post-Deployment Web App

Checklist Security Testing
8

9
Run-time Container
Scanning
 Tools Integrated as part of CI/CD in Axis Platform:
• Jenkins HOI Centralized (To automate CI/CD process)
• Jenkins slave (An instance running on AWS cloud used to run the application specific jobs)
• CD tools: Amazon EKS - for deployments with kubectl or helmcharts & various others (Jboss
CLI, OCP cluster, Azure CLI)
• BitBucket Centralized SCM (For version control of source code)
• Checkmarx (For SAST and OSA scan)
• NodeJs, NPM, Maven & various others (To build the application and provide the final artifact)
• Jfrog Centralized Artifactory (To store various versions for artifacts for application)
• Notary (For Signing Docker images)
• Twistlock (For docker image scan)
• Acunetix (To run DAST scan)
• Gemalto, Hashi-Corp Vault (For Key and Secret Management)*
• Cluster Monitoring & Logging (Cloud Watch, Elasticsearch, Fluentd, and Kibana)*
• Dynatrace (Application performance monitoring)*
• Cypress (For functional testing)*
• Email & Teams Notifications

Note: * tools mentioned above are Application specific and depends upon Application Architecture
Jenkins DevSecOps CI pipeline:
Jenkins DevSecOps CD pipeline:
Jenkins DevSecOps CICD pipeline:
Pipeline
​
Control Gates and Thresholds
CI Pipeline CD Pipeline

Pre-Deployment
Checkmarx Twistlock Check
Cluster health, Namespace,
Resource Quota. PSPs,
Static Application Dependency Image Scan for server reachability etc.
Security Testing (SAST) Analysis** Microservices

Non-Prod
Pipeline

High Medium Low High Medium Low Critical High Medium Low Verify Image
Control Gates is enabled in the pipeline as per the threshold Control Gates is enabled in the Signature
defined in infosec policy pipeline as per the threshold
Prod
e.g. block all high & medium vulnerabilities in lower environments and defined in infosec policy Pipeline
block high, medium and low for production* e.g. block all critical & high
vulnerabilities in lower environments Break pipeline Break pipeline on
and block critical, high, medium and
on signature checklist failure
low for production
mismatch
Any breach in threshold breaks the pipeline and notifies the team with along with the scan report from the respective tool

Threshold values are defined at the pipeline level and are Threshold values are defined at
protected against accidental change the server level * Example threshold, not actual
65
** Currently not operational
 Success Stories
​
 Success Stories so far…
​
Branch of Future Branch of Future – UAT Pipeline

2.48 3.23
5 5
Continuous Integration AS-IS Current
6
5
4
3
2
1
0
Branch of Future – SAST & Image Scan integration with the Pipeline

As-Is Score Current Score

2.41 4.25
5 5
AS-IS Current
Continuous Deployment
6
5
4
3
2
1
0 DAST integration Teams integration

As-Is Score Current Score

67
Success Stories so far…contd.
​

68
 • Uniqueness of DevOps and its practices to make Axis
High Performing

Summary • DevOps differ from traditional IT Operations and how

the world is embracing the culture
• Integrating security as the DNA of DevOps, resulting
​ DevSecOps, as the recommended delivery
methodology in highly regulated industries
• The Big Picture for DevSecOps – Continuous
Integration and Deployment as our primary scope
• Axis Centralized DevSecOps Platform – HOI
• Project archetypes and respective pipeline mapping,
architecture and topology
• Pipeline governance and control gates
• Application onboarding journey and touchpoints in Axis
HOI platform
• Success story for Branch of Future and more….
 Q&A
​
Thank You