PRACTICAL DEVOPS TOOLS :

Preparing for the LPI DevOps Tools Engineer

Certification

v1.3

FONGAN TOUSSIDO Gilbert 2022

 HELLO!
2

I am FONGAN TOUSSIDO Gilbert,

Cloud DevOps Engineer with five (05) years of experience

across AWS and Azure Cloud platforms.

Active member of the LPI (Linux Professional Institute).

3

SUMMARY

PART I : SOFTWARE ENGINEERING PART IV: CONFIGURATION MANAGEMENT

Module I-1 : Modern Software Development

Module IV-1 : Ansible
Module I-2 : Standard Components and Platforms for Software
Module IV-2 : Other Configuration Management Tools
Module I-3 : Source Code Management
Module I-4 : Continuous Integration and Continuous Delivery

PART II : MACHINE DEPLOYMENT

PART V: SERVICE OPERATIONS

Module II-1 : Virtual Machine Deployment

Module V-1 : IT Operations and Monitoring
Module II-2 : Cloud Deployment
Module V-2 : Log Management and Analysis
Module II-3 : System Image Creation

PART III: CONTAINER MANAGEMENT

Module III-1 : Container Usage

Module III-2 : Container Deployment and Orchestration
Module III-3 : Container Infrastructure
4

GETTING STARTED

Always focus on the objective exams as they

describe what will or will not be covered in the
exam.

Please take notes of the key concepts and

commands related to the various topics covered,
as you will often be asked to provide commands
and sample configurations on the exam.

Prepare the environment of its physical machine

to be virtualized to carry out various
manipulations on the technologies approached
through a solution such as Virtualbox.

Use the Gitlab repository to make practical

examples of all the open-source DevOps tools in
this exam.
5

Instructions for use

Target : This docs is intended for Summary & Pre-requisites

Students who are interested to start their career in DevOps by It is presented a technological development environment to initiate the
preparing the LPI DevOps Tools Engineer Certification. reader to the use of DevOps tools to enable the efficient management of
Junior Engineers who work as System Administrators, infrastructure through practical cases and thus will help in the
Software Engineers, DevOps and Cloud Engineer who need to preparation of the certification,
improve their skills in a specific area.
It is necessary to have basic knowledge of Linux.

More info and updates

https://learning.lpi.org/en/learning-materials/all-materials/#devops-version-10

https://gitlab.com/GilbertFongan/devops-book-labs
6

Instructions for use

Clone the Gitlab repository of the book in order to have all the codes in your local environment
7

Instructions for use

Clone the Gitlab repository of the book in order to have all the codes in your local environment
8

PART I.
Software Engineering
…
9

MODULE I-1 : Modern Software

Development
10

PLAN

SOFTWARE DEVELOPMENT LIFE CYCLE (SDLC) :

-- Waterfall

-- Iterative
V-model

- Agile model
DevOps

API CONCEPTS AND STANDARDS :

-- REST API,

- CORS Headers,
CSRF Token

MODERN SOFTWARE ARCHITECTURE :

-- Monolithic

-- Service Oriented Architecture

Microservice
Serverless

MODULE I-1
11

SDLC

Software Development Life Cycle (SDLC) is a process used by the software industry to design, develop and
test high quality softwares. The SDLC aims to produce a high-quality software that meets or exceeds
customer expectations, reaches completion within times and cost estimates.

The SDLC methodology focuses on the following phases of software development :

Requirement analysis
Planning
Software design
Software development
Testing
Deployment
Following are the most important and popular SDLC models:

Waterfall model
Iterative model
V model
Agile model

MODULE I-1
12

SDLC/ Waterfall model

First process model to be used widely in Software Engineering.

Linear sequential flow.

Phase in the development process begins only if the previous phase is complete. The outcome of one
phase acts as the input for the next phase sequentially.

MODULE I-1
13

SDLC/ Waterfall model

The sequential phases in Waterfall model are :
Requirement Gathering and analysis : Requirements of the system to be develop are captured and
documented in a requirement specification document

System Design : Helps in specifying hardware and system requirements and helps in defining the
overall system architecture.

Implementation : The system is first developed in small programs called units, which are integrated in
the next phase. Each unit is developed and tested for its functionality.

Integration and testing : All units developed in the previous phase are integrated into a system after
testing of each unit. Post integration the entire system is tested for any faults and failures.

Deployment of system : Once the functional and non-functional testing is done, the product is
deployed in the customer environment or released into the market.

Maintenance : Patches are released to fix some issues which come up in the client environment.
Maintenance is done to deliver these changes in the customer environment.

MODULE I-1
14

SDLC/ Waterfall model

Advantage and disadvantages are :

Advantages Disadvantages

Simple and easy to understand. Works well for

smaller projects where requirements are very well High amounts of risk an uncertainly
understood

Highly-disciplined model and Phases are completed Not a good model for complex and object-oriented
one at a time projects

Clearly defined stages. Process and results are well Dificulty to go back and change the functionnality in
documented testing stage

MODULE I-1
15

SDLC/ Iterative model

The basic idea behind this method is to develop a system through repeated cycles (iterative) and in smaller
portions at a time

Starts with a simple implementation of a subset of the software requirements

Iteratively enhances the evolving versions until the full system is implemented

Design modifications are made, and new functional capabilities are added (at each iteration)

MODULE I-1
16

SDLC/ Iterative model

Advantage and disadvantages are :

Advantages Disadvantages

Simple and easy to understand. Works well for

smaller projects where requirements are very well High amounts of risk an uncertainly
understood

Highly-disciplined model and Phases are completed Not a good model for complex and object-oriented
one at a time projects

Clearly defined stages. Process and results are well Dificulty to go back and change the functionnality in
documented testing stage

MODULE I-1
17

SDLC/ V-model
It is also known as Verification and Validation model

execution of processes happens in a

sequential manner in a V-shape

extension of the waterfall model and is based

on the association of a testing phase for each
corresponding development stage

The following illustration is a representation of the

different phases :

MODULE I-1
18

SDLC/ V-model
Advantage and disadvantages are :

Advantages Disadvantages

highly-disciplined model and Phases are completed

High risk and uncertainty.
one at a time

Not a good model for complex and object-oriented

Simple and easy to understand and use.
projects

Easy to manage due to the rigidity of the model.

Each phase has specific deliverables and a review Poor model for long and ongoing projects.
process.

MODULE I-1
19

SDLC/ Agile model

Agile SDLC model is a combination of iterative and incremental process models with focus on process
adaptability and customer satisfaction by rapid delivery of working software product.

Following are the Agile Manifesto principles

Individuals and interactions : self-organization and motivation are important, as are interactions like
co-location and pair programming.
Working software : Communication with the customers to understand their requirements, instead of
just depending on documentation.
Customer collaboration : Continuous customer interaction is very important to get proper product
requirements.
Responding to change : Focused on quick responses to change and continuous development.

MODULE I-1
20

SDLC/ Agile model

MODULE I-1
21

SDLC/ Agile model

Advantage and disadvantages are :

Advantages Disadvantages

Promotes teamwork and cross training and Suitable An overall plan, an agile leader and agile PM
for fixed or changing requirements practice is a must without which it will not work.

Strict delivery management dictates the scope,

Delivers early partial working solutions and Minimal
functionality to be delivered, and adjustments to
rules, documentation easily employed.
meet the deadlines.

Depends heavily on customer interaction, so if

Enables concurrent development and delivery within
customer is not clear, team can be driven in the
an overall planned context.
wrong direction.

MODULE I-1
22

SDLC/ Synthesis

Agile software development has broken down some of the silos between
requirements analysis, testing and development.

Deployment, operations and maintenance are other activities which have

suffered a similar separation from the rest of the software development process.

The DevOps movement is aimed at removing these silos and encouraging

collaboration between development and operations.

DevOps has become possible largely due to a combination of new operations

tools and established agile engineering practices,

MODULE I-1
23

SDLC/ DevOps
DevOps combines development (dev) and operations(ops) to increase the efficiency, speed and security of
software development and delivery compared to traditional processes.
It is defined as a software engineering methodology which aims to integrate the work of software
development and software operations teams by facilitating a culture of collaboration and shared
responsibility.
These four (04) key principles can improve the organization's software development practice :

Automation of the Sofware development lifecycle

Collaboration and communication
Continuous improvement and minimization of waste
Hyperfocus on user needs with short feedback loops.

MODULE I-1
24

SDLC/ DevOps benefits

The benefits of DevOps are :

MODULE I-1
25

SDLC/ DevOps and Agile Model

The following show the difference between Agile and DevOps:

Parameter/Software Development
Agile DevOps
Technologies

Purpose Manage end-to-end engineering

Manage complex projects
processes

Task Focuses on constant changes Focuses on constant testing and delivery

Implementation Possible within a range of tactical Collaboration (doesn't have any

frameworks (sprint, safe and scrum) commonly accepted framework)

Feedback By the customer By the internal team

Goal Gap between customer need and Gap between development + testing and
development & testing teams Ops

Advantage Shorter development cycle and improved Supports Agile's release cycle
defect detection

Tools used JIRA, Kanboard Ansible, Jenkins, Git

MODULE I-1
26

API Concepts and Standards

The term API is an acronym that stands for Application Programming Interface.

For example, think of an API like a menu in a restaurant.

The menu provides a list of pizzas you can order, along with a description of each pizza.

You don't know exactly how the restaurant prepares this food, and you don't really need to.

MODULE I-1
27

API Concepts and Standards

MODULE I-1
28

API Concepts and Standards / REST API

REST is an acronym for REpresentational State Transfer and an architectural style for distributed
hypermedia systems.

REST is a way for two computer systems to communicate over HTTP in a similar way to web browsers and
servers.

MODULE I-1
29

API Concepts and Standards / 6 REST API Constraints

Client-Server Architecture : Client and server systems can be improved and updated independently each
other

Statelessness : All client requests are treated equally. There's no special, server-side memory of past
client activity. The responsibility of managing state is on the client.

Cacheability : Clients and servers should be able to cache resource data that changes infrequently
further improving scalability and performance.

Layered System : A client cannot ordinarily tell whether it is connected directly to the end server or an
intermediary along the way. Intermediary servers can also improve system scalability.

Code on demand (optional) : Servers can temporarily extend or customize the functionality of a client by
transferring executable code.

Uniform interface : All resources should be accessible through a common approach such as HTTP GET
and similarly modified using a consistent approach.

MODULE I-1
30

API Concepts and Standards / What is JSON ?

JSON stands for JavaScript Object Notation

JSON is a lightweight format for storing and transporting data
JSON is often used when data is sent from a server to a web page
JSON is "self-describing" and easy to understand

Syntax rules

Data is in name/value pairs

Data is separated by commas
Curly braces hold objects
Square brackets hold arrays

MODULE I-1
31

RESTful Web Service (Request)

A RESTful web service request contains :

An Endpoint URL : An application implementing a RESTful API will define one or more URL endpoints
with a domain, port, path, and/or query string for example, https://mydomain/user/123?format=json
The HTTP method : Differing HTTP methods can be used on any endpoint which map to application
create, read, update, and delete (CRUD) operations :

HTTP method CRUD Action

Returns requested data/Same as GET but does not return a body/Returns

GET/HEAD/OPTIONS read
the supported HTTP methods

POST create Returns requested data

PUT or PATCH update Updates an existing record

DELETE delete Deletes an existing record

HTTP headers : Information such as authentication tokens or cookies can be contained in the HTTP
request header.
Body Data : Data is normally transmitted in the HTTP body in an identical way to HTML <form>
submissions or by sending a single JSON-encoded data string
MODULE I-1
32

RESTful Web Service(Response)

The response payload can be whatever : data, HTML, an image, an audio file, and so on.

Data responses are typically JSON-encoded, but XML, CSV, simple strings, or any other format can be
used.
Return format could be specified in the request. For example, /user/123?format=json or
/user/123?format=xml

An appropriate HTTP status code should also be set in the response header.

MODULE I-1
33

REST challenges
Several challenges are possible :

API Versioning : API changes are inevitable, but endpoint URLs should never be invalidated when they're
being used internally and/or by third-party applications

Authentication : Client-side applications on the same domain as the RESTful API will send and receive
cookies. An API request can therefore be validated to ensure a user is logged in and has appropriate
rights.

Security : A RESTful API provides another route to access and manipulate your application.

Common best practices :

Use HTTPS
Use a robust authentication method
Use CORS to limit client-side calls to specific domains
Provide minimum functionality
Validate all endpoint URLs and body data
Avoid exposing API tokens in client-side Javascript
Block unexpectedly large payloads.
MODULE I-1
34

CORS headers and CSRF token

A CSRF (Cross-Site Request Forgery) token is a unique, secret, unpredictable value that is generated by
the server-side application

CSRF token is transmitted to the client in such a way that it is included in a subsequent HTTP request
made by the client.

CSRF tokens can prevent CSRF attacks by making it impossible for an attacker to construct a fully valid
HTTP request suitable for feeding to a victim user.

MODULE I-1
35

CORS headers and CSRF token

Cross-origin resource sharing (CORS) is a mechanism that consists of adding HTTP headers to allow a user
agent to access resources on a server located on another origin than the current site.

CORS is used to bypass a certain basic setting like SOP (Same-Origin Policy) which prohibits loading
from other servers when visiting a website

CORS does not protect against CSRF attacks or unwanted users

MODULE I-1
36

Modern Software Architecture

The architecture of a system describes its major components, their relationships (structures), and how
they interact with each other.

Software architecture and design includes several contributory factors such as Business strategy,
quality attributes, human dynamics, design, and IT environment. Software architecture # Software
design.

MODULE I-1
37

Modern Software Architecture

Software Architecture serves as a blueprint for a system. It provides an abstraction to manage the system
complexity and establish a communication and coordination mechanism among components.
Fundamental properties and define guidelines
Cross-cutting concerns and high-impact
Communicate with business stakeholders
Non-functional
Manage uncertainty
requirements
Conceptual integrity
Scope: System

Software design provides a design plan that describes the elements of a system, how they fit, and work
together to fulfill the requirement of the system. The objectives of having a design plan are as follows

Detailed properties
Communicate with developers
Functional
Individual components
requirements
Use guidelines
Avoid uncertainty
Scope: Module
MODULE I-1
38

Modern Software Architecture / Types

Monolithic Architecture Service Oriented
Different types of architectures
Architecture (SOA)

Microservices
Architecture

Serverless Architecture

MODULE I-1
39

Monolithic Architecture
A Monolithic architecture is a traditional model of a software program, which is built as a unified unit that is
self-contained and independent from other applications.

Traditional solution
Comfortable fo small teams
Interconnected and interdependent
Software self-contained

MODULE I-1
40

Monolithic Architecture

Advantage and disadvantages are :

Advantages Disadvantages

Slower development speed and not adapted to

Easy development and deployment
change for deployment

Not scalability (for individual components) and not

High performance task
reliability

Simplified and fast testing Lack of flexibility

Easy debugging Barrier to technology adoption

MODULE I-1
41

Service-Oriented Architecture (SOA)

In SOA, a service is a self-contained unit of software designed to complete a specific task.
Service-oriented architecture allows various services to communicate using a loose coupling system to
either pass data or coordinate an activity.

The defining concepts of SOA are list with high priority :

Business value
Strategic goals
Basic interoperability
Shared services
Continued improvement

MODULE I-1
42

Service-Oriented Architecture (SOA)

Advantage and disadvantages are :

Advantages Disadvantages

Service reusability High overhead

Easy maintenance High investment

Platform independent Complex service management

Availability, reliability and scalability Complex maintenance

MODULE I-1
43

Microservice Architecture
Microservice is a type of service-oriented software architecture that focuses on building a series of
autonomous components that make up an application. It is an architectural style that structures an
application as a collection of services that are :

Highly maintainable and testable

Loosely coupled
Independently deployable
Organized around business capabilities
Owned by a small team

MODULE I-1
44

Microservice Architecture

Advantage and disadvantages are :

Advantages Disadvantages

Increased agility Increased complexity

Improved workflows More expensive

Decreased the amount of time it takes to improve

Greater security risks
production

Ability to scale horizontally Different programming languages

MODULE I-1
45

Comparison of Architectures

Companies that have evolved from a monolithic approach to microservices :

MODULE I-1
46

Comparison of Architectures

In resume :

Monolithic apps consist of interdependent, indivisible units and feature very slow development speed.
SOA is broken into smaller, moderately coupled services and features slow development.
Microservices are very small, loosely coupled independent services and feature rapid continuous
development.

MODULE I-1
47

Serverless Architecture

Serverless architecture is an approach to software design that allows developers to build and run services
without having to manage the underlying infrastructure.

Code execution is managed by a server

Serverless doesn't mean "no server"
Third-party cloud service like AWS/Azure takes full responsibility for these servers

MODULE I-1
48

Serverless Architecture

Advantage and disadvantages are :

Advantages Disadvantages

Easy to deploy Security dependent on the service provider

Privacy & Vendor lock-in : Shared resources in

Enhanced scalability
Cloud

Lower costs Not for long-term tasks

Accuracy on function development Complexity of troubleshooting

MODULE I-1
49

Which Architecture?

Which architecture to choose?

MODULE I-1
50

Quizz

1. Which of the following software development 2. A service should be provided to arbitrary clients on the
process are Agile? (Choose Two correct answers.) Internet using HTTPS. Any standard clients on the Internet
should be able to consume the service without further
configuration. Which of the following approaches can be used
Kanban to implement these requirements? (Choose Three correct
Rational Unified Process answers.)
V-Model
SCRUM Configure the Web servers to not use a server certificate
when serving HTTPS
Waterfall
Generate self-signed certificates during the deployment
of each backend server
Use a certificate issuing service to request certificates
during each server deployment
Use a load balancer that decrypts incoming requests
and passes them on HTTP
Install a wildcard certificate and the respective private
key on all the backend servers.
MODULE I-1
51

MODULE I-2 : Standard

Components and Platforms
for Software
52

PLAN

INFRASTRUCTURE:
DATABASE

-- Mutable -- SQL
Immutable
- NoSQL
Serverless
DATA STORAGE :

--
MESSAGE QUEUES

Object

-
CLOUD COMPUTING
Block
File -- Key features
DATA STRUCTURE: -- Benefits
Service model
Deployment model
CAP & ACID

MODULE I-2
53

Mutable infrastructure

Mutable Server means the infrastructure will be continually updated, tweaked, and tuned to meet the ongoing
needs of the purpose it serves.

Ability to change
Updating Operating System
Updating Software

MODULE I-2
54

Mutable infrastructure

Advantage and disadvantages are :

Advantages Disadvantages

IT team does not need to build servers from scratch Configuration drift (harder to diagnose and manage
every time a change is required. each server)

Roll out updates for individual servers, which makes

Indiscrete versioning
the update process faster.

Ensure that the infrastructure used meets the Updates can fail due to several reasons. Debugging
specific needs of each user. is time consuming due to update tracking problems.

MODULE I-2
55

Immutable infrastructure

Immutable Server means the infrastructure cannot be modified once deployed. When changes are
necessary, it is recommended to deploy afresh, add infrastructure and decommission old infrastructure.
No updates, security patches or configuration changes
New version of the architecture is built and deployed
New servers are deployed instead of updating the ones already used

MODULE I-2
56

Immutable infrastructure

Advantage and disadvantages are :

Advantages Disadvantages

Discrete versioning (Each server version is

Impossible to modify existing servers
independent of the other.)

Easier tracking, testing different servers and rolling In case of problems, servers with the same
back configuration need a complete overhaul.

Great for interdependent environments such as Externalize data storage instead of copying it to a
cloud technologies. local disk.

MODULE I-2
57

Data Storage

Data storage is the retention of information using technology specifically developed to keep that data and
have it as accessible, as necessary.

File Storage is a hierarchical storage methodology used to organize and store data on a computer hard
drive or on a network-attached storage (NAS)
Block Storage is when a category of data storage is saved in huge volumes known as blocks
Object Storage is a computer data storage architecture that manages data as objects

MODULE I-2
58

Data Storage Comparison

Object Storage File-based Storage Block-based storage

TRANSACTION UNITS Objects(files with custom metadata) Files Blocks

No in-place update support/Updates

SUPPORTED TYPE OF UPDATES Supports in-place updates Supports in-place updates
create new object versions

PROTOCOLS REST and SOAP over HTTP SMB and NFS SCSI, Fibre Channel, SATA

Relatively static file data and as Transactional data and frequently

BEST SUITED FOR Shared file data
Cloud storage changing data

Simplified access and management

BIGGEST STRENGTH Scalability and distributed access High performance
of shared files

Difficult to extend beyond the data

Doesn’t provide a sharing protocol Difficult to extend beyond the data
LIMITATIONS center
with a locking mechanism center

MODULE I-2
59

Type of Data Structure

For the analysis of data, it is important to understand that there are three common types of data structures :

Semi-Structured Data is a form of structured data that does not conform with the formal structure of
data models associated with relational databases or other forms of data tables
Structured Data is data that adheres to a pre-defined data model and is therefore straightforward to
analyze.
Unstructured Data is information that is not organized in a pre-defined manner.

MODULE I-2
60

Storage Use Case

Native cloud applications

Big data analysis
Internet of Things
Storage and distribution of rich media content
Backup and archiving

MODULE I-2
61

CAP & ACID

In normal operations, your data store provides all three functions. But the CAP theorem maintains that when
a distributed database experiences a network failure, you can provide either consistency or availability.

Consistency : All clients see the same data at the

same time

Availability : The system continues to operate even

in the presence of node failures

Partition tolerance : The system continues to

operate despite network failures

MODULE I-2
62

CAP & ACID

ACID describe the set of properties of database transactions that guarantee data integrity despite errors,
system failures, power failures, or other issues.

Atomicity : Commits finish an entire operation

successfully or the database rolls back to its prior state

Consistency : Any change maintains data integrity or is

cancelled completely

Isolation : Any read or write will not be impacted by

other reads or writes of separate transactions

Durability : Successful commits will survive

permanently

MODULE I-2
63

Database

There are following types of databases :

Relational Database : These databases are categorized by a set of tables where data gets fit into a pre-
defined category.
- The table consists of rows and columns where the column has an entry for data for a specific category
- Rows contains instance for that data defined according to the category.
- The Structured Query Language (SQL) is the standard user and application program interface.
NoSQL Database : These are used for large sets of distributed data.

- There are some big data performance issues which are effectively handled by relational databases, such

- There
kind of issues are easily managed by NoSQL databases.
are very efficient in analyzing large size unstructured data that may be stored at multiple virtual
servers of the cloud.

MODULE I-2
64

Database/Relational Database

Based on the relational model of data

Comply with ACID guarantees
Relational Database systems (RDBS) use SQL
Relational model organizes data into one or more tables
Each row in a table has its own unique key (primary key)
Rows in a table can be linked to rows in other tables by adding a foreign keys
Exemple : MysSQL (MariaDB), Oracle, Postgres, IBM DB, ...

MODULE I-2
65

Database/NoSQL Database

Not only SQL

Storing semi-structured and non-structured data
Horizontal scalability and partition tolerance
Sub-second response times are required for large volumes of data, compromising consistency and
referential integrity.
Finer control over availability
Simplicity of design

MODULE I-2
66

Database/Difference between SQL and NoSQL

MODULE I-2
67

DataBase Comparison

Pros Cons

Standardized schema Hardware

Large user community Data normalization
SQL
No code required Rigidity
ACID compliance Resource-intensive scaling

No standardized language
Continuous availability
Smaller user community
Query speed
NoSQL Inefficiency with complex queries
Agility
Data retrieval inconsistency
Cost

MODULE I-2
68

Message Queues

Message queuing allows applications to communicate by sending messages to each other. The message
queue provides temporary message storage when the destination program is busy or not connected.

Producer creates message and send it to queue which stores the message if consumer is busy
Consumer retrieve the message from the queue and start processing it
Queue temporarily locks the message to prevent it from being read by another consumer
Consumer delete the message from the queue after it completes the message processing

MODULE I-2
69

Cloud Computing/Definition

Cloud Computing "Using resources without directly owning them" is a model that allows ubiquitous,
convenient, on-demand access to a shared network and a set of configurable computing resources. "NIST"

05 Key features
03 Service model
04 Deployment model
Benefits

MODULE I-2
70

Cloud Computing/key features

Elasticity and rapid Resource pooling

evolution

On-demand service

Measurable and
billable service Universal access via
the network

MODULE I-2
71

Cloud Computing/Benefits

Benefit from massive Increase speed and agility Stop spending money
economies of scale to manage data centers

Shift from capital Go global in minutes Stop guessing

expenditures (CAPEX) to capacity
operational expenditures
(OPEX)

MODULE I-2
72

Cloud Computing/Service model

MODULE I-2
73

Cloud Computing/Service model

Infrastructure As a Service (IAAS):

Amazon EC2
Azure VM, Digital Ocean, Linode

Platform As A Service (PAAS) :

AWS ElasticBeanstalk
Azure App Services, Google App Engine

Software As A Service (SAAS) :

Amazon Rekognition, Comprehend
Gmail, Dropbox, Outlook, Zoom

MODULE I-2
74

Cloud Computing/Deployment model

PRIVATE PUBLIC
Different types of architectures

COMMUNITY HYBRID

MODULE I-2
75

Cloud Computing/Stakeholders

MODULE I-2
76

Quizz

1. Which of the following statements are true 2. An online shop needs to store information about clients and
regarding immutable servers ? (Choose Two orders. A list of fixed properties order clients and orders exists.
correct answers.) The data storage should enforce specific data types on these
properties and ensure that each order is associated with an
existing client. Which of the following cloud services is capable
In case of small changes, immutable servers of fulfilling these requirements?
must be rebuilt and redeployed.
Preparation and configuration tasks are
An Object store like OpenStack Swift.
moved from the time of deployment to the
time of building An in-memory database like MEMCACHED
Immutable servers store persistent data and A relational database like MariaDB
cannot be deleted without data loss. A messaging service like OpenStack Zaqar
Immutable servers cannot use external A NoSQL database like MariaDB
services such as databases and object
stores
All interactions with immutable servers have
to happen through shared file systems.

MODULE I-2
77

MODULE I-3 : Source Code

Management
78

PLAN

VERSION CONTROL SYSTEM:

GIT COMMAND

-- Benefits

-- Local
Centralized
Distributed
GIT BRANCHING

GIT MERGING

GIT REBASE
GIT :

-- Installation

-- Configuration
Terminology

- LifeCycle
Workflow

MODULE I-3
79

Version Control Systems

Version control systems are a category of software tools that helps in recording changes made to files by
keeping a track of modifications done in the code in a special kind of database.

Version control are sometimes referred to as :

Source code management systems (SCMS)

Version Control Systems (VCS)
Revision Control Systems (RCS)
Code Repositories

MODULE I-3
80

Version Control Systems / Benefits

Benefits of the Version Control System are :

Enhance the project development speed by providing efficient collaboration,

Reduce possibilities of errors and conflicts meanwhile project development through traceability to every
small change
Contribute from anywhere irrespective of the different geographical locations
Helps in recovery in case of any disaster or contingent situation
Inform stakeholders about Who, What, Why changes have been made

MODULE I-3
81

Version Control Systems / Types

Three types of Version Control Systems :

Local Version Control Systems :

- Contain database that kept all the changes to files under revision control.
- Keep patch sets (differences between files) in a special format on disk

Centralized Version Control Systems

- Have a single "Central" copy of your project on a server

- Commit changes to this central copy
- Never have a full copy of project locally

Distributed Version Control Systems

-Version control is mirrored on every developer's computer

- Allows branching and merging to be managed automatically
- Ability to work offline

MODULE I-3
82

Version Control Systems /Local

It is one of the simplest forms :

Like a VCS but without a remote repository => No remote server

Manage and version all the files only within your local system
All the changes are recorded in a local database
Every developer has their own computers and are not sharing anything

MODULE I-3
83

Version Control Systems /Centralized

The concept of a Centralized system is that it works on a Client-Server relationship. The repository is located
at one place and provides access to many clients.

MODULE I-3
84

Version Control Systems /Distributed

In a Distributed system every user has a local copy of the repository in addition to the central repository on
the server side.

MODULE I-3
85

Version Control Systems

Distributed Version Control Systems: contain multiple repositories. Each user has their own repository and
working copy.

To make your changes visible to others, 4 things are required:

You commit
You push
They pull
They update

MODULE I-3
86

Git

Open-source version control system

Provides strong support for non-linear development
Distributed repository model
Cryptographic authentication of history
Capable to efficiently handling small to large sized projects

Advantages Disadvantages

Complex and bigger history log difficult to

Super-fast and efficient performance, cross-platform
understand

Code changes easy and clearly tracked Does not support keyword expansion

Easy maintainable and robust Does not support timestamp preservation

MODULE I-3
87

Git installation

On Windows :
https://gitforwindows.org/
http://babun.github.io/ (Shell Emulator)

On OS X :
https://sourceforge.net/projects/git-osx-installer/
On Linux
<your package manager> install git
(apt-get, rpm,…)

MODULE I-3
88

Git configuration

Git provides the git config tool to set configuration variables

Git stores all global configurations in ".gitconfig" file located in your home directory
To set configuration values as global, add the "--global" option
Git stores values in the "/etc/gitconfig" file that contains the configuration for every user and repository
on the system
You can use the "--system" option to apply configuration and ensure you have root rights

User name $ git config –global user.name " GilbertFongan"

$ git config --global user.email [email protected]

User email

$ git config --global core.editor "notepad++.exe –multiInst -

Notepad++ as Default editor nosession"

List global configuration $ git config –-global --list

MODULE I-3
89

Git terminology

Workspace (Working directory) : contains the files just modified

Staging Area (indexed) : allows you to store selected changes to be committed
Local repository (committed/HEAD) : committed code, ready to be sent to a remote server
Remote repository : Remote server which contains publicly accessible code ( Gitlab, GitHub, Bitbucket).
Service hosting Git repos

MODULE I-3
90

Git terminology

MODULE I-3
91

Git Life Cycle

General workflow is as follows :

Clone the Git repository as a working copy

Modify the working copy by adding/editing files
Update the working copy by taking other developer's changes
Review the changes before commit
Commit changes. If everything is fine, then you push the changes to the repository (remote)
After committing, if you realize something is wrong , then you correct the last commit and push the
changes to the repository (remote)

MODULE I-3
92

Git Life Cycle

MODULE I-3
93

Git Workflow

MODULE I-3
94

Git command

git init Convert an existing unversioned project(workspace) to git repository or to create a new empty git repository. ".git" subdirectory will be created

Download an existing git repository to your local computer. git clone -b branch_name <git url>: The -b argument lets you specify
git clone
a specific branch to clone instead of the branch the remote HEAD is pointing to, usually the master branch.

Current branch | Files that have differences between Workspace ↔ Staging area (Untracked(new) files and Unstaged changes) | Files that have
git status
differences between Staging ↔ Local Git Repository (Uncommitted changes)

git add Add changes in the workspace to the staging area. git add <file-name> or git add . to add all files

git commit Add changes in the staging area to the local Git repository. git commit: Staging area → Local git repository | git commit -a: Workspace →
Local git repository (Untracked files are not included, only those that have been added with git add at some point) . git commit -m
‘commit message’

git pull Update local git repository from the corresponding remote git repository. git pull <remote> <local>:Local git repository ← Remote git
repository

git push Add changes in the local git repository to the remote repository. git push <remote> <local>: Local git repository → Remote git repository

git branch List all local branches. git branch -a: List all remote branches as well | git branch -d <branch>: Delete the specified branch | git
branch <new branch>: Create a new branch

git checkout Navigate between different branches. git checkout <branch> | git checkout -b <new branch>: Create a new branch from your
current branch and switch to it.

git merge Integrate changes from multiple branches into one. git merge <branch>

MODULE I-3
95

Git command
Manage connections to remote repositories. It allows you to show which remotes are currently connected, but also to add new connections or
remove existing ones.
git remote
git remote -v: List all remote connections | git remote add <name> <url>: Create a new remote connection | git remote rm
<name>: Delete a connection to a remote repository | git remote rename <old name> <new name>: Rename a remote connection

Update local git repository from the corresponding remote git repository. Git fetch does not change your workspace, it keeps the fetched content
separate until it is merged. git fetch <remote> <local> | git checkout <remote>/<local>: To view the change | git fetch vs git
git fetch
pull: git pull = git fetch + git merge

git stash Takes your uncommitted changes (staged and unstaged), saves them for later use

git fork It is a copy of a repository. It allows you to feely experiment with changes without affecting the original project.

git head HEAD is a reference to the last commit in the currently check-out branch

git revert Revert some existing commits. Given one or more existing commits, revert the changes that the related patches introduce, and record some
new commits that record them. This requires your working tree to be clean (no modifications from the HEAD commit).

git reset Reset current HEAD to the specified state. git reset HEAD~ --hard to remove the last commit

git log Display committed snapshots.

git cherry-pick Sometimes you don't want to merge a whole branch into another, and only need to pick one or two specific commits (Cherry picking).

git diff Show changes between commits, commit and working tree

MODULE I-3
96

Git command
git blame Show what revision and author last modified each line of a file

git tags Ability to tag specific points in a repository's history as being important (v1.0, v2.0)

git rebase Involves moving code to a new base commit or combining a sequence of commits

git squash To squash or regroup previous commits into one. This is a great way to group certain changes together before sharing them with others.

.gitignore A text file which tells which files and folders to ignore in a project. A local ".gitignore" file is usually placed in the root directory of a project. You
can also create a global ".gitignore" file and any entries in that file will be ignored in all of your Git repositories.

MODULE I-3
97

Git / Recovering from mistakes

To copy a file from the working directory to the staging area, we use git
add.
To save the staging area in the git repository and create a new commit, we
use git commit.
To copy a file from the Git repository to the staging area, we use git reset.
To copy a file from the staging to the working directory (thus deleting the
current modifications), we use git checkout.
To view the changes between the working directory and the staging area,
we use git diff.
To see the changes between the staging area and the last commit, we use
git diff --cached.

MODULE I-3
98

Git branching / Commits

1 2

4 3

MODULE I-3
99

Git branching
1 2

4 3

MODULE I-3
100

Git branching

1 2

MODULE I-3
101

Git basic merging

1 2

4 3

MODULE I-3
102

Git basic merging

1 2

MODULE I-3
103

Git basic merging

MODULE I-3
104

Rebase and merge

Rebasing and merging are both designed to integrate changes from one branch into another
branch but in different ways.
Merge is the result of the combination of commits in feature branch
Rebase add all the changes in feature branch starting from the last commit of the master
branch

Rebasing a feature branch into master leads to move the base of the feature branch to master
branch's ending point.
Merging takes the contents of the feature branch and integrates it with the master branch. As a
result, only the master branch is changed. The feature branch history remains same.
Merging adds a new commit to your history.

MODULE I-3
105

Rebase and merge

MODULE I-3
106

Quizz

1. Which of the following git commands is used to 2. Which of the following information is contained in the
manage files in a repository? (Choose Two correct output of “git status”? (Choose Three correct answers.)
answers.)

Changed files that will not be part of the next commit

Git rm
Locked files which cannot be edited until the lock is
Git cp released
Git move Changed files that will be part of the next commit
Git mv Unchanged files which have not been edited locally
Git copy Untracked files which are not object to version control.

MODULE I-3
107

MODULE I-4 : Continuous

Integration and Continuous
Delivery
108

PLAN

TRADITIONAL INTEGRATION:
CONTINUOUS DEPLOYMENT

CONTINUOUS INTEGRATION:
CI/CD DEPLOYMENT

--
SOFTWARE TESTING :

Manuel & automatic testing

-- Blue & Green Deployment
Canary deployment

-- White Box testing

Black Box testing
Tools JENKINS

CONTINUOUS DELIVERY -- Workflow

-- Build stages
Architecture Master & Slave
Jenkins Declarative Pipeline

MODULE I-4
109

Traditional Integration
In Traditional Integration or/software development cycle :

Each developer gets a copy of the code from the central repository
All developers begin at the same starting point and work on it
Each developer makes progress by working on their own team
Each developer add methods and functions, shaping the code to meet their
needs
Meanwhile, the other developers and teams continue working on their own tasks,
solving the problems they have been assigned

The main factors that can make these problems escalate

The size of the team working on the project
The amount of time passed since the developer got latest version of the code
from the central repository
MODULE I-4
110

Solution for problems faced in Traditional Integration

MODULE I-4
111

Benefits of CI/CD

MODULE I-4
112

Continuous Integration
Continuous Integration :
Software development practice
Developers integrate code into a shared repository frequently
Each integration is verified by an automated build and automated tests to detect integration errors as
quickly as possible
This approach leads significantly to develop cohesive software more rapidly

MODULE I-4
113

Benefits of Continuous Integration

MODULE I-4
114

Software Testing

Software Testing is a method to check whether the actual software product matches expected requirements
and to ensure that software product is defect free. Some prefer saying Software testing definition as a White
Box, Black Box Testing and Grey Box Testing.

Here are the benefits of using software testing :

Cost-Effective
Security
Product quality
Customer Satisfaction

MODULE I-4
115

Software Testing / Types

In order to better understand the concepts of Continuous Delivery and Continuous Deployment ,
we need to understand what is the Software Testing and what are its different types :

Manual Testing : Testing software only by human intervention. It may

include detailed step-by-step test cases for testing periods.
Advantages : Cost-Effective, Nothing can beat the human eye, User experience
modification, Flexibility.

Automation Testing : Testing by using automation tools.

Advantages : Scheduling, Regression testing is easy, Reusability of test scripts,
Saves time.

MODULE I-4
116

Software Testing / Approaches

There are mainly three approaches to Software Testing

MODULE I-4
117

Software Testing / Approaches

Testing Advantages Disadvantages

White Box Testing (Clear box - Complicated

- Performed at the initial stages
testing/Glass box testing) - Requires highly skilled
- More in depth
resources
- Find hidden defects
- Tools may not be readily
- Helps in code optimization
available

Black Box Testing (Behavioral - Exposes inconsistencies in

testing) specifications - Test may be tough to design
- No need to understand - Many bugs can go undetected
programming

Grey Box Testing (Mixture of - Develop more intelligent tests

white-box and black-box) - Clear goals while testing - Complicated
Overall quality of the software is - Hard to detect bugs
enhanced

MODULE I-4
118

Black box testing

Black Box Testing is further classified into two categories :

Functional Testing : verify that there are no gaps between developed features/functions and required
features/functions.

Unit Testing, Integration Testing , System Testing , Acceptance Testing.

Non-Functional Testing : Focus on the non-functional parts of the software like:

Memory leaks, Performance, Load, Scalability, Volume, Usability.

MODULE I-4
119

Software Testing / Levels

Unit Testing : tiniest testable component of the software. The aim is to

ensure the proper functioning of each unit.

Integration Testing : Individual units are grouped for testing. The aim is to
detect errors in the integrated unit's interaction.

System Testing : The integrated software is tested wholly. The aim is to

assess the conformity of the software with the established requirements
and end-to-end testing

Acceptance Testing : Software is assessed for acceptability. It is verified

against the requirements to ensure it is adequate for delivery.

MODULE I-4
120

Software Testing / Resume

MODULE I-4
121

Software Testing / Tools

Today, many software testing tools are of great importance, especially for automation testing.

MODULE I-4
122

Continuous Delivery
Continuous Delivery:
Software development practice where code changes are automatically prepared for a release to production.
Expands upon continuous integration by deploying all code changes to a testing environment after the build
stage.
Developers will always have a deployment-ready build artifact that has passed through a standardized test
process

MODULE I-4
123

Benefits Continuous Delivery

MODULE I-4
124

Continuous Deployment
Continuous Deployment:

It eliminates the human safeguards against unproven

code in live software.

It should only be implemented when the development

and IT teams rigorously adhere to production-ready
development practices

MODULE I-4
125

Continuous Delivery vs Continuous Deployment

With Continuous Delivery, every code change is built, tested and then pushed to a non-production testing
or staging environment
There can be multiple, parallel test stages before a production deployment
The difference between Continuous Delivery and Continuous Deployment is the presence of a manual
approval to update to production.

MODULE I-4
126

CI/CD DevOps Stages

MODULE I-4
127

CI/CD Deployment

Blue / Green Deployment is a technique for deployments where the existing running deployment is left in
place. A new version of the application is installed in parallel with the existing version.
When the new version is ready, cut over to the new version by changing the load balancer configuration.

MODULE I-4
128

CI/CD Deployment

Canary Deployment are like Blue/Green, although only a small amount of the servers are upgraded. Then,
using a cookie or similar, a fraction of users are directed to the new version.

MODULE I-4
129

CI/CD Tools

MODULE I-4
130

Jenkins

Jenkins :

Open-source Continuous Integration Server

Written in Java with plugins built for CI purpose
Easy to install and use
Multi-technology and Multi-platform
Widely used, extensible and free.
Used to manually, periodically, or automatically build software development
projects

MODULE I-4
131

Why Jenkins?
Easy to install :
Download one file -> jenkins.war
Run one command –> java-jar jenkins.war
Easy to use :
Create a new job-checkout and build a small project
Checking a change-watch it build
Create/fix a test – Watch it build and run/checkin and watch it pass
Multi-technology :
Build C, Java, C#, Python, Perl, SQL
Test with JUnit, NUnit, MSTest
Great extensibility :
Support different VCS
Code quality metrics , Build notifiers and UI customization

MODULE I-4
132

Jenkins Interface

Offers many types of projects :

Free style
Building a Maven Project
Pipeline and multibranch pipeline (most used for
Git projects)

Ability to clone an existing project.

MODULE I-4
133

Jenkins / Building stages

Getting the Build Environment variables

Build triggers
sources Environment Sonar configuration
Workspace cleanup
Remote triggers (hook github,
Which SCM? rules
gitlab)
Following another Build
Periodically

Post-Build Maven, gradle, ant

Publication of reports Script for the
actions Build scripts
Send notifications
Shell scripts
(Email..)
Publication of javadoc

MODULE I-4
134

Jenkins / User interface

MODULE I-4
135

Jenkins workflow

MODULE I-4
136

Jenkins / Getting the sources

Different types of sources

(Git, Subversion, CVS)

Possibility to add behaviors

MODULE I-4
137

Jenkins / Building stage

Many build tools for

several languages
Java : Maven, Gradle, Ant
.Net : MsBuild
iOS
Shell Scripts

MODULE I-4
138

Jenkins / Post-Build stage

Multiple notifications mechanisms

Slack
SMS
Email

MODULE I-4
139

Jenkins / Post-Build stage

Many types of publishable reports

Reports on statistical analysis

of the code ( Checkstyle, PMD,
Findbug, ...)

Unit test execution and

coverage report (Junit,
Cobertura, TestNG, JaCoCo...)

JavaDoc publication

MODULE I-4
140

Jenkins /Build result

MODULE I-4
141

Jenkins / Architecture

Jenkins supports Master-Slave architecture

Jenkins can run the same test case on different
environments in parallel using Jenkins Distributed
Builds.
Known as Jenkins Distributed Builds.
Which in turn helps to achieve the desired results
quickly.
All the job results are collected and combined on the
Master node for monitoring.

MODULE I-4
142

Jenkins / Architecture

Jenkins Master :
Scheduling and execute build jobs directly
Dispatching builds to the slaves for the actual execution.
Monitor the slaves (possibly taking the online and offline as required)
Recording and presenting the build results.

Jenkins Slave :
It hears request from the Master Instance
Slaves can run a variety of Operating Systems.
The job of a slave is to do as they are told to, which involves executing build jobs
dispatched by the Master.
A project can be configured to always run on a particular Slave machine/type or simply let
Jenkins pick the next available Slave

MODULE I-4
143

Jenkins / Architecture example

MODULE I-4
144

Jenkins / Setup Master and Slaves

Go to the Manage Jenkins section and scroll down to the section of Manage Nodes

MODULE I-4
145

Jenkins / Setup Master and Slaves

On New Node

Give a name for the Node, Choose the Permanent Agent option and click on OK

MODULE I-4
146

Jenkins / Setup Master and Slaves

MODULE I-4
147

Jenkins Declarative Pipeline

Declarative Pipeline is a relatively recent addition to Jenkins Pipeline, which features a more simplified and
customized syntax in addition to the Pipeline subsystems. Declarative "Section" blocks for common
configuration areas like :

Stages
Tools
Post-build actions
Notifications
Environment
Build agent
All wrapped up in a pipeline { } step,
with syntactic and semantic validation available.

MODULE I-4
148

Jenkins Declarative Pipeline

The Stages section contains one or more stage blocks.

Stages block look the same as the new block-scoped stage step
Think of each stage block as like an individual Build Step in a Freestyle job
There must be a stages section present in your pipeline block
Example
stages {
stage("build") {
timeout(time: 5, units: 'MINUTES') {
sh './run-some-script.sh'
}
}
stage("deploy") {
sh "./deploy-something.sh"
}
}

MODULE I-4
149

Jenkins Declarative Pipeline

The Agent determines where your build runs.

Current possible settings :

Agent label : :"" - Run on any Node

agent docker : "ubuntu" - Run on any Node within a Docker container of the "Ubuntu" image.
Agent docker : "ubuntu", label:"foo" - Run on a node with the label "foo" within a Docker container of the
"Ubuntu" image
Agent none - Don't run on a Node at all, manage Node blocks yourself within your stages.

There must be an agent section in your pipeline block.

MODULE I-4
150

Jenkins Declarative Pipeline

The Tools section .

Allows you to define tools to auto-install and add to the PATH

Doesn't work with agent docker:'ubuntu'.
This will be ignored if agent none is specified
The tools section takes a block of tool name/tool version pairs, where the tool version is what you've
configured on this Master
Example
tools {
maven “Maven 3.3.9”
jdk “Oracle JDK 8u40”
}

MODULE I-4
151

Jenkins Declarative Pipeline

The Environment section .

Block of key=value pairs that will be added to the environment when the build runs in.

Example

environment {
FOO = “bar”
BAZ = “faz”
}

MODULE I-4
152

Jenkins Declarative Pipeline

The Notifications and Post Build section .

Post Build and notifications both contain blocks with one or more
build condition keys and related step blocks.
The steps for a particular build condition will be invoked if that build
condition is met.
Post Build checks its conditions and executes them, if satisfied,
after all stages have completed, in the same Node/Docker container
as the stages.
Notifications checks its conditions and executes them, if satisfied,
after Post Build, but doesn't run on a Node at all.

MODULE I-4
153

Jenkins Declarative Pipeline

The Notifications and Post Build examples.

notifications {
success { hipchatSend 'Build passed' }
failure {
hipchatSend 'Build failed' mail to:'[email protected]',
subject:'Build failed',
body:'Fix me please!'
}
}
----------------------------------------------
postBuild {
always { archive "target/**/*" junit 'path/to/*.xml' }
failure {
sh './cleanup-failure.sh'
}
}

MODULE I-4
154

Jenkins Declarative Pipeline

pipeline {
tools {
maven 'Maven 3.3.9'
jdk 'oracle JDK 8u40'
}
// run on any executer
agent label:''
stages {
stage('build') {
sh 'mvn clean install -Dmaven.test.failure.ignore=true'
}
}
postBuild {
always {
archive 'target/**/*'
junit 'target/surefire-reports/*.xml'
}
}
notification {
success {
mail(to:'[email protected]', subject:"SUCCESS:
${currentBuild.fullDisplayName}", body:"Huh, we're success." )
}
failure {
mail(to:'[email protected]', subject:"FAILURE:
${currentBuild.fullDisplayName}", body:"Huh, we're failure." )
}
unstable {
mail(to:'[email protected]', subject:"UNSTABLE:
${currentBuild.fullDisplayName}", body:"Huh, we're unstable." )
}
}
}

MODULE I-4
155

Jenkins Declarative Pipeline with slave

pipeline {
agent none
stages {
stage('distribute') {
parallel (
'windows': {
node('windows') {
bat 'print from windows'
}
},
'mac': {
node('osx') {
sh 'print from mac'
}
},
'linux': {
node('linux') {
sh 'print from linux'
}
}
)
}
}
}

MODULE I-4
156

Quizz

1. Which of the following statements are true about 2. Which of the following post condition exist in a Jenkins
Jenkins? (Choose Two correct answers.) Declarative Pipeline?

Specific to Java based applications Always

Can delegate tasks to slave nodes Never
Only works on local files and cannot use Pending
SCM repositories
Success
‘Functionality’ is determined by plugins Partial.
Includes a series of integrated testing suites

MODULE I-4
157

PART II.
Machine Deployment
158

MODULE II-1 : Virtual Machine

deployment
159

PLAN

VIRTUAL MACHINE:

VAGRANT :

-- Features

- Architecture
Workflow

VAGRANTFILE:

-- Configure

-- Options
Providers

- Provisioners
Boxes

VAGRANT COMMAND

MODULE II-1
160

Virtual Machine

Virtual Machine (VM) is a software implementation of a machine (computer) that executes programs like a
physical machine.

A Virtual Machine provides an interface identical to the underlying bare hardware.

The operating system creates the illusion of multiple processes, each executing on its own processor
with its own (virtual) memory.

MODULE II-1
161

Types of Virtual Machine

There are two types :

System Virtual Machine-Hardware Virtual Machine :

Provides a complete system platform environment which supports the execution of a complete operating
system (OS).

Process Virtual Machine-Application Virtual Machine

Provides a platform-independent programming environment that abstracts away details of the underlying
hardware or operating system from software or application runtime.

MODULE II-1
162

Virtual Machine Architecture

MODULE II-1
163

Virtual Machine

Advantage and disadvantages are :

Advantages Disadvantages

Difficulty in direct access to

Familiar Interfaces
hardware

Isolation of OS and Reduction of RAM degradation with the

cost creation of new VMs

High Availability and Scalability Disk Space degradation with the

creation of new VMs

Backup with Fast Recovery Less efficient than physical

Machine

MODULE II-1
164

Vagrant for VM Deployment

Vagrant is a wrapper around Virtual Machines :

Open-source tool for building and distributing development environments.

Developed by Mitchell Hashimoto (Founder of HashiCorp)
First stable version 1.0 and was released in March 2012 (the current 2.3.0)
Create standard environments using provisioning scripts
Allow working on the same base configuration with remote access to boxes
Help test and debug remotely
Development environments managed can run on Local Virtualized Platform (Virtualbox,
VMWare) , Cloud (AWS, Azure, Openstack), and Containers (Docker)
It provides a simple and easy way to use Command-line client (Manage these environments)
and interpreter (Vagrantfiles : text-based definitions of what each environment looks like)

MODULE II-1
165

Vagrant features

Vagrant uses Providers and Provisioners as building blocks to manage

development environments.

Provisioners are tools that allow users to customize the configuration

of virtual environments.
Example : File, Shell, Chef and Puppet

Providers are the services that Vagrant uses to set up and create
virtual environments.
Example : VirtualBox, VMWare, Hyper-V, Docker, AWS...

MODULE II-1
166

Vagrant Architecture

MODULE II-1
167

Vagrant Workflow

(1) Developer Create Vagrantfile and run the VM

(2) Vagrant connects to the VirtualBox provider
to set up virtual environment.
(3) VirtualBox start the VM
(4) Vagrant executes the provisioners
(5) Provisioners install tools on VM startup
(6) Developer can access the VM through SSH
connection

MODULE II-1
168

Vagrantfile

Vagrantfile is a Ruby file that instructs Vagrant to create, depending on how it is executed, new Vagrant
machines or boxes.
Vagrant Box is considered as an image, a template from which we will deploy our future virtual machines.

Vagrant Box is a compiled Vagrantfile describing a type of Vagrant machines. A new Vagrant machines
can be created from a Vagrant Box
Vagrantfile can directly create one or more Vagrant machines
# Simple Vagrantfile example

Vagrant.configure("2") do |config|
config.vm.box = "hashicorp/bionic64"
config.vm.hostname = "node1"
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.synced_folder "../data", "/vagrant_data"
config.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--memory", 1024 * 4]
config.vm.provision :shell, path: "bootstrap.sh"
end

MODULE II-1
169

Vagrantfile

Vagrant.configure("2"): returns the Vagrant configuration object for the new box. Config alias is
used to refer to this object. The version 2 of Vagrant API is used
Vm.box : is the base box that we are going to use. The schema for box names is the maintainer
account in Vagrant Cloud followed by the box name.
Vm.hostname : sets the hostname of the box
Vm.network : Configures network
vm.synced_folder : to configures the synced folders between the host and the guest
Vm.provider : Configures settings specific to a provider. Allows overriding options for the Virtual
Machine provider. For example: memory, CPU, ...
Vm.provision : to specify the name of the file that is going to be executed at the machine creation

MODULE II-1
170

Vagrantfile Configure Network

The parameter Vm.network:

Port forwarding : Forward all requests from a service running on port 80 of the Vagrant Virtual Machine
to port 8080 of the host machine.
config.vm.network "forward_port", guest:80,host:8080

Static IP : Assign a private IP address to the Virtual Machine

config.vm.network "private_network", ip: "192.168.33.10"

By default, networks are private (only accessible from the host machine)
Use the flag "public_network" to make the guest network accessible from the LAN (Loacl Area Network)

config.vm.network "public_network", ip: "10.0.0.1"

MODULE II-1
171

Vagrantfile Options

MODULE II-1
172

Vagrant provisioners

Allows initial configuration of the VM to easily set up your VM with evrything it needs to run your
software
Important part of making VM creation repeatable
Scripts made for provisioning can typically be used to set up production machines quickly as well
Some available provisioners :

# Shell provisioning example # File provisioning example

Vagrant.configure("2") do |config| Vagrant.configure("2") do |config|

#…other configuration #…other configuration

config.vm.provision :"shell" do |s| config.vm.provision :"file" do |f|

s.inline = "echo hello Fongan" f.source= ".gitconfig"
s.path = "scripts/bootstrap.sh" f.destination = "~/.gitconfig"
end end
end end

MODULE II-1
173

Vagrant Boxes

Box is the base image used to create a virtual environment with Vagrant
A box is a compressed file containing the following :
Vagrantfile : the information from this will be merged into your Vagrantfile that is created when you run
vagrant init boxname in a folder.
Box-disk.vmdk : The Virtual Machine image.
Box.ovf : Defines the virtual hardware for the box
Metadata.json : Inform Vagrant about the provider the box work with.

vagrant box list See a list of all installed boxes on your computer

vagrant box add <name><url> Download a box image to your computer

vagrant box outdated Check for updates vagrant box update

vagrant boxes remove <name> Deletes a box from the machine

Vagrant package Packages a running VirtualBox environment in a reusable box

MODULE II-1
174

Vagrant command (1/3)

Creating a VM : Vagrant init : Initialize vagrant with Vagrantfile and ./.vagrant directory

vagrant init - m Create a minimal Vagrantfile (no comments or helpers)

vagrant init - f Create a new Vagrantfile, overwriting the one at the current path

vagrant init -box-version Create a Vagrantfile, locking the box to a version constraint

Vagrant init <boxpath> Initialize Vagrant with a specific box. To find a box, go to the public Vagrant box catalog. For example,
vagrant init ubuntu/trusty64

Starting a VM
vagrant up Starts vagrant environment (also provisions only on the FIRST vagrant up command)

vagrant resume Resume a suspended machine (vagrant up works just fine for this as well)

vagrant provision Forces re-provisioning of the vagrant machine

Vagrant reload Restarts vagrant machine, loads new Vagrantfile configuration

Vagrant reload --provsion Restart the virtual machine and force provisioning

MODULE II-1
175

Vagrant command (2/3)

Getting into a VM

vagrant ssh Connects to machine via SSH

vagrant ssh <boxname> If you give your box a name in your Vagrantfile, you can ssh into it with. Boxname works from any directory

Stopping a VM

vagrant halt Stops the Vagrant machine

vagrant suspend Suspends a Virtual Machine (remembers state)

Saving Progress
Vagrant snapshot save [options][vm-name] <name> Allows us to save the VM so that we can roll back at a later time.

MODULE II-1
176

Vagrant command (3/3)

Other tips

vagrant -v Get vagrant version

vagrant status Outputs status of the Vagrant machine

vagrant global-status Outputs status of all vagrant machines

vagrant global-status --prune Outputs status of all vagrant machines, but prunes invalid entries

Vagrant provision --debug Use the debug flag to increase the verbosity of the output

Vagrant push Vagrant can be configured to deploy code on remote central registry

Vagrant up –provision | tee provision.log Runs vagrant up, forces provisioning and logs all output to a file

MODULE II-1
177

Quizz

1. Which of the following are default Vagrant 2. Which of the following elements are present in a Vagrant
providers? box file ?

VirtualBox A vagrant guest configuration file that is used to create

instances of the box
VMWare
Configuration files for provisioners such as Ansible
Hyper-V
The installer for the Vagrant version which is required to
Docker
run the box
A metadata file describing the box and its requirements
A base file system image in a format supported by the
provider of the box

MODULE II-1
178

MODULE II-2 :
Cloud deployment
179

PLAN

CLOUD DEPLOYMENT MODEL:

-- Private

-- Public
Hybrid
Community

CLOUD SERVICE MODEL:

-- Cloud Foundry

- Openshift
Openstack

CLOUD INIT:

-- Syntax
Example

MODULE II-2
180

Cloud Deployment model

Cloud can be classified in terms of who owns and manages the cloud. Types of Cloud (Deployment Model)

Public Cloud,
Private Cloud,
Hybrid Cloud,
Community Cloud,

MODULE II-2
181

Private Cloud

A private Cloud or internal Cloud is used when the Cloud infrastructure, proprietary network or data
center, is operated solely for a business or organization, and serves customers within the business fire-
wall.

Most of the private Cloud are large company or government departments who prefer to keep their data in
a more controlled and secure environment.

The difference between a private Cloud and public Cloud is that in a private Cloud-based service, data
and processes are managed within the organization without the restrictions network bandwidth, security
exposures and legal

MODULE II-2
182

Private Cloud

MODULE II-2
183

Cloud Service Model

MODULE II-2
184

PaaS & IaaS Platform

Cloud Foundry : provides a highly efficient, modern model for cloud native application
delivery on top of Kubernetes.
Application and services centric lifecycle API
Container-based architecture
External dependencies are considered services

Openshift : cloud-based Kubernetes platform that helps developers build applications.

Managing applications written in different languages
Uses a hyper-visor to abstract the layer from the underlying hardware

Openstack :
Virtual servers and other resources are made available to customers
Interrelated components that control diverse, multi-vendor hardware pools of
processing, storage and networking resources throughout a data center
MODULE II-2
185

Cloud Deployment
Cloud-init :

Industry standard multi-distribution method for cross-platform Cloud instance initialization

Supported across all major public cloud providers.
Customize a new server installation during its deployment using data supplied in configuration files
Cloud init's behavior can be configured via user-data.
User-data can be given by the user at instance launch time via "--user-data" or "--user-data-file"
argument to a run instances command within CLoud platform's CLI tool.
Modular and highly configurable
Supported user data formats :
Shell scripts(starts with #!), Cloud config files (starts with #cloud-config).

MODULE II-2
186

Cloud-init Modules

Cloud-init has modules for handling : Some of the things it configures are :

Disk configuration Setting a default local

Command execution Setting hostname
Creating users and groups Generate SSH private keys
Package management Adding SSH keys to user's .ssh/authorized_keys to log
Writing content files in
Bootstrapping Chef/Puppet/Ansible Setting up ephemeral mount points
Additional modules can be written in Python if
desired

MODULE II-2
187

Cloud Config Syntax

Run 'apt-get upgrade' on first boot

#cloud-config
apt_upgrade : true

Enable byobu by default for all system users

#cloud-config
byobu_by_default : system

Import ssh keys for launchpad user 'smoser' and add his ppa
#cloud-config
ssh_import_id : [smoser]
apt_sources :
- source : "ppa:smoser/ppa"

Run a few commands on first boot

#cloud-config
runcmd:
- [ wget, http://slashdot.org, - 0, /tmp/index.html]
- [ sh, -xc, "echo $(data) ':hello world'"]

MODULE II-2
188

Cloud-init Example
Configuration of instance through "user-data" provided to cloud-init
The most popular formats for scripts user-data is the cloud-config.
Example of YAML file "cloud-init.yaml"

#cloud-config

package_update: true
packages:
- apt-transport-https
- ca-certificates
- curl
- gnupg-agent
- software-properties-common

runcmd:
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- apt-get update -y
- apt-get install -y docker-ce docker-ce-cli containerd.io
- systemctl start docker
- systemctl enable docker

final_message: "The system is finally up, after $UPTIME seconds"

File compatible with Ubuntu instance. Necessary to adapt this file if you are using another operating
system
MODULE II-2
189

Cloud-init Example

Some explanations :
- package_update : Update of the apt database on first boot.
- packages : The list of packages to install
- runcmd : Contains a list of commands to be executed
- final_message : This message will be displayed at the end of the first start (Find it in the logof Cloud-init)

Use cloud-init configuration file with multipass to validate that it works

$ multipass launch –n my-testinit –-cloud-init cloud-config.yaml

Access to Docker on new machine with the following command :

$ multipass exec my-testinit –-sudo docker ps

Check the cloud-init log :

$ multipass exe my-testinit -–sudo cat /var/log/cloud-init-output.log

MODULE II-2
190

Cloud-init Example on Azure VM

MODULE II-2
191

Quizz

1. What must be the first line of a plain text user- 2. How is cloud-init integrated with a managed system image?
data configuration containing YAML configuration
for cloud-init?
Provides the cloud-init-worker command which has to
be invoked periodically within the running instance
Cloud-config
Provides its own startup mechanism which replaces the
--cloud-config instance’s original init system such as system
#1 /user/bin/cloud-init Provides system units which must be included in
several stages of the booting process of instance
[cloud-config]
Provides a Linux Kernel Module that must be included
#cloud-config
and loaded in the instance’s initframfs
Provides the cloud-init-daemon service which is
launched during startup and keeps the instance in sync
with the desired configuration.

MODULE II-2
192

MODULE II-3 : System Image

Creation
193

PLAN

PACKER :

-- Advantages
Use cases

PACKER INSTALLATION

PACKER WORKFLOW

PACKER BUILD

PACKER PROVISION

PACKER POST PROCESS

USE CASE ON AWS

PACKER COMMAND

MODULE II-3
194

Packer
Packer is an open-source tool for creating identical machine images for multiple platforms from a single
source configuration
Packer is lightweight, runs on every major operating system
Packer does not replace configuration management like Chef/Puppet when building images, on the
contrary it uses them to install software onto the image.

MODULE II-3
195

Advantages of using Packer

Super-fast infrastructure deployment
Packer images allow you to launch completely provisioned and configured machines in seconds, rather
than several minutes or hours.
Machines can also be launched in seconds without waiting for a typically much longer provisioning time

Multi-provider portability
Packer creates identical images for multiple platforms : Run development in desktop virtualization
solutions like VMWare/VirtualBox , staging/QA in a private Cloud like Openstack and production in
AWS/Azure.

Improved stability
Packer installs and configures all the software for a machine at the time the image is built
If there are bugs in these scripts, they'll be caught early, rather than several minutes after a machine is
launched.
MODULE II-3
196

Use Cases Packer

The following are use cases of Packer :

Continuous Delivery
Packer is lightweight, portable and command-line driven. This makes it the perfect tool to put in the middle of
your Continuous delivery pipeline.

Dev/Prod Parity
Packer helps keep development, staging and production as similar as possible

Appliance/Demo Creation
Packer is perfect for creating appliances and disposable product demos. As your software changes, you can
automatically create appliances with the software pre-installed.

MODULE II-3
197

Supported Platforms

Supported platform are :

You can add support to any platform by extending Packer using plugins

MODULE II-3
198

Packer Installation

Ubuntu/Debian: HashiCorp officially maintains and signs packages

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install packer

Mac OS : with Homebrew

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/packer

Windows :
$ brew tap hashicorp/tap
$ brew install hashicorp/tap/packer

MODULE II-3
199

Packer Workflow

The main terminology of Packer are :

Templates : JSON files containing the build information
Builders : Platform specific building configuration
Provisioners : Tool that install software after the initial OS install
Post-processors : Actions to happen after the image has been built

MODULE II-3
200

Packer Build
packer.json
{
"variables": {
"aws_access_key":"{{env'AWS_ACCESS_KEY'}}",
"aws_secret_key":"{{env'AWS_SECRET_KEY'}}"
},
"builders":[{
"type":"amazon-ebs",
"access_key":"{{user'aws_access_key'}}",
"secret_key":"{{user'aws_secret_key'}}",
"region":"us-east-1",
"source_ami":"ami-fce3c696",
"instance_type":"t2.micro",
"ssh_username":"admin",
"ami_name":"yourApp {{timestamp}}"
}]
}

$ packer validate packer.json : Validate command

Packer can create multiple images for multiple platforms in parallel, all configured from a single template [8].

MODULE II-3
201

Packer Build Command with vars

Via Command line using –var flag

> packer build

–var 'aws_access_key=id'
–var 'aws_secret_key=Secret'
Packer.json

Via File with -var-file flag

variables.json
{
"aws_access_key":"accessKey",
"aws_secret_key":"secretKey"
}

$ packer build -var-file=variables.json packer.json

Var-file flag can be specified multiple times and variables from multiple files will be read and applied. Combining the –var and –var-file flags together also works how you'd expect. Flags
set later in the command override flags set earlier.

MODULE II-3
202

Packer Provision
packer.json
{
"variables": ["..."],
"builders": ["..."],
"provisioners": [{
"type":"shell",
"inline":[
"sleep 30",--waiting for SSH to be available
"sudo apt-get update",
"sudo apt-get install –y redis-server"
]
},
{
"type":"shell",
"script":"./scripts/install-java.sh",
}]

Others – Remote shell, File uploads, Ansible (local&remote), Chef, Puppet, Salt, PowerShell etc.

MODULE II-3
203

Packer Post Process

packer.json
{
"variables": ["..."],
"builders": ["..."],
"provisioners": ["..."],
"post-processors": [
{
"type":"compress",
"format": "tar.gz"
}
]
}

Others – Amazon Import, CheckSum, Docker Push/Tag/Save, Google Compute Export, Vagrant, vSphere.

MODULE II-3
204

Packer/Use case on AWS

MODULE II-3
205

Packer command

CLI

packer init Install required plugins

Packer plugins required List plugins that will be installed by “packer init”

Packer plugins installed List installed plugins

Packer build Build image from template.Takes a template and runs all the builds within it in order to generate a set of
artifacts. Use –force to forces a builder to run

Packer fmt Format HCL2 configuration files to a canonical format and style

Packer validate Check that a template is valid

Packer inspect See components of a template

Packer hcl2_upgrade Convert JSON to HCL2

MODULE II-3
206

Quizz

1. What does the command packer validate template.json 2. Which of the following sections must exist in a Packer
do? template ?

Verifies that the latest build of the template can be Images

run without downloading additional images or
Provisioners
artifacts
Builders
Verified that the file template.json is a syntactically
correct and complete Packer template Variables
Verified that all existing artifacts generated by Post-processors
template.json have their original checksums
Verified that all source images referenced in
template.json are available and have valid
cryptographic signatures
Verifies that all images generated previously by
template.json still use the most recent source
images

MODULE II-3
207

PART III.
Container Management
…
208

MODULE III-1 : Container Usage

209

PLAN

VIRTUALIZATION
CONTAINERIZATION OF APP :

--
DOCKER

-- Functionality
--
Dockerfile
Instructions

-- Benefits
Architecture
Engine
Example
Command

MULTI-STAGE BUILD
CONTAINER RUNTIME

DOCKER IMAGES

-- Image

-- Registries
Naming and tagging

- Layers
Commands

MODULE III-1
210

Virtualization

In Computing, Virtualization refers to the act of making a virtual version of one thing, together with virtual
hardware platforms, storage devices, and electronic network resources.
Containerization is a form of virtualization where applications run in isolated user spaces, called containers,
while using the same shared operating system (OS).

MODULE III-1
211

Advantages of Containerization over Virtualization

MODULE III-1
212

Docker
Docker is a software that runs on Linux and Windows environments
It creates, manage and orchestrates Containers.
The Docker project is open-source and the upstream lives in the moby/moby repo on GitHub
Docker, Inc. Is the overall maintainer of the open-source project and offers commercial versions of
Docker with support contracts.
There are two main editions of Docker : Enterprise Edition (EE) and Community Edition (CE)
Docker version numbers follow the YY.MM-xx versioning scheme "19.03.12(25 juin 2020)"
Tool that is designed to benefit both developers and IT operators, making it a part of many DevOps
toolchains.

MODULE III-1
213

Docker Technologies
Docker technologies include at least three things to be aware :

The runtime
The daemon or Engine
The orchestrator

MODULE III-1
214

Docker Functionalities & Properties

Docker Functionalities

Docker Properties

MODULE III-1
215

Benefits of Docker

MODULE III-1
216

Docker Architecture

MODULE III-1
217

Docker Engine/Docker Daemon

Docker Engine is the infrastructure plumbing software that runs and orchestrates containers (VMWare
admin -> like ESXi).
Docker Engine is modular in design with many swappable components. Where possible, these are based
on open-standards outlined by the Open Container Initiative (OCI).
Docker Engine is made from many specialized tools APIs, execution driver, runtime, shims etc.
All other Docker, Inc. And 3rd party products plug into the Docker Engine and build around it.

MODULE III-1
218

Containers

Container is the runtime instance of an image. In the same way that we can start a VM from Virtual Machine
template.
Run until the App they are executing exits and share the OS/kernel with the host they're running on.

MODULE III-1
219

Containers Commands

Docker container run –it ubuntu /bin/bash Start an Ubuntu container in the foreground, and tell it to run the Bash shell

[Ctrl + PQ] Detach your shell from the terminal of a container and leave the container running (UP) in the background

Lists all containers in the running (UP) state. -a flag you will also see containers in the stopped (Exited)
Docker container ls
state

Docker container exec –it Let's you run a new process inside of a running container. This command will start a new Bash Shell inside
<cointainer-name or container-id> bash of a running container and connect to it.

Docker container stop Stop a running container and put it in the Exited(0) state
<cointainer-name or container-id>

Docker container start Restart a stopped (Exited) container

<cointainer-name or container-id>

Docker container rm Delete a stopped container

<cointainer-name or container-id>

Docker container inspect Show detailed configuration and runtime information about a container
<cointainer-name or container-id>

MODULE III-1
220

Docker Images

Docker images is like VM templates (Admin)

Docker images is like classes (Developer)
Docker Images are considered build-time constructs, whereas containers are run-time constructs

You start by pulling images from an image registry ( Docker Hub)

The pull operation downloads the image to your local Docker host where you can use it to start one or
more Docker containers.
Images are made up of multiple layers that get stacked on top of each other and represented as a single
object.
Inside of the image is a cut-down operating (OS) and all of the files and dependencies required to run an
application
Containers are intended to be fast and lightweight; images tend to be small.

MODULE III-1
221

Docker Image Registries

Docker images are stored in image registries. The most common registry is Docker Hub
(https://hub.docker.com)
The Docker client is opinionated and defaults to using Docker Hub.
Image registries contain multiple image repositories

MODULE III-1
222

Docker Image Registries

Docker Hub also has the concept of official repositories and unofficial repositories

Official repositories : contain images that have been vetted by Docker. Inc.
Examples : nginx (https://hub.docker.com/_/nginx/), mongodb (https://hub.docker.com/_/mongo/)
Unofficial repositories : you should not expect them to be safe, well-documented or built according to
best practices.

MODULE III-1
223

Docker Image naming and tagging

Images from official repositories are as simple as giving the repository name and tag separated by a
colon(:). Docker image pull <repository>:tag
If you do not specify the image tag, Docker will assume you are referring to the image tagged as latest
Image is tagged as a latest does not guarantee it is the most recent image in a repository.
A single image can have as many tags as you want

MODULE III-1
224

Image and layers (1/2)

A docker image is just a bunch of loosely-connected read-only layers

Docker takes care of stacking these layers and representing them as a single unified object

To see the layers of an image, you can inspect the image with the docker image inspect command

$ docker image inspect gilbertfongan/newflaskapp

[
{
"Id": "sha256:12d37cae3628e78ba13b96c0fde2a1fedd59f6afb1ff10bb4880f971199927c2",
"RepoTags": [
"gilbertfongan/newflaskapp:latest"
],
<snip>
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:4942a1abcbfa1c325b1d7ed93d3cf6020f555be706672308a4a4a6b6d631d2e7",
"sha256:5395b1dd90da45239b11b2b05354e0e1429aa312cb4a998e517810e831cbeedf",
"sha256:b0fee53fe4bff191ae47e70eeec4615a54d7b3a0f0e655604993bba83f8f72fb",
"sha256:1bbae6f1e7fc51dba8a83df090afa3723cdee50c95490b0c09cb7f0060d0d215"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]

MODULE III-1
225

Image and layers (2/2)

All Docker images start with a base layer, and as changes are made and new content is added, new
layers are added on top.
Multiple images can, and do, share layers. This leads to efficiencies in space and performance.
These lines tell us that Docker is smart enough to recognize when it’s being asked to pull an image layer that
it already has a copy of/

MODULE III-1
226

Multi-architecture images

A single image (repository:tag) can have an image for Linux on x64, Linux on PowerPC, Windows x64,
ARM etc.
To make this happen, the Registry API supports two important construct :
- Manifest lists : a list of architectures supported by a particular image tag. Each supported architecture then
has its own “manifest detailing the layers it’s composed from.
- Manifests : containing image config and layer data

MODULE III-1
227

Containers image Commands

Docker image pull The command to download images. By default, images will be pulled from repositories on Docker Hub
<image_name>: <image_tag>

Docker image ls Lists all of the images stored in your Docker host’s local cache

Docker image inspect Gives all the details of an image layer data and metadata
<image_name>: <image_tag>

Docker image rm Delete an image

<image_name>: <image_tag>

Docker rmi Delete an image. It’s impossible to delete an image associated with a container in the runnin(Up) or
<image_name>: <image_tag> stopped (Excited) states

MODULE III-1
228

Containerizing an App

The process of taking an application and configuring it to run as a container is called “Containerizing”,
Sometimes we call it “Dockerizing”.
Containers are all about apps. They're about making apps simple to build, ship, and run.
The process of containerizing an app looks like this :

1. Start with your application code.

2. Create a Dockerfile that describes your app,its
dependencies, and how to run it
3. Feed this Dockerfile into the Docker image
build command.
4. Sit back while Docker builds your application
into a Docker image and push it to the registry.
5. Run and execute the container

MODULE III-1
229

Dockerfile

Dockerfile is the blueprint that describes the application and tells Docker how to build it into an image
The directory containing the application is referred to as the build context
It’s a common practice to keep your Dockerfile in the root directory of the build context
Dockerfile starts with a capital “D” and is all one word “Dockerfile”
It can help bridge the gap between development and operations
Should be treated as code, and checked into a source control system
If an instruction is adding new content such as files and programs to the image, it will create a new
layer. If it is adding instructions on how to build the image and run the application, it will create
metadata.

MODULE III-1
230

Dockerfile Instructions
INSTRUCTION DESCRIPTION

FROM First instruction in Dockerfile and it identifies the image to inherit from

MAINTAINER Provides visibility and credit to the author of the image

RUN Executes a Linux command for configuring and installing

ENTRYPOINT The final script or application used to bootstrap the container, making it an executable application

CMD Provide default arguments to the ENTRYPOINT using a JSON array format

LABEL Name/value metadata about the image

ENV Sets environment variables

COPY Copies file into the container

ADD Alternative to copy

WORKDIR Sets working directory for RUN, CMD, ENTRYPOINT, COPY, and/or ADD instructions

EXPOSE Ports the container will listen on

VOLUME Creates a mount point

USER User to run RUN, CMD, and/or ENTRYPOINT instructions

MODULE III-1
231

Dockerfile Example

Dockerfile is a text document that contains all the commands a user could call on the command line to
assemble an image

$ cat Dockerfile

FROM ubuntu
RUN apt-get update && apt-get install -y python3 pip
&& pip install flask
COPY app.py /opt/
ENTRYPOINT FLASK_APP=/opt/app.py flask run --
host=0.0.0.0 --port=8080

MODULE III-1
232

Docker Build image

The Docker build command builds Docker images from a Dockerfile and a “context”

# docker build . –t ACCOUNT_ID/IMAGE_NAME

# cat > Dockerfile

FROM ubuntu
RUN apt-get update && apt-get install -y python3 pip
RUN pip install flask
COPY app.py /opt/
ENTRYPOINT FLASK_APP=/opt/app.py flask run --host=0.0.0.0 --
port=8080

MODULE III-1
233

Docker Push image

Once we’ve created an image, it’s time to store it in an image registry to keep it safe and make it available to
others

To push an image to Docker Hub, we need to login with the Docker ID

Before we can push an image, we need to tag it in a special way (if we don’t specify values of registry or
tag, Docker will assume Registry=docker,io and Tag=latest
# docker image tag newflaskapp:latest gilbertfongan/newflaskapp:latest

Now we can push it to Docker Hub

MODULE III-1
234

Docker Push image

MODULE III-1
235

Dockerfile/Multi-stage Builds
Docker images with complexity and big Instructions are bad => More potential vulnerabilities and
possibly a bigger attack surface
Multi-stage builds are all about optimizing builds without adding complexity.
With multi-stage builds, we have a single Dockerfile containing multiple FROM instructions. Each FROM
instruction is a new build stage that can easily COPY artefacts from previous stages.

MODULE III-1
236

Dockerfile/Multi-stage Builds EXAMPLE

FROM node:latest AS storefront

WORKDIR /usr/src/atsea/app/react-app

COPY react-app .

RUN npm install

RUN npm run build

FROM maven:latest AS appserver

WORKDIR /usr/src/atsea

COPY pom.xml .

RUN mvn -B -f pom.xml -s /usr/share/maven/ref/settings-docker.xml dependency:resolve

COPY . .

RUN mvn -B -s /usr/share/maven/ref/settings-docker.xml package -DskipTests

FROM java:8-jdk-alpine

RUN adduser -Dh /home/gordon gordon

WORKDIR /static

COPY --from=storefront /usr/src/atsea/app/react-app/build/ .

WORKDIR /app

COPY --from=appserver /usr/src/atsea/target/AtSea-0.0.1-SNAPSHOT.jar .

ENTRYPOINT ["java", "-jar", "/app/AtSea-0.0.1-SNAPSHOT.jar"]

CMD ["--spring.profiles.active=postgres"]

MODULE III-1
237

Containerizing an App Commands

Commands DESCRIPTION

Docker image build –t <repository_name>:<tagname> Command that reads a Dockerfile and containerizes the application.
<build_context> -t flag tags the image
-f flag lets you specify the name and location of the Dockerfile

Docker image push <repository_name>:<tagname> Push containerized app to image registry (by default Docker Hub)

Docker login [OPTIONS][SERVER] Log in into a Docker registry

Docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG] Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE

MODULE III-1
238

Docker workflow

MODULE III-1
239

Docker Usage Sheet

Docker create <image> Create a container without starting it Docker build <URL> Create an image from a Dockerfile

Docker rename <CONTAINER_NAME> Rename a container Docker build -t <URL> Build an image from a Dockerfile and tags it
<NEW_CONTAINER_NAME>
Docker pull <IMAGE> Pull an image from a registry
Container Lifecycle

Image Lifecycle
Docker run <IMAGE> Create and start a container
Docker push <IMAGE> Push an image to a registry

Docker run - -rm <IMAGE> Remove a container after it stops

Docker import <URL/FILE> Create an image from a tarball

Docker run -td <IMAGE> Start a container and keep it running

Docker commit <CONTAINER> Create an image from a container
<NEW_IMAGE_NAME>
Docker run -it <IMAGE> Create, start the container, and run a command
in it
Docker rmi <IMAGE> Remove an image
Docker run –it-rm <IMAGE> Create, start the container, and run a command
in it; after executing, the container is removed
Docker save <IMAGE> > <TAR_FILE> Save an image to a tar archieve stream to stdout
Docker rm <CONTAINER> Delete a container if it’s running with all parent layers, tags and versions

Docker update <CONTAINER> Update the configuration of a container Docker load <TAR_FILE/STDIN_FILE> Load an image from a tar archieve as stdin

MODULE III-1
240

Docker Usage Sheet

Docker start <CONTAINER> Start a container

Docker ps List running containers
Docker stop <CONTAINER> Stop a running container
Docker ps -a List running and stopped containers
Docker restart <CONTAINER> Start a running container and start it upagain
Start & Stop

Docker logs <CONTAINER> List the logs from a running container

Docker pause <CONTAINER> Pause processes in a running container
Docker inspect <OBJECT_NAME/ID> List low-level information on an object
Docker unpause <CONTAINER> Unpause processes in a container

Information
Docker wait <CONTAINER> Block a container until other containers stop Docker events <CONTAINER> List real time events from a container

Docker kill <CONTAINER> Kill a container by sending SIGKILL to a

running container Docker port <CONTAINER> Show port (or specific) mapping from a container

Docker attach <CONTAINER> Attach local standard input, output, and error
streams to a running container Docker top <CONTAINER> Show running processes in a container

Docker stats <CONTAINER> Show live resource usage statistics of containers

Docker network ls List of networks

Docker diff <CONTAINER> Show changes to files (or directories) on a
filesystem
Docker network rm <NETWORK> Remove one or more networks
Networking

Docker image ls Show all locally stored images

Docker network inspect <NETWORK> Show information on one or more networks

Docker history <IMAGE> Show history of an image

Docker network connect <NETWORK> Connect a container to a network
<CONTAINER>

Docker network disconnect <NETWORK> Disonnect a container from a network

<CONTAINER>

MODULE III-1
241

Quizz

1. Which of the following instructions in a Dockerfile can 2. Which of the following mechanisms are used for service
download a file via HTTP into the container image? discovery in a container environment ?

CURL The container platform offers a command like

docker service discover which should be run within
COPY
a container
WGET
The container platform sets environment variables
ADD containing service information within the containers
PLACE The container platform maintains DNS records
which point to container offering a specific service
The container platform lists localhost ports
assigned to containers in each container’s
/etc/services file
The container platform mounts the sockets for all
available services into the container’s file systems.

MODULE III-1
242

MODULE III-2 : Container

Deployment and Orchestration
243

PLAN

DOCKER COMPOSE
DOCKER SWARM:

-- Compose file
--
- Deploy
Commands
--
Swarm Cluster
Swarm services
Swarm network mode
Commands
DOCKER NETWORKING

-- Drivers DOCKER STACKS

- Service Discovery
KUBERNETES

--
Commands
Architecture

-- Pods
Services

- Deployment
Commands

MODULE III-2
244

Docker Compose

In the beginning was Fig, created by Orchard

In 2014, Docker Inc. acquired Orchard and re-branded Fig as Docker Compose
Compose still an external Python binary that you have to install on a host running the Docker Engine.
Docker Compose is a tool for defining and running multi-container Docker applications.
Compose use a YAML file to configure application’s services
Then , with a single command , you create and start all the services from your configuration

MODULE III-2
245

Docker Compose

Docker Compose define multi-container (multi-service) apps in a YAML file, pass the YAML file to the docker-
compose binary, and Compose deploys it through the Docker Engine API.
Docker Compose lets you describe an entire app in a single declarative configuration file
Deploy an entire app with a single command (docker-compose up)
Once the app is deployed, you can manage its entire life cycle with a simple set of commands

MODULE III-2
246

Compose File
The example shown on the right shows a simple Compose file that defines a
small Flask app with two services, a networks and a volumes [Simple web
server that counts the number of visits and stores the value in Redis].

Compose uses YAML files to define multi-service applications

YAML is a superset of JSON so any JSON file should be valid YAML
The default name for the Compose YAML file is “docker-compose.yml”
A custom filenames can be specified with the use of –f flag

The Docker Compose file has 4 top-level keys :

- Version
- Services
- Networks
- Volumes
MODULE III-2
247

Compose File
The structure of the Docker Compose File is as follows

Version : The version of the Compose file format (API) [Which can be depending on the Docker Engine
release]. This does not define the version of Docker Compose or the Docker Engine
Services : Define the different application services (Compose will deploy each of these services as its
own container).
- Build : Build a new image using the instructions in the Dockerfile in the current directory.
- Command : Run a command (For Example to run Python app as the main App in the container.)
- Ports : Map port 5000 inside the container (target) to port 5000 on the host (published)
- Networks : Which network to attach the service’s container to. Network should be already defined in the
netwotks top-level key.
- Volumes : Which volume to attach the service’s container to. The Volume should be already defined in the
volume top-level key.

MODULE III-2
248

Deploy with Compose (1/4)

Example with Flask App : https://gitlab.com/GilbertFongan/devops-book-labs/-/tree/main/Module4-
Docker/Docker-Ubuntu-VM/counter-app

Dockerfile : Describes how to build the image for the web-fe service
App.py : Is the python Flask application code
Requirements.txt : List the Python Packages required for the App.
Docker-compose.yml : Describes how Docker should deploy the App.

MODULE III-2
249

Deploy with Compose (2/4)

The following command can be used to execute a Docker Compose command to bring up a Compose App
(Multi-container App defined in a Compose file) : $ docker-compose up

Builds all required images

Creates all required networks
Starts all required containers.
The Docker-compose up command uses the Docker-compose.yml or Dockercompose.yml files by default.
In case the name of the compose file is different, it is necessary to specify the –f flag to indicate the custom
file.
$ docker-compose –f my-custom-docker-compose-file.yml up
The –d flag can also be specified to bring the App up in the background : $ docker-compose up -d

MODULE III-2
250

Deploy with Compose (3/4)

After running the Docker-Compose command on our repository, we can discover three images built or
pulled as part of the deployment

The list of running containers is as follows (We can notice that the names of each containers is prefixed
with the name of the project or name of the working directory.),

With the scalability feature of the compose service, each container has a numeric suffix that indicates the
instance number.
MODULE III-2
251

Deploy with Compose (4/4)

In addition to the services, Docker-compose also created the networks and volumes

Thus, the application successfully deployed, it can be accessed

To stop the Docker-Compose App without deleting the images and volumes created

MODULE III-2
252

Docker Compose Commands

COMMANDS DESCRIPTION

Docker-compose up Deploy a Compose App. It expects the Compose file to be called docker-compose.yml or docker-
compose.yaml, but you can specify a custom filename with the –f flag. It’s common to start the App in the
background with the –d flag

Docker-compose stop Stop all containers in a Compose App without deleting them from the system

Docker-compose restart Restart a Compose App that has been stopped with docker-compose stop. If you have made changes to
your Compose App since stopping it, these changes will not appear in the restarted App. You will need to
re-deploy the App to get the changes.

Docker-compose ps List each container in the Compose App. It show the current state, the command each one is running, and
network ports

Docker-compose down Stop and delete a running Compose App. It deletes containers and networks, but not volumes and images.

MODULE III-2
253

Docker Networking

At the highest level, Docker networking comprises three major components :

The Container Network Model (CNM)

Libnetwork
Driver

MODULE III-2
254

Docker Networking

Docker runs applications inside of containers, and these need to communicate over lots of different
networks.
Docker has solutions for container-to-container networks, as well as connecting to existing networks
and VLANs.
Docker networking is based on an open-source pluggable architecture called the Container Network
Model (CNM)
Libnetwork is Docker’s real-world implementation of CNM, and it provides all of Docker’s core
networking capabilities.
Drivers plug in to libnetwork to provide specific network topologies such as VXLAN overlay networks.

MODULE III-2
255

Docker Networking Drivers

MODULE III-2
256

Docker Networking Drivers

Docker ships with a set of native drivers that deal with the most common networking requirements :

Bridge (default) : Containers in local Docker0 bridge

Null : Containers has no network interface
Host : Containers use host’s network interface
Overlay : Multi-host
MACVLAN : for existing VLANs

MODULE III-2
257

Null Network (None Driver)

The None driver option for container networking disables the networking of a container while allowing
the very same container to use a custom third-party network driver, if needed, to implement its
networking requirements.

None provides the functionality of disabling networking

Form a container with none Network :

$ docker run --rm -dit \

--network none \
--name no-net-alpine \
alpine:latest \
ash

Using ‘—network none’ will result in a container with no eth0.

MODULE III-2
258

Single-host Bridge Network (1/4)

It is the simplest type of docker network.

Single-host means it only exists on a single Docker host and can only connect containers that are on the
same host
Bridge means that it’s an implementation of an 802.1d bridge (layer 2 switch)

Docker on Linux creates single-host bridge networks with the built-in bridge driver, whereas Docker on
Windows creates them using the built-in nat driver.

MODULE III-2
259

Single-host Bridge Network (2/4)

By default, every Docker host gets a single-host bridge network. On Linux it’s called “bridge”, and on
Windows it’s called “Nat”. All new containers will attach to that network unless you override it on the
command line with the –network flag

Command to create a new single-host bridge network called “localnet”

MODULE III-2
260

Single-host Bridge Network (3/4)

Containers on Bridge networks can only communicate with other containers on the same network.
However, you can get around this using Port Mappings.
Port Mappings map a container port to a specific port on the Docker Host. Any traffic hitting the Docker
host on the configured port will be directed to the container.

For this example, illustrated by this diagram, the application running in the
container is operating on Port 80
All incoming traffic on the Host 10.0.0.15 with port 5000 is mapped to port
80 of the running container.

MODULE III-2
261

Single-host Bridge Network (4/4)

MODULE III-2
262

Single-host User-Defined Bridge Network

MODULE III-2
263

Single-host Bridge Network Comparison

Features Default User-Defined

Better isolation and interoperability between
No Yes
containerized applications
Automatic DNS resolution between containers No Yes
Attachment and detachment of containers on the fly No Yes
Configurable bridge creation No Yes
Linked containers share environment variables Yes No

MODULE III-2
264

Single-host Bridge Network Use Case

MODULE III-2
265

Host Network
The Host Network driver option, as opposed to the bridge, eliminates the network isolation between the
container and the host system by allowing the container to directly access the host network.

Features :
An overlay network is used to manage swarm and service-related traffic
Docker daemon Host network and ports are used to send data for individual swarm service

Advantages :
Optimizes the performance (eliminating the need for NAT since the container ports are automatically
published and available as host ports).
Handles a large range of ports
Does not require “userland-proxy” for each port.

NOTE : The host networking driver only works on Linux hosts

MODULE III-2
266

Host Network

MODULE III-2
267

Multi-Host Overlay Network (1/3)

The Multi-Host Overlay Network driver option is a network type that spans multiple hosts, typically part of a
cluster, allowing the containers traffic to be routed between hosts as containers or services from one host
attempt to talk to others running on another host in the cluster..

Allow to create a flat, secure, layer-2 Network, spanning multiple hosts.

Docker provides a native driver for overlay networks. This makes creating them as simple as adding the -
--d overlay flag to the docker network create command.

MODULE III-2
268

Multi-Host Overlay Network (2/3)

Provisioning for an Overlay Network is automated by Docker Swarm Control Plane

MODULE III-2
269

Multi-Host Overlay Network (3/3)

To create a new overlay network : $ docker network create –d overlay devops

To list all networks on each node : $ docker network ls

To attach a service to the Overlay Network : $ docker service create –name=test-devops –network devops –p 80:80

gilbertfongan/demo:v1

To inspect a created Overlay Network : $ docker network inspect devops

MODULE III-2
270

Multi-Host Overlay Network Use case

MODULE III-2
271

MAC VLAN (1/4)

The MACVLAN Network driver allows a user to change the appearance of a container on the physical
network.
A container may appear as a physical device with its own MAC address on the Network.
The container is directly connected to the physical network instead of having its traffic routed through
the Host Network.
MACVLAN Network is used to assign MAC address to the virtual Network interface of containers

MODULE III-2
272

MAC VLAN (2/4)

Advantages :

Simple and lightweight architecture

Direct access between physical Network and Containers
Containers receive routable IP addresses that are present on the subnet of the physical Network

Precautionary measures :

Cut down the large number of unique MAC to save the Network from damage
Handle “promiscuous mode” which isn’t allowed on most public Cloud Platforms

MODULE III-2
273

MAC VLAN (3/4)

MODULE III-2
274

MAC VLAN (4/4)

The MACVLAN driver needs this arguments about the Network : Subnet info, Gateway, Range of IP’s it
can assign to containers, Interface or Sub-interface on the Host to use.

Create a new MACVLAN Network called “macvlan5” that will connect containers to VLAN5:
$ docker network create –d macvlan \
--subnet= 172.16.0.0/24 \
--ip-range=172.16.0.0/25 \
--gateway=10.0.0.1 \
-o parent=eth0.5 \
macvlan5

The MACVLAN5 Network is ready for containers. Create a container to deploy with the Network:

$ docker container run –d --name=test-macvlan –-network macvlan5 alpine sleep 1d

MODULE III-2
275

MAC VLAN Use Cases

Low-latency applications
Network design which needs containers to be on the same subnet and use Ips as the external Host
Network

MODULE III-2
276

Docker Network Service Discovery

Service discovery allows all containers and Swarm services to locate each other by name. The only
requirement is that they have to be on the same Network.
This leverages Docker’s embedded DNS Server and the DNS resolver in each container

1. Ping c2 command invokes the local DNS resolver to resolve the name « c2 ». All Docker containers have a local DNS

2.
resolver
In case that the local resolver doesn’t have an IP adress for « c2 » in its local cache, it initiates a recursive query to the

3.
Docker DNS Server. The local resolver is pre-configured to know how to reach the Docker DNS server.
Docker DNS server holds name-to-IP mappings for all containers created with the --name or --net-alias flags. It know

4.
the IP adress of container « c2 »

5.
[Same Network] DNS resolver returns the IP adress of « c2 » to the local resolver in « c1 »
The ping command is sent to the corresponding target IP adress of « c2 »

MODULE III-2
277

Docker Network Commands

COMMANDS DESCRIPTION

Docker network ls List all networks on the local Docker host.

Docker network create Creates new Docker networks.

By default, it creates them with the NAT driver on Windows and the Bridge driver on Linux.Driver type can
be specify with the –d flag

Docker network inspect Provides detailed configuration information about a Docker Network

Docker network prune Deletes all unused networks on a Docker host.

Docker network rm Deletes specific networks on a Docker host

MODULE III-2
278

Docker Swarm (1/5)

A Swarm is a term used to describe many Docker hosts or systems. So, Docker Swarm is a tool used for
managing container configuration. The main features of Docker Swarm are :

Security : Nodes Allow enforcement of encryption and mutual authentication to enhance Hight security
in communications between nodes.
Scalability : Automatic addition or removal of tasks that allow users to scale up or down as per their
needs
Decentralized design : Allow to create a Swarm from one disk image
Integration : The cluster management has been integrated with Docker Engine. This allows users to
manage swarms without requiring another orchestrations software.
Rolling updates : Services updates on nodes can be made incrementally during rollout. In case of a
problem, you can roll back to a previous safe service.
Declarative Service : Allow to define the required state of a service
Service discovery : Embedded DNS server can be used to query a container that runs within the Swarm.
MODULE III-2
279

Docker Swarm (2/5)

Docker Swarm is two (02) main things
Enterprise-grade secure cluster of Docker hosts (Clustering)
Engine for orchestrating microservices apps (Orchestration)
Clustering :
Groups one or more Docker nodes and lets user manage them as a cluster
Encrypted distributed cluster store and Network
Mutual TLS and secure cluster join tokens
PKI makes managing and rotating certificates as an easy task
Add and remove nodes
Orchestration :
Rich API exposed allow user to deploy and manage complex microservices Apps with ease
Define Apps in declarative manifest files and deploy them with native Docker commands
Rolling updates rollbacks and scaling operations.

MODULE III-2
280

Docker Swarm (3/5)

MODULE III-2
281

Docker Swarm (4/5)

Nodes are configured as managers or workers :

- Managers : Control Plan of the Cluster – Manage the state of the cluster and dispatch tasks to workers
- Workers : Accept tasks from Managers and execute them
Configuration and state of the Swarm is held in a distributed etcd database located on all managers

MODULE III-2
282

Docker Swarm (5/5)

Swarm uses TLS to encrypt communications, authenticate nodes, and authorize roles:
The atomic unit of scheduling on a Swarm is the service

- Service is a higher-level construct that wraps some advanced features around containers.
- A task or replica is a container wrapped in a service
High-level view of Swarm cluster :

MODULE III-2
283

Swarm Mode
Replicated services (default) : This deploys a desired number of replicas and distributes them as evenly
as possible across the cluster.
Global services : This runs a single replica on every node in the Swarm.

MODULE III-2
284

Build Swarm Cluster

To build a Swarm cluster with manager nodes and worker nodes :
Initialize the firs manager node
Join additional manager nodes (optional)
Join worker nodes

The following tasks can be performed :

Choose a node to initialize a new Swarm (Manager)
$ docker swarm init \
--advertise-addr 172.17.8.104:2377 \
--listen-addr 172.17.8.104:2377

MODULE III-2
285

Build Swarm Cluster

"docker swarm init": Initialize a new swarm and make this node the first manager.
"--advertise-addr" : Swarm API endpoint that will be used to connect to the manager by other nodes in the
Swarm. It's an optional flag and gives control over which IP gets used on nodes with multiple IPs.
"--listen-addr" : The node will accept Swarm traffic on this IP Address. Sometimes used to restrict Swarm
to a particular IP on a system with multiple IPs.

The default port that swarm mode operates on is 2377 for secured(HTTPS) client-to-swarm connections

Join a worker to the Swarm (Copy the output of the previous command)
$ docker swarm join-token worker (Run from Manager 1 node to extract the following commands and tokens
required to add a new workers)
$ docker swarm join --token XXXXXXXXXXXXXXX 172.17.8.104:2377 (Run from Worker)

MODULE III-2
286

Build Swarm Cluster

Join a Manager Node to the Swarm
$ docker swarm join-token manager //(Run from Manager 1 node to extract the following commands and tokens
required to add a new workers)
//Run from Manager 2
$ docker swarm join \
--token XXXXXX...XXXXXXXXX\
172.17.8.4:2377 \
--advertise-addr 172.17.8.40:2377 \
--listen-addr 172.17.8.4:2377

172.17.8.40 IP Address of the new Manager Node.

List the nodes in the Swarm
$ docker node ls

MODULE III-2
287

Swarm Services

To deploy new service :

Swarm manager accepts your service definition as
the desired state for the service.
it schedules the service on nodes in the swarm as
one or more replica tasks.
The task run independently of each other on nodes
in the Swarm

MODULE III-2
288

Swarm service create/update

Create a service
$ docker service create --name hello-world \
--replicas 1 \

alpine ping docker.com

$ docker service create --name redis \

-p 6369:6369
--replicas 2 \
redis

MODULE III-2
289

Swarm service create/update

Viewing and inspecting services
$ docker service ls

$ docker service ps redis

$ docker service inspect redis

MODULE III-2
290

Swarm service create/update

Scaling services (help with overloaded traffic)
$ docker service scale redis=4

$ docker service ls

$ docker service ps redis

MODULE III-2
291

Docker Network mode (ingress load balancing)

Swarm supports two(02)publishing modes that make services accessible from outside of the Cluster

Ingress mode (default) : Services published (with --publish) can be accessed from any node in the
Swarm

# docker service create –d –name example \

--publish published=5000, target=80 \
nginx

Host mode : Services published (with --publish and add mode=host) can only be accessed via nodes
running service replicas

# docker service create –d –name example2 \

--publish published=5001, target=8080, mode=host \
nginx

MODULE III-2
292

Swarm Rolling Updates

Docker services update is very easy and simplify

Create a new Overlay network

$ docker network create -d overlay devops

The overlay network creates a new layer 2 container network on top of potentially multiple different underlying networks

MODULE III-2
293

Swarm Rolling Updates

Verify network
$ docker network ls

Deploy a new service and attach it to the network

$ docker service create --name=test-devops --network devops -p 80:80 --replicas 12 gilbertfongan/demo:v1

MODULE III-2
294

Swarm Rolling Updates

Passing the service, the –p 80:80 flag ensure that a Swarm-wide mapping is created that maps all traffic,
coming into any node in the Swarm on port 80, through to port 80 inside of any service replica.
Ingress mode : Define the mode of publishing a port on every node in the Swarm
Deploy the update
$ docker service ls

$ docker service update --image gilbertfongan/demo:v2 --update-parallelism 2 --update-delay 20s test-

devops

MODULE III-2
295

Docker Swarm Commands

COMMANDS DESCRIPTION

Docker swarm init Initialize/Create a new swarm

Docker swarm join-token Reveals the command and tokens needed to join workers and managers to existing Swarms.
Docker swarm join-token manager command is used to expose the command to join a new manager
Docker swarm join-token worker command is used to expose the command to join a new worker

Docker node ls List all nodes in the Swarm including which are managers and leader

Docker service create Command to create a new service

Docker service ls List running services in the Swarm and gives basic info on the state of the service and any replicas it's
running

Docker service ps <service> Give more detailed information about individual service replicas

Docker service inspect Give very detailed information on a service

Docker service scale Scale the number of replicas in a service up and down

Docker service update Update many of the properties of a running service

Docker service logs View the logs of a service

Docker service rm Delete a service from the Swarm

MODULE III-2
296

Docker Stacks
Help to define complex multi-service apps in a single declarative file.

While Docker is a great tool for development and testing, Docker stacks are great tools for scale and
production.
Provide a simple way to deploy the App and Manage its entire lifecycle :
- Health checks
- Scaling
- Updates and Rollbacks
The stack file includes the entire stack of services that make up the Appin form of Compose file :
- Services
- Volumes
- Networks
- Secrets

MODULE III-2
297

Docker Stacks
Stacks are often compared to Compose with the only difference being that it deploys on a cluster swarm
They are at the top of the Docker application hierarchy.
They build on top of services, which turn build on top of containers

MODULE III-2
298

Stacks deploy example

Create a NGINX and MySQL container

Define services "web-app.yml"

Version : '3'
Services :
Web:
image:nginx
Ports:
- "8081:80"
mysql:
Image: mysql
Environment:
MYSQL_ALLOW_EMPTY_PASSWORD : "yes"

MODULE III-2
299

Stacks deploy example

Deploy the stack
$ docker stack deploy –c web-app.yml webapp
Creating network webapp_default
Creating service webapp_web
Creating service webapp_mysql

A default network is created

List Docker stacks
$ docker stack ls
NAME SERVICES ORCHESTRATOR
webapp 2 Swarm
$ docker stack services webapp
ID NAME MODE REPLICAS IMAGE PORTS
j0uprwm4oua9 webapp_mysql replicated 1/1 mysql:latest
qg4nqt7sdmo3 webapp_web replicated 1/1 nginx:latest *:808
1->80/tcp

MODULE III-2
300

Stacks deploy example

Task making up the services
$ docker stack ps webapp
ID NAME IMAGE NODE DESIRED
STATE CURRENT STATE ERROR PORTS
ow1buo3i4mfj webapp_mysql.1 mysql:latest node1 Running Running 12
minutes ago
jnv24yvl3ekz webapp_web.1 nginx:latest node2 Running Running 12
minutes ago

Delete Docker stacks

$ docker stack rm webapp
Removing service webapp_mysql
Removing service webapp_web
Removing network webapp_default

MODULE III-2
301

Docker Stacks Commands

COMMANDS DESCRIPTION

docker stack deploy Command used to deploy and update stacks of services defined in a stack file which is usually docker-
stack.yml

Docker stack ls List all stacks on the Swarm, including how many services they have

Docker stack ps Gives detailed information about a deployed stack. List which node each replica is running on, and shows
desired state and current state

docker stack rm Delete a stack from the Swarm.

MODULE III-2
302

Kubernetes (K8s)

Service for Cluster Management

Open sourced by Google
Applications orchestrator
Comparable to Docker Swarm
Automates the deployment, scaling and management of containerized
microservice applications
Support Docker and Rkt runtimes
Portable and flexible

MODULE III-2
303

Kubernetes

MODULE III-2
304

Kubernetes Architecture

MODULE III-2
305

Kubernetes Architecture

Master :

Responsible for cluster management

Entry point for administrative commands
Several masters behind a load balancer for HA architecture

Worker (node) :

Launch Pods of an application

Communicates with the Master
Provides resources to Pods

MODULE III-2
306

Kubernetes Architecture / Master

Kube-apiserver
- Entry point exposing the K8s HTTP REST API
Etcd
- Provide consistent and highly available key-value store used for persisting cluster state
- Configuration and service discovery
Kube-scheduler
- Select the node on which a pod will be launched
- Considers the resources needed and those available
Kube-controller-manager
- Process for managing the different controllers (node, replication, endpoint)
- Corrective actions if necessary
Cloud-controller-manager
- Provides cloud-provider specific integration capability into the core control loop
- Add additional controller to handle Persistent Volume Labels
MODULE III-2
307

Kubernetes Architecture / Worker (node)

Kubelet
- Process running on each machine in the cluster
- Ensure that a pod's containers rotate according to specification
- Communicates with the Master
Kube-proxy
- Allows services to be exposed to the outside
- Manages network rules
Container runtime engine
- Containerd(docker)
- CRI-O
- Rkt
MODULE III-2
308

Kubernetes Additional Services

Kube-dns

Provides cluster wide DNS Services. Services are resolvable to :

<service>.<namespace>.svc.cluster.local

Heapster
Metrics collector for Kubernetes cluster, used by some resources such as the Horizontal Pod
Autotscaler

Kube-dashboard

General purpose web-based UI for Kubernetes

MODULE III-2
309

Kubernetes Concept
Cluster : A collection of hosts that aggregate their available resources including CPU, RAM,
Disk, and their devices into a usable pool.
Master : A collection of components that make up the control plane of Kubernetes and are
responsible for all cluster decisions including both scheduling and responding to cluster
events.
Node/Worker : A single host, physical or virtual capable of running pods.He is managed by
the Master(s), and at a minimum runs both Kubelet and Kube-proxy to be considered part of
the Cluster.
Namespace : A logical cluster or environment. Primary method of dividing a cluster or
scoping access
Label : Key-value pairs that are used to identify, describe and group together related sets of
objects. Labels have a strict syntax and available character set.
Selector : Use labels to filter or select objects.

MODULE III-2
310

Kubernetes Object Categories

Management of applications launched on the Cluster (Deployment, Pod)

Discovery and Load Balancing (Service)

Configuration of applications (ConfigMap, Secret)

Storage (PersistentVolume, PersistentVolumeClaim)

Cluster configuration and metadata (Namespace)

MODULE III-2
311

Kubernetes Init and Config

Initialize the cluster with a Pod Network
$ kubeadm init --apiserver-advertise-address=MASTER_IP --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all

MODULE III-2
312

Kubernetes Init and Config

Join the cluster with a Worker
$ kubeadm join 10.10.1.120:6443 --token achk05.u8mglsx3nt027tcs --discovery-token-ca-cert-hash sha256:ebc61f8b4c310372d3ef5539a67d52ad76859468eb63cac4dbd50f5d6fb110b0

Check all nodes from Master node

$ kubectl get nodes

MODULE III-2
313

Kubernetes Init and Config

Show Merged Kubeconfig settings
$ kubectl config view

Display list of contexts

$ kubectl config get-contexts

Set the default context to MY_CLUSTER_NAME

$ kubectl config use-context MY_CLUSTER_NAME

Get a list of users

$ kubectl config view –o jsonpath='{.users[*].name}'

Display addresses of the Master and services

$ kubectl cluster-info

MODULE III-2
314

Kubernetes Objects Structure

ApiVersion : Identifies the version of the schema the object should have
Kind : Identifies the schema the object should have
Metadata : Adding name, labels, annotations, timestamp, namespace
Spec : Component specification / description
apiVersion: apps/v1
apiVersion: v1 apiVersion: v1 apiVersion: extensions/v1beta1 kind: Deployment
kind: Pod kind: Service kind: Ingress metadata:
metadata: metadata: metadata: name: nginx-deployment
labels: labels: name: www-domain labels:
app: vote app: vote namespace: vote app: nginx
name: vote name: vote spec: spec:
namespace: vote namespace: vote rules: NodePort replicas: 3
spec: spec: - host: www.example.com selector:
containers: type: NodePort http: matchLabels:
- name: www ports: paths: app: nginx
image: nginx:1.14.2 - name: "vote-service" - backend: spec:
port: 5000 serviceName: www containers:
targetPort: 80 servicePort: 80 - name: nginx
nodePort: 31000 image: nginx:1.14.2
selector: ports:
app: vote - containerPort: 80

MODULE III-2
315

Kubernetes Pods

Smallest unit of deployment in Kubernetes

A group of one or more application containers
- Shared storage, as Volumes
- Share Network, as a unique cluster IP address
Application split into several specifications of Pods
Configuration information that determine how a container should run

MODULE III-2
316

Kubernetes Pods Example

Here is an example of configuration for the "Hello World !
apiVersion: v1
Containers : describes the containers that make up our Pod, it is a Yaml list (there kind: Pod
metadata:
can be several containers) ;
- Container.name : the name of the container, which may or may not be the name
name: hello-world
spec:
containers:
- name: hello-world
of the induced Pod;
- Container.image : The Docker image called to generate the container (then Pod)
image: tutum/hello-world
imagePullPolicy: Always

- Container.imagePullPolicy: "Always" to get the latest version of the Docker

ports:
- containerPort: 80
stdin: true
tty: true
image tag
- Container.ports: The listening ports of the container
livenessProbe:
httpGet:

- Container.stdin & container.tty : To invoke a TTY/Stdin to be able to execute

path: /
port: 80
initialDelaySeconds: 10
periodSeconds: 10
commands and enter the container with a Shell
- Container.livenessProbe : Probe that ensures the proper functioning of the
container, and more precisely of the service embedded by the container.

MODULE III-2
317

Kubernetes Pods Commands

COMMANDS DESCRIPTION

Kubectl create -f POD.yaml Command used to run Pod based on the YAML file "POD.yaml"

Kubectl get pods List all pods deployed

Kubectl describe pod POD_NAME Describe a given pod

Kubectl logs POD_NAME [-c CONTAINER_NAME] Get logs from a Pod or/and specifically a container running inside the Pod

Kubectl exec POD_NAME [-c CONTAINER_NAME] -- COMMAND Execute command in an existing Pod or/and specially a container running inside the Pod

Kubectl delete pod POD_NAME Remove or delete a Pod

Kubectl port-forward POD_NAME HOST_PORT:CONTAINER_PORT Forwarding Port in a Pod (Allows to publish the port of a Pod on the host machine)

MODULE III-2
318

Kubernetes Services

Abstraction way to expose an Application running on a set of Pods

Ensures decoupling
- Replicas/microservice instances
- Microservice consumers
Load balancing between the underlying Pods
Persistent IP Address

MODULE III-2
319

Kubernetes Services types

MODULE III-2
320

Kubernetes Services types

These services can be used in different ways based on the types.

ClusterIP (default) : Exposes the Service on an internal IP in the cluster. This type makes the
Service only reachable from within the Cluster
NodePort : Exposes the Service on the same port of each selected Node in the Cluster using
NAT (Network Address Translation). Makes the Service accessible from outside the Cluster
using <NodeIP>:<NodePort>.
LoadBalancer : Creates an external load balancer in the current cloud provider (AWS, Azure,
GCP) and assigns a fixed, external IP to the Service.

MODULE III-2
321

Kubernetes Services Example

Here is an example of configuration for the "vote" service

ApiVersion : It specifies api version for the service

apiVersion: v1
Kind : It specifies what you are going to achieve with the file (deployment, pod, kind: Service
metadata:
service, secret, jobs, ingress) labels:
app: vote
Metadata : Information about the kind specified. name: vote
namespace: vote
Spec : Specification related to this kind spec:
selector:
Selector : Be careful to specify the correct label name here what we have used app: vote
type: NodePort
while creating the deployment. Else, it won't work properly. ports:
- name: "vote-service"
Type : Type of service (ClusterIP, NodePort, LoadBalancer,...) port: 5000
targetPort: 80
Ports : Specify the name of the service port, port number and target port nodePort: 31000

MODULE III-2
322

Kubernetes Service Commands

COMMANDS DESCRIPTION

Kubectl create -f SERVICES.yaml Command used to create services based on the YAML file "SERVICES.yaml"

Kubectl get services List all services deployed in the namespace. With [--all-namespaces] to list in all namespaces

Kubectl describe svc SERVICE_NAME Description of a Service

Kubectl delete svc SERVICE_NAME Remove or delete a Service

Kubectl port-fordward svc/SERVICE_NAME 5000 Listen on Local port 5000 and forward to port 5000 on service backend

Kubectl port-fordward svc/SERVICE_NAME 5000:TARGET_PORT Listen on Local port 5000 and forward to service target port

MODULE III-2
323

Kubernetes Deployment
A Kubernetes deployment is a resource object that provides declarative updates to
applications and allows you to explain life cycle.

Different levels of abstraction

- Deployment
- ReplicaSet
- Pod
Pod generally created via a Deployment
A Deployment manages ReplicaSets
ReplicaSet
- A version of the application
- Manages a set of Pods of the same specification
- Ensures that the Pods are active
MODULE III-2
324

Kubernetes Deployment

Deployment defines a "desired state"

- Specification of a Pod and the desired number of replicas
A controller to converge the "current state"

Manages updates of an application

- Rollout/Rollback/Scaling
- Creation of a new ReplicaSet when updating the application
Different update strategies
- Rolling update, blue/green, canary
MODULE III-2
325

Kubernetes Deployment Example

The following is an example of a Deployment. It creates a ReplicaSet to bring up
three nginx Pods:

ApiVersion : It specifies api version(v1) for the service apiVersion: apps/v1

kind: Deployment

Kind : It specifies what you are going to achieve with the file (deployment, pod, metadata:
name: nginx-deployment

service, secret, jobs, ingress) labels:

app: nginx

Metadata.labels : Pods are labeled app:nginx spec:

replicas: 3

Spec.replicas: Number of replicas created by the deployment (03) selector:

matchLabels:

Spec.selector : Defines how the Deployment finds which Pods to manage app: nginx
spec:

Spec.selector.matchLabels : Map of key-value pairs containers:

- name: nginx

Spec.containers : Create one container, name it and specify image and ports image: nginx:1.14.2
ports:
- containerPort: 80

MODULE III-2
326

Kubernetes Deployment Commands

COMMANDS DESCRIPTION

Kubectl create deployment nginx –image=nginx Command used to create deployment based on the YAML file "DEPLOY.yaml"
Kubectl apply -f DEPLOYMENT.yaml

Kubectl get deployments List all deployments in the namespace. With [--all-namespaces] to list in all namespaces

Kubectl get deployment DEPLOYMENT List a particular deployment

Kubectl describe deployment DEPLOYMENT Description of a Deployment

Kubectl delete deploy DEPLOYMENT Remove or delete a Deployment. Add [-l name=myLabel] to delete deployment with Label name=myLabel

Kubectl set image deployment/DEPLOYMENT www=image:v2 Rolling update "www" containers of "DEPLOYMENT", updating the image

Kubectl rollout history deployment/DEPLOYMENT Check the history of deployments including the revision

Kubectl rollout undo deployment/DEPLOYMENT Rollback to the previous deployment revision

Kubectl rollout undo deployment/DEPLOYMENT --to-revision=2 Rollback to a specific revision

Kubectl rollout status –w deployment/DEPLOYMENT Watch rolling update status of "DEPLOYMENT" deployment until completion

Kubectl rollout restart deployment/DEPLOYMENT Rolling restart of the "DEPLOYMENT" deployment

Kubectl autoscale deployment DEPLOYMENT --min=2 --max=10 Auto scale a deployment "DEPLOYMENT"

MODULE III-2
327

Quizz (1/2)

1. Which of the following elements exist in a Docker 2. In a Docker Swarm, what is the unique property of a
Compose file (Choose THREE correct answers)? replicated service (Choose TWO correct answers) ?

Volumes It can run multiple times in the Swarm

Services It runs a shadow copy of each container on another
Swarm node
Containers
It always runs all replicas on the same single node
Networks
in the Swarm
Nodes
It always runs on every n ode in the Swarm
It can run on an arbitrary number of Swarm nodes

MODULE III-2
328

Quizz (2/2)

1. Which of the following container properties are shared 3. The following command is issued on two docker nodes
by all containers within a Kubernetes Pod (Choose TWO “docker network create --driver bridge isolated_new”
correct answers)? Afterwards, one container is started at each node with the
parameter --network=isolated_new. It turns out that the
containers cannot interact with each other. What must be
The container base image done to correct that (Choose TWO correct answers) ?
All labels assigned to the containers
All resources limits and requests Add the option “—inter-container” to the docker
The IP address and networking ports network create command
All storage volumes Start the containers on the same node
Change the “—network” parameter of Docker create
with “nofence” at the end
2. Which element exist in the highest level of the definition
of every Kubernetes Object? Use an overlay network instead of a bridged network
Use a host network instead of a bridged network

MODULE III-2
329

MODULE III-3 : Container

Infrastructure
330

PLAN

DOCKER MACHINE

DOCKER DESKTOP

MODULE III-3
331

Docker Machine

Docker Machine lets you create Docker hosts on your computer (VirtualBox, VMWare), on
cloud providers (AWS, Azure), and inside your own data center.
It creates servers, installs Docker
It allows us to control the Docker engine of a VM created using docker-machine remotely
Docker Machine is another command-line utility used for managing one or more local or
remote machines
Local machines are often run in separate VirtualBox instances.

MODULE III-3
332

Docker Machine Driver

The drivers concept act as a connector to 3rd party services such as Azure, AWS, etc.
Allows to create a complete set of resources around the VM to easily manage it from each
service's admin portal
Generic driver allows you to convert an actual(existing) VM into a Docker-machine

MODULE III-3
333

Docker Machine Driver

MODULE III-3
334

Docker Machine Creation

Install Docker-machine (old method-depreciated)

$ curl -L https://github.com/docker/machine/releases/download/v0.14.0/docker-machine-uname -s-uname -m >/tmp/docker-machine
&& chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

Verifying version
$ docker-machine version

Create a Docker Machine on VirtualBox

$ docker-machine create -d virtualbox new-machine

Create a Docker machine with "generic" driver (remote existing VM)

$ docker-machine create --driver generic \

--generic-ip-address=${MACHINE_IP} \
--generic-ssh-key ${SSH_PUBLIC_KEY}\
--generic-ssh-user ${SSH_USER}\
new-machine

MODULE III-3
335

Docker Machine Creation

Create Docker Machine on Azure

docker-machine create --driver azure \
--azure-availability-set="MACHINE_NAME-as" \
--azure-subscription-id="${SUBSCRIPTION}" \
--azure-location "${AZURE_LOCATION}" \
--azure-open-port 80 \
--azure-open-port 443 \
--azure-size "${AZURE_MACHINE_SIZE}" \
--azure-subnet "${AZURE_VNET_NAME}-subnet" \
--azure-vnet "${AZURE_VNET_NAME}" \
--azure-resource-group "${RESOURCE_GROUP}" \
new-machine

List the machine you have created

$ docker-machine ls

Start a Docker Machine

$ docker-machine start new-machine

MODULE III-3
336

Docker Machine Commands

Deploy containers to a remote host (new-machine)

- Change the local docker environment variables to the new-machine ones
- Validate the active Docker Machine you point to
$ eval $(docker-machine env demo-machine)

- Run a container : Docker commands are not run locally but on Docker-machine
$ docker-machine active

- Curl the container deployed on Docker-machine

$ docker run –p 80:80 hello-world

$ curl $(docker-machine ip new-machine):80

SSH to Docker Machine

$ docker-machine ssh new-machine

Copy files to/from the machine

$ docker-machine scp ~/loalfile.txt new-machine:~/

MODULE III-3
337

Docker Machine Commands

COMMANDS DESCRIPTION

Docker-machine config Print the connection config for machine

Docker-machine env Display the commands to set up the environment for the Docker client

Docker-machine inspect Inspect information about a machine

Docker-machine ip Get the IP address of the machine

Docker-machine kill Kill a machine

Docker-machine ls List machines

Docker-machine provision Re-provision existing machines

Docker-machine regenerate-certs Regenerate TLS Certificates for a machine

Docker-machine restart Restart a machine

Docker-machine start Start a machine

Docker-machine status Get the status of a machine

Docker-machine stop Stop a machine

Docker-machine upgrade Upgrade a machine to the latest version of Docker

MODULE III-3
338

Docker Desktop

Docker Desktop is an easy-to-install application for your Mac, Linux, or Windows environment that
enables you to build and share containerized applications and microservices.

Docker Desktop replaces Docker Machine !!!

It provides a simple interface that enables you to manage your containers, applications, and
images directly from your machine without having to use the CLI to perform core actions
https://docs.docker.com/desktop/

MODULE III-3
339

Quizz

1. Which docker-machine sub command outputs a list of 2. Which of the following command list machines?
command that set environment variables which are
required 10 make dock, with a Docker host managed by
docker-machine ? Docker engine ls
Docker node ls
Docker machine ls
Docker machine list

MODULE III-3
340

PART I.V.
Configuration Management
…
341

MODULE IV-1 : Ansible

342

PLAN

ANSIBLE ARCHITECTURE

ANSIBLE INSTALLATION

ANSIBLE CONFIGURATION

ANSIBLE INVENTORY

ANSIBLE COMMANDS

ANSIBLE PLAYBOOKS

-- Structure

- Example
Roles and variables

ANSIBLE GALAXY

ANSIBLE TOWER

MODULE IV-1
343

Ansible

Ansible® is an open source, command-line IT automation software application

written in Python supported by Red Hat
Immutable infrastructure approach
It allows you to configure systems, deploy software and orchestrate more advanced
computing tasks
Continuous deployments or permanent updates without downtime
It also enables provisioning of virtual machines, containers and network, as well as
complete cloud computing infrastructures.
It also has a strong focus on security and reliability, featuring minimal moving parts
It uses OpenSSH for transport (with other transports and pull modes as alternatives)
Uses a human-readable language that is designed for getting started quickly without
a lot of training.

MODULE IV-1
344

Why Ansible?

MODULE IV-1
345

Ansible features

Management Node : Controles the entire execution of the

playbook
- Enables SSH connection
- Pushes and executes the small modules on the host machine
- Installs the software
Inventory : Provides the list of hosts where the Ansible need to
be run
Ansible removes the modules once those are installed so
expertly
There are no daemons, servers, or databases required

MODULE IV-1
346

Ansible vocabulary (1/2)

Management Machine : Machine on which Ansible is installed. Since Ansible is agentless, no software

is deployed on the managed servers.

Inventory: A file containing information about the managed servers

Playbook : A simple file in YAML format defining the target servers and the tasks to be performed

Play : An execution of Playbook

Task : A block defining a procedure to be executed (e.g. create a user or a group, install a software

package, etc)

Module : Group of similar Ansible commands that are expected to be executed from the client-side

MODULE IV-1
347

Ansible vocabulary (2/2)

Tag : Name set of the task and could be used later for just issuing certain group tasks or specific tasks

Role : Allows organizing the Playbooks and all the other necessary files (templates, scripts, etc) to

facilitate the sharing and reuse of code.

Collection : Includes a logical set of playbooks, roles, modules, and plugins

Facts : Global variables containing information about the system (machine name, system version,

network interface and configuration)

Notifier : Attributed to the task that shall call the handler and when the output is modified.

Handlers : To cause a service to be stopped or restarted in the event of a change

MODULE IV-1
348

Ansible architecture (1/2)

MODULE IV-1
349

Ansible architecture (2/2)

Modules : Ansible stack all functions as module utilities for reducing the duplication and
handling the maintenance
Plugins : Amplify Ansible's Core functionality. They are executed on the control node
Inventory : Depicts the machine that it shall handle in the file and gathers every machine in a
group which you have chosen
APIs : The Ansible APIs function as the bridge of Public and Private cloud services
CMDB : Kind of repository that acts as the data warehouse for IT installations
Hosts : Node systems that are automated using Ansible and machines like Linux, and
Windows.
Networking : Ansible is used for automating different networks and this uses the simple,
powerful, secure agentless automation framework for IT development and operations.

MODULE IV-1
350

Ansible in DevOps

The integration is a major factor for modern test-driven and application design. Ansible helps in
integrating it by providing a stable environment for both the Operations and Development and it
results in Continuous orchestration

MODULE IV-1
351

Ansible Installation

Install on Linux Control Node :

- Redhat/CentOS : $ sudo yum install epel-release && sudo yum install ansible

- Fedora : $ sudo dnf install ansible

- Ubuntu : $ sudo apt-get install ansible

- PIP : $ sudo pip install ansible

Install on Windows (Via WSL)

$ sudo apt-get update

$ sudo apt-get install python-pip git libffi-dev libssl-dev -y
$ pip install --user ansible pywinrm

MODULE IV-1
352

Ansible Configuration

Ansible supports several sources for configuring its behavior

- Configuration settings
- Command-line options
- Playbook keywords
- Variables
Each category overrides any information from all lower-precedence categories
Last "defined" wins and overrides any previous definitions

MODULE IV-1
353

Ansible Configuration settings

Configuration settings include both values from the ansible.cfg file and environment variables.

- ANSIBLE_CONFIG (environment variable if set)

- Ansible.cfg (in the current directory)
- ~/.ansible.cfg (in the home directory)
- /etc/ansible/ansible.cfg (default)
Content of config file

MODULE IV-1
354

Ansible SSH Key Authentication

Ansible uses SSH for authentication and assumes keys are in place (Controller)
$ ssh-keygen

Activate PubkeyAuthentication & ChallengeResponseAuthentication (Target)

MODULE IV-1
355

Ansible SSH Authentication

Setting up and transferring SSH keys allows playbooks to be run automatically (Controller)
$ ssh-copy-id [email protected]

Connecting with SSH key authorized on Target Node without asking password (Controller)
$ ssh [email protected]

Using passwords is possible (optionally)

MODULE IV-1
356

Ansible Inventory
The default location for inventory is a file called /etc/ansible/hosts
A different inventory file can be specified at the command line using the "-i <path>" option
Multiple inventory file can be used at the same time
Inventory can be pulled from dynamic or Cloud sources or different formats (YAML, ini)
Example (Basic INI & YAML)

mail.example.com all:
hosts:
[webservers] mail.example.com:
foo.example.com children:
bar.example.com webservers:
hosts:
[dbservers] foo.example.com:
one.example.com bar.example.com:
two.example.com dbservers:
three.example.com hosts:
one.example.com:
two.example.com:
three.example.com:

MODULE IV-1
357

Ansible Ad hoc Commands (1/4)

Ad-hoc commands are quick and easy but not reusable

Uses the /usr/bin/ansible command-line tool to automate a single task on one or more managed
nodes
They are used when you want to issue some commands on one or more server(s)
Format
$ ansible [pattern] -m [module] -a "[ARGUMENTS]" or
$ ansible <HOSTS> [-m <MODULE_NAME>] -a <"ARGUMENTS"> -u <USERNAME> [--become]

- HOSTS : Entry in the inventory file. To specify all host, use "all" or "*"
- MODULE_NAME : Modules available in the Ansible such as file, copy, yum, shell and apt
- ARGUMENTS : Pass values required by the module and can change according to the module used
- USERNAME : Specify the user account in which Ansible can execute commands.
- Become : Specify when to run operations that need sudo privilege.
MODULE IV-1
358

Ansible Ad hoc Commands (2/4)

Parallelism and shell commands

- To run reboot for all company servers in the group webservers in 11 parallel forks :
$ ansible webservers –a "sbin/reboot" -f 11 –u USERNAME

Managing files for file transfer : For SCP (secure copy protocol) to transfer many files to multiple
machines in parallel
- Transferring file on many machine in webservers group
$ ansible webservers –m copy –a "src=/etc/ssh/sshd_config dest=/tmp/sshd_config"

- Creating a new directory

$ ansible webservers –m file –a "dest=/tmp/new_dir mode=777 owner=user group=userg state=directory

- Deleting directories (recursively) and files

$ ansible webservers –m file –a "dest=/tmp/new_dir state=absent"

MODULE IV-1
359

Ansible Ad hoc Commands (3/4)

Managing packages
- Ensure that yum package is installed
$ ansible webservers –m yum –a "name=nano state=present"

- Ensure a specific version of a package is installed

$ ansible webservers –m yum –a "name=nano-7.0 state=present"

- Ensure a package is the latest version

$ ansible webservers –m yum –a "name=nano state=latest"

- Ensure a package is not installed

$ ansible webservers –m yum –a "name=nano state=absent"

Managing Users and Groups

- Create user accounts
$ ansible all –m user –a "name=gilbert password=<encrypted password>"

- Remove user accounts

$ ansible all –m user –a "name=gilbert state=absent"

MODULE IV-1
360

Ansible Ad hoc Commands (4/4)

Managing services
- Ensure a service is started on all webservers group
$ ansible webservers –m service –a "name=httpd state=started"

- Restart the service

$ ansible webservers –m user –a "name=httpd state=restarted"

- Check if a service is stopped

$ ansible webservers –m user –a "name=httpd state=stopped"

Gathering facts
- Discovered variables about a system. Facts can be used to implement conditional execution of
tasks and get ad hoc information about the systems.
$ ansible all –m setup

MODULE IV-1
361

Ansible Playbooks

Ansible Playbooks offer a repeatable, reusable, simple configuration management and multi-

machine deployment system

Playbooks are the files where Ansible code is written (in YAML format): variables, files,

templates, etc.

These files are descriptions of the desired state of your systems

Configuration management runbook with powerful control over scripting orchestrating

Describes which hosts to configure, and ordered list of tasks to perform on those hosts

Can use Version Control System

MODULE IV-1
362

Ansible Playbook Structure

Playbooks are a collection of one or more plays

Each play map a set of instructions (tasks) defined against a particular host
Tasks call modules
MODULE IV-1
363

Ansible Playbook Example

PLAYBOOK
-
name : Test connectivity to target servers
PLAY : Test connectivity to target servers
hosts: all
HOSTS : All (All machine in the inventory file)
tasks:
TASK : Gathering facts - name: Ping test
TASK : Ping test ping:

PLAY : RECAP

MODULE IV-1
364

Ansible Playbook Example

---
PLAYBOOK - hosts: all

PLAY : Install Apache to target server remote_user : vagrant

become: yes
HOSTS : All (All machine in the inventory file) tasks:

TASK : Gathering facts - name: Install httpd

yum: name=httpd update_cache=yes state=latest
TASK : Install httpd notify :

HANDLERS : Restart httpd - restart httpd

handlers:
PLAY : RECAP - name : restart httpd
service: name=httpd state=restarted

MODULE IV-1
365

Ansible Expressions (1/3)

Conditionals
Use variables to depend on the value of other variables
- Based on ansible_facts
tasks:
- name: Shut down Debian flavored systems
ansible.builtin.command: /sbin/shutdown -t now
when: ansible_facts['os_family'] == "Debian"

- Using Register and When : Based on registered variables

Configure a service after it is upgraded by an earlier task
tasks:

- name: Register a variable

ansible.builtin.shell: cat /var/www/html/index.html
register: html_contents

- name: Use the variable in conditional statement

ansible.builtin.shell: echo “html server contains the word sale"
when: html_contents.stdout.find(‘sale') != -1

MODULE IV-1
366

Ansible Expressions (2/3)

Iterating over a simple list ---

- Using loop - hosts: all

remote_user : vagrant
become: yes

For simple loops and is equivalent to with_list tasks:

- name: Add several users
ansible.builtin.user:
name: "{{ item }}"
state: present
groups: "wheel"
loop:
- testuser1
- testuser2

- Using with_ ---

- hosts: all
Be careful when changing with_items to loop, as with_items performed implicit become: true
gather_facts: no
single-level flattening. tasks:

The playbook above uses with_items to create users - name: install packages
apt:
name: "{{item}}"
state: present
with_items:
- apache2
- ufw
- mysql

MODULE IV-1
367

Ansible Expressions (3/3)

Using var_files
---
- hosts: all
vars_files:

Variables may also be included in a separate file, - vars.yml

become: true

using the vars_files section. tasks:

- name: install packages

The var “http_package” is filled in “vars.yml” apt:

name: "{{http_package}}"
state: present

Jinja2 expression
Ansible uses Jinja2 templating to enable dynamic expressions and access to a variables and facts.
Example : Create a template for a configuration file and deploy it to multiple environments

MODULE IV-1
368

Ansible Playbook roles & vars

Two of the key components of making playbooks reusable are Ansible variables and roles
Roles allow you to call a set of variables, tasks, and handlers by simply specifying a defined role

---
- hosts: linuxservers [linuxservers]
tasks:
ansibleclient2 ansible_host=172.17.11.5 webserver_pkg=httpd
- name: Install Apache Web Server
yum: name=httpd state=latest
[linuxservers:vars]
- name: upload index page
get_url: url=http://www.purple.com/faq.html dest=/var/www/html/index.html http_server_port=80

notify:
- openport80
- startwebserver
handlers:
- name: openport80
service: name=httpd state=started
- name: startwebserver
firewalld: port=80/tcp permanent=true state=enabled immediate=yes

MODULE IV-1
369

Ansible Playbook roles & vars

Creating a roles
$ sudo ansible-galaxy init linuxwebserver

$ sudo tree linuxwebserver

- Defaults : Default variables for the role

- Files : Contains files which can be deployed via this role
- Handlers : Contains handlers, which may be used by this role or outside
- Meta : Defines some meta data for this role
- Tasks: Contains the main list of tasks to be executed by the role
- Templates : Contains templates which can be deployed via this role
- Tests : Contains test example using this role
- Vars : Other variables for the role
MODULE IV-1
370

Ansible Playbook roles & vars

Edit tasks files
$ sudo nano linuxwebserver/tasks/main.yml

Edit Handlers files

$ sudo nano linuxwebserver/handlers/main.yml

MODULE IV-1
371

Ansible Playbook roles & vars

Write playbook which call the role created
$ pwd
$ sudo mkdir ~/deploy_with_role
$ cd ~/deploy_with_role
$ sudo nano webserver-roles.yml

Edit Inventory files

$ sudo nano hosts

MODULE IV-1
372

Ansible Playbook roles & vars

Result and display
$ ansible-playbook webserver-roles.yml –i hosts

MODULE IV-1
373

Ansible Playbook roles & vars

Ansible variables are one of the ways that make playbooks more generic.
Variables can be defined in multiple locations

- Inventory file
- Playbook
- Role definition
- Runtime

MODULE IV-1
374

Ansible Playbook roles & vars/Inventory

Variables can be assigned right along with the host definition in inventory file
Variables are set at both a group and individual host level
Actions : Task on role definition/Handlers/Inventory

MODULE IV-1
375

Ansible Playbook roles & vars/Playbook

The same variables can also be defined directly within the plays without calling the role
---
- hosts: linuxservers
remote_user : vagrant
Become : yes
vars :
http_server_port:80
webserver_pkg : httpd

tasks:
- name: Install Apache Web Server
yum: name={{webserver_pkg}} state=latest
notify:
- openport80
- startwebserver
- name: upload index page
[linuxservers]
get_url: url=http://www.purple.com/faq.html dest=/var/www/html/index.html
ansibleclient2 ansible_host=172.17.11.5
handlers:
- name: openport80 [linuxservers:vars]
service: name=httpd state=started
- name: startwebserver
firewalld: port={{http_server_port}}/tcp permanent=true state=enabled immediate=yes

MODULE IV-1
376

Ansible Playbook roles & vars/Role

Variables can be defined in either the “vars” or “default” directory of the role
“Default” directory stores values that are default settings which can be overridden when
using “vars” directory

MODULE IV-1
377

Ansible vars precedence model

Priority increases as you move down the list

To override absolutely any other defined variable, you should do it at runtime with the “–e”
flag

MODULE IV-1
378

Ansible Playbook roles & vars/Runtime

Despite having changed the packages to be installed and the server port on the inventory,
their explicit definition via the runtime prevails

MODULE IV-1
379

Ansible Galaxy

Ansible galaxy is essentially a large public repository of Ansible roles

Contains many roles that are constantly evolving and increasing

MODULE IV-1
380

Ansible Tower

MODULE IV-1
381

Ansible Tower
Ansible Tower (AWX) is a web-based solution that makes Ansible even more easy to use

MODULE IV-1
382

Quizz

1. Which Ansible Keyword allows tasks to be run only 5. An Ansible variable file contains the following content
under specific conditions? FlaskApp:
Version : 2.6
2. Which Ansible command is used to create and edit Which of the following strings can be used to reference
encrypted files which contain secrets used by ansible the defined variable?
when configuring remote systems?
?

3. Which configuration option in the Ansible inventory is

FlaskApp(Version) :
used to control privilege escalation of the remote user?
FlaskApp [‘Version’]
FlaskApp.Version
4. Which Ansible keyword is used in a playbook to store
the result of a task in a variable? FlaskApp{{version}}
FlaskApp-Version

MODULE IV-1
383

MODULE IV-2 : Other Tools

384

PLAN

-
CHEF

-
Chef Expression

-
Chef Workstation

-
Chef Server

-
Chef Nodes
Chef Tools

PUPPET

-- Architecture
Commands

MODULE IV-2
385

Chef
Chef® is an open source, systems management and cloud infrastructure automation platform
Chef transforms infrastructure into code to automate server deployment and management.
Chef is a configuration management tool for dealing with machine setup on physical servers, virtual
machines and in the cloud
Used by several companies including Facebook, Yahoo, Etsy.
There are three major Chef components : Workstation, Server, and Nodes

MODULE IV-2
386

Chef Expression (1/2)

Recipe
It can be defined as a collection of attributes which are used to manage the infrastructure
Cookbook
It is a collection of recipes
Resource
It is the basic component of a recipe used to manage the infrastructure with different kind of states
- Package
- Service
- User
- Group
- Template
- Cookbook_file
- Execute
- File
MODULE IV-2
387

Chef Expression (2/2)

Roles
- Help to improve configuration for redundant servers that all perform the same basic tasks.
- It is a categorization that describes what a specific machine is supposed to do
- Determines tools and settings to give to a specific machine
Environment
A designation meant to help an administrator know what stage of the production process a server is a part of.
By default, an environment called ”_default” is create. Environments can be created to tag a server as part of a
process group

Data Bags
- Data Bags store global variables as JSON data .
- Indexed for searching and can be loaded by a Cookbook or accessed during search
MODULE IV-2
388

Chef Workstation

The Workstation is the location from which all of Chef configurations are managed
Holds all the configuration data that can later be pushed to the central Chef Server
These configurations are tested in the workstation before pushing it into the Chef Server
A workstation consists of a command-line tool called Knife, that used to interact with the Chef Server
There can be multiple Workstations that together manage the central Chef Server
Workstations are responsible for performing the below functions :

- Writing Cookbooks and Recipes that will later be pushed to the central Chef Server
- Managing Nodes on the central Chef Server

MODULE IV-2
389

Chef Workstation/Recipes

A Chef recipes is a file that groups related resources, such as everything needed to configure a web server,
database, or a Load balancer.
These Recipes describe a series of resources that should be in a particular state, i.e. Packages that should
be installed, services that should running, or files that should be written.

#linux install httpd server, add fw rules and start service

package "httpd" do
action :install
end

include_recipe "apache::fwrules"

service "httpd" do
action [ :enable, :start]
end

MODULE IV-2
390

Chef Workstation/Cookbook

A Chef cookbook is the fundamental unit of configuration and policy distribution

Defines a scenario and contains everything that is required to support that scenario

- Recipes that specify which Chef Infra buit-in resources to use (in order)
- Attributes values which allow environment-based configurations such as dev or prod
- Custom resources for extending Chef Infra beyond the built-in resources
- Files and Templates for distributing information to systems
- Metadata (metadata.rb) which contains information about the cookbook such as the name, description
and version

#linux install httpd server, add fw rules and start service

package "httpd" do
action :install
end

MODULE IV-2
391

Chef Workstation / Managing Nodes

The Workstation will have the required command-line utilities, to control and manage every
aspect of the central Chef Server

Adding a new Node to the central Chef Server

Deleting a Node from the central Chef Server
Modifying Node configurations etc.

To perform above functions, Workstation have two major components :

Knife Utility : Command line tool for communication with the central Chef Server (Adding,
removing, changing configurations of Nodes – Upload Cookbooks and roles)
Local Chef repository : Store every configuration component of central Chef Server (With
Knife utility)

MODULE IV-2
392

Chef Server

Chef Server acts as a hub for configuration data

Stores Cookbooks, the policies applied to Nodes, and metadata that describes each
registered Node that is being managed by the Chef-Client
Send configuration details (Recipes, Templates, and file distributions) to the Node through
Chef-client
Scalable approach which distributes the configuration effort throughout the organization

MODULE IV-2
393

Chef Nodes

Nodes can be a cloud-based virtual server or an on-premise physical server, that is managed
using central Chef Server.
Chef-client is the main component agent present on the Node that will establish communication
with the Central Chef Server

The following functions are performed by Chef Client :

Responsible for interacting with the central Chef Server

Manages the initial registration of the Node to the central Chef Server
Pull down and applies Cookbooks on the Node
Periodic polling of the central Chef Server to fetch new configuration items if any

MODULE IV-2
394

Chef Tools

Chef-solo (standalone mode)

- Chef-solo is a command that executes Chef Client in a way that does not require the Chef
Server to converge Cookbooks
- Run as a daemon
- Uses Chef Client’s local mode and does not support some functionality (Centralized
distribution of Cookbooks and Authentication)
- Cookbook can be run from a local directory and a URL at which a “tar.gz” archive is located
Knife
- Interaction between a local chef-repo and the Chef Server
- Helps users to manage Nodes, Cookbooks and recipes, Roles, Environments, and Data Bags
MODULE IV-2
395

Chef / Workflow

MODULE IV-2
396

Puppet

Puppet uses a declarative language that models the infrastructure as a series of resources

Manifests consist of a set of JSON files, pull together these resources and define the

desired state of the final platform

Puppet stores manifests on the servers and uses them to create compiled configuration

instructions as needed, feeding them to agents via REST APIs

Facter is a puppet tool that discovers and reports facts about nodes which are then used to

create the manifests and configurations.

Facts include built-in details of the overall platform

Puppet architecture , master and agents (can run in a server-only model with command line

access)

MODULE IV-2
397

Puppet Architecture

MODULE IV-2
398

Puppet Architecture

Puppet Master

- Handles all the configuration related process in the form of puppet codes
- Create catalog and sends it to the targeted Puppet Agent
- Installed on a Linux based system
- SSL certificates are checked and marked
Puppet Node (Slave or Agent)

- Client installed, maintained, and managed by the Puppet Master

- Agent can be configured on any OS
- Applies configurations receive from Master to get the system into a desired state
- Send a report back to the Master
MODULE IV-2
399

Puppet Architecture

Config Repository

Storage area where all the servers and nodes related configurations are stored, and these
configurations can be pulled as per requirements

Facts

Key-value data pair. It contains information about the node or Master Machine.
It represents a puppet client states such as Operating System, network interface, IP Address, etc.

Catalog
Compiled format which result from the entire configuration and manifest files.
Catalog can be applie on the target machine.

MODULE IV-2
400

Puppet Command

COMMANDS DESCRIPTION

Puppet facter Show all facters

Puppet agent --enable Enable puppet agent to run on node

Puppet resource package Show all installed packages

Puppet resource Show all managed resources

Puppet module list List all installed module

Puppet config print all Print all configuration settings

MODULE IV-2
401

Quizz

1. Which of the following commands lists the cookbooks 2. What is the Puppet equivalent to an Ansible Playbook
available on a Chef Server ? called ?

Chef-server cookbook list Puppet Playbook

Kitchen cookbook list Puppet Manifest
Chef-solo cookbook list Puppet Catalog
Chef-client cookbook list Puppet Factsheet
Knife cookbook list Puppet Declaration

MODULE IV-2
402

PART V.
Service Operations
…
403

MODULE V-1 : IT Operations

and Monitoring
404

PLAN

DEVOPS MONITORING

-
PROMETHEUS

-
Architecture

-
Configuration file

-
Metric types
Exporter

GRAFANA

-- Data source
Dashboard

MODULE V-1
405

DevOps Monitoring

DevOps Monitoring is the practice of tracking and measuring the performance and health of

systems and applications in order to identify and correct issues early

Collect data on everything from CPU utilization to disk space to application response times

Helps teams avoid outage or degradation of service

Be able to debug and gain insight

Continuous monitoring is the process of regularly and vigilantly checking systems,

networks, and data for signs of performance degradation

MODULE V-1
406

Prometheus

Prometheus is an open-source technology designed to provide monitoring and alerting

functionality for Cloud-native environments.
Collect and store metrics as time-series data, recording information with a timestamp
Collect and record labels (key-value pairs)

Key features of Prometheus include :

Multidimensional data model
PromQL
No reliance on distributed storage
Pull model
Pushing time-series data
Monitoring target discovery
Visualization

MODULE V-1
407

Prometheus Architecture

MODULE V-1
408

Prometheus Architecture

Prometheus server : Scrapes and stores time series data

Client Libraries : Help in pulling metrics from the applications and send it to the Prometheus
Server (C#, Node.js…)
Service Discovery : Plays a key role in dynamic environments. Enables Prometheus to
identify the applications it needs to monitor and pull metrics from
Local storage : Store the data locally in a custom data store
Alertmanager : Receive alerts from the Prometheus Server and turn them into notifications
Exporter : Libraries and servers which help in exporting existing metrics from third-party
systems as Prometheus metrics (HAProxy, StatsD, Graphite)
Node Exporter : For physical and virtual machine metrics – hardware and kernel metrics
(Network, disk, CPU, RAM)
Data Visualization : Provides direct access to enter any expression and visualize its result
either in a table or graphed over time (Console templates, Grafana)

MODULE V-1
409

Prometheus Configuration file (1/2)

Default configuration file has four YAML blocks

Global : Contains global settings for controlling the Prometheus server’s behaviour

- Scrape Interval : Specifies the interval between scrapes of any application or service
- Evaluation Interval : Tells Prometheus how often to evaluate its rules
Alerting : Provided by an independent alert management tool called Alertmanager.
List each alertmanager used by the Prometheus server

MODULE V-1
410

Prometheus Configuration file (2/2)

Rule file : List of files that contain recording or alerting rules.

- Recording rules : Allow you to precompute frequent and expensive expressions and
to save their result as derived time series data

- Alerting rules : Allow you to define alert conditions. Prometheus will re-evaluate these
rules every 15 seconds (#evaluation_interval).

Scrape Configuration : Specifies all the targets that Prometheus will scrape

MODULE V-1
411

Prometheus Metric Types

Prometheus client libraries offer four core metric types

MODULE V-1
412

Prometheus Dashboard

MODULE V-1
413

Exporter
There are several libraries and servers which help in exporting existing metrics from third-party
systems as Prometheus metrics and maintained as part of the official Prometheus GitHub
- Databases (e.g., MySQL server exporter-Official)
- Hardware (e.g., Node exporter-Official)
- Trackers and CI (e.g., Jenkins exporter)
- Messaging systems (e.g., Kafka exporter)
- HTTP (e.g., Apache exporter)
Software exposing Prometheus metrics
- Ansible Tower (AWX)
- Kubernetes (direct)
Other utilities
- Java/JVM (EclipseLink metrics collector)
- Node,js (swagger-stats)
MODULE V-1
.
414

Data Visualization

Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary
push gateway for short-lived jobs

Its stores all scraped samples locally and runs rules over this data to either aggregate and
record new time series from existing data or generate alerts

An API consumers like Grafana can be used to visualize the collected data
- Used to query and visualize metrics
- Support multiple backend (Prometheus, MySQL, Datadog)
- Combine data from different sources
- Dashboard visualization fully customizable
- Each panel has a wide variety of styling and formatting options
- Supports templates and collection of add-ons and pre-built dashboards
MODULE V-1
415

Grafana

MODULE V-1
416

Grafana Data Sources

MODULE V-1
417

Grafana Dashboard

MODULE V-1
418

Grafana Dashboard

MODULE V-1
419

Quizz

1. How does Prometheus gather information about 2. Which section of the Prometheus configuration defines
monitored hosts and services ? which nodes are monitored?

It runs scripts on the Prometheus server which Scrape_config

perform tests and returns various metrics
Rules
It opens a webhook where monitored applications
Targets
have to submit various metrics
Nodes
It implements the ICMP and SNMP protocols to ping
and query remote services Listener
It uses HTTP to retrieve JSON encoded metrics
from the monitored objects
It queries a relational database for metrics written to
the database by monitored applications.

MODULE V-1
420

MODULE V-2 : Log

Management and Analysis
421

PLAN

LOGS
ELASTICSEARCH
LOGSTASH
KIBANA

MODULE V-2
422

Logs

Most of applications, containers and Virtual Machines constantly generate information about

numerous events

These events can be anything from severe errors to a simple notice

Collecting and analyzing this log data become challenging in a dynamic architecture or

microservice environment

Lots of users, systems (Routers, firewalls, servers) and logs (Netflow, syslogs, access logs,

service logs and audit logs)

Elastic stack (Logstash, Elasticsearch and Kibana) is used as a reference implementation.

MODULE V-2
423

Why Logs analysis

Monitor all above given issues as well as process operating systems

Helps DevOps teams to gain insights into their applications that allow them to better

identify areas for improvement

Helps to map patterns of user behavior

Reducing Customer Churn

Accelerating Releases

Optimizing Production Infrastructure Costs

MODULE V-2
424

ELK Stack

ELK Stack is a collection of three (03) open-source products (Elasticsearch, Logstash, and
Kibana).
- Elasticsearch : used for storing logs
- Logstash : used for both shipping as well as processing and storing logs
- Kibana : used for visualization of data through a web interface
ELK stack provides centralized logging in order to identify problems with servers or
applications

It allow you to search all the logs in a single place

Helps to find issues in multiple servers by connecting logs during a specific time frame

MODULE V-2
425

ELK Architecture

MODULE V-2
426

Elasticsearch

Elasticsearch is an open-source, extensively distributable, promptly adaptable, web search tool

that is available through a broad and expounds API.

Elasticsearch can control incredibly quick searches that help your information revelation
applications

NoSQL database built with RESTful API

Simplifies data ingest, visualization, and reporting
It helps in fast and Incisive search against large volumes of data
Real-time data and real-time analytics
Wide set of features like scalability, Hight availability and multi-tenant

MODULE V-2
427

Elasticsearch Architecture (1/5)

MODULE V-2
428

Elasticsearch Architecture (2/5)

Cluster (Elasticsearch cluster)

One or more servers collectively providing indexing and search capabilities node

Node
Single physical or virtual machine that holds full or part of your data and provides computing
power for indexing and searching your data

A node must accomplish several duties such as :

- Storing the data

- Performing operations on data (indexing, searching, aggregation, etc.)
- Maintaining the health of the cluster
MODULE V-2
429

Elasticsearch Architecture (3/5)

Based on the responsibilities, the following are different types of nodes that are supported

- Data Node
Node that has storage and compute capability. Participate in the CRUD, search, and aggregate
operations

- Master Node
Nodes are reserved to perform administrative tasks. It track the availability/failure of the data
nodes. They are responsible for creating and deleting the indices

- Coordinating-Only Node
Act as a smart load balancers. They are exposed to end-user requests. It appropriately redirects
the requests between data nodes and master node

MODULE V-2
430

Elasticsearch Architecture (4/5)

Index
It is a container to store data like a database in the relational databases. An index contains a
collection of documents that have similar characteristics or are logically related (e.g customer
data product catalog).

Document
Document is the piece indexed by Elasticsearch and it is represented in the JSON format

Mapping types
Mapping types is needed to create different types in an index and specified during index creation

MODULE V-2
431

Elasticsearch Architecture (5/5)

Shards
Shard is a full-featured subset of an index and help with enabling Elasticsearch to become
horizontally scalable (An index can store millions of documents and occupy terabytes of data,
this can cause problems with performance, scalability, and maintenance).

Replication
To ensure fault tolerance and high availability, Elasticsearch provides this feature to replicate the
data (Shards can be replicated)

- High Availability : Data can be available through the replica shard even if the node failed
- Performance : search queries will be executed parallelly across the replicas

MODULE V-2
432

Logstash

Logstash is a free and open-source, server-side data processing pipeline that can be used to
ingest data from multiple sources, transform it, and then send it to further processing or storage

Data collection pipeline tool that collect data inputs and feeds into Elasticsearch

Gathers all types of data from different sources and makes it available for further use

Unify data from disparate sources and normalize the data into a desired destination

Consists of three components :

- Input : passing logs to process them into machine understandable format
- Filters : It is a set of conditions to perform a particular action or event
- Output : Decision maker for processed event or log

MODULE V-2
433

Logstash Architecture

MODULE V-2
434

Logstash Data Inputs

Inputs are the starting point of Logstash configuration. By default, Logstash will automatically
create a stdin input if there are no inputs defined.

Some of the more commonly-used inputs are :

File : Reads from a file on the filesystem, much like the UNIX command tail -DF

Syslog : Listens on the well-known port 514 for syslog messages and parses.

Redis : Reads from a redis server, using both redis channels and redis lists

Beats : Processes events sent by Beats

MODULE V-2
435

Logstash Data Inputs Example

MySQL log file input {
file {
path => "/var/log/mysql/mysql.log“, "/var/log/mysql/mysql-show.log“, "/var/log/mysql/mysql-error.log“
type => “mysql"
}
}

Syslog input {
syslog {
port => 514
codec => cef
syslog_field => “syslog”
grok_pattern => “<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp} CUSTOM GROK HERE"
}
}

Redis
input {
redis {
id => “my_plugin_id”
}
}

MODULE V-2
436

Logstash Filters

Filters are intermediary processing devices in the Logstash pipeline. They can be combined with
conditionals to perform an action on an event if it meets certain criteria.

Some useful filters are :

GROK : Parse unstructured log data into something structured and queryable
MUTATE : Perform general transformations on event fields. Help to rename, remove,
replace, and modify fields in your events
DROP : Drop everything that gets to the filter
CLONE : Make a copy of an event, possibly adding or removing fields
GEOIP : Add information about geographical location of IP addresses (Displays amazing
charts in Kibana)

MODULE V-2
437

Logstash Filters Example

DROP
filter {
if [loglevel] == "debug" {
drop { }
}
}

GROK (from http request log 55.3.244.1 GET /index.html 15824 0.043)
filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}

MUTATE
filter {
mutate {
split => [“hostname”,”,”]
add_field => {“shortHostname”=>”%{hostname[0]}”}
}
}

MODULE V-2
438

Logstash Outputs

Outputs are the final phase of the logstash pipeline. An event can pass through multiple outputs,
but once all output processing is complete, the event has finished its execution
Some useful outputs are :

Elasticsearch : Store logs in Elasticsearch

File : Writes events to files on disk
Graphite : Writes metrics to Graphite
Http : Sends events to a generic HTTP/HTTPS endpoint
Statsd : Sends metrics using the Statsd network daemon
S3 : Sends Logstash events to the Amazon Simple Storage Service

MODULE V-2
439

Logstash Filters Example

Elasticsearch output {
elasticsearch {
hosts => "hostname"
data_stream => "true"
}
}

FILE
output {
file {
path => “C:/devops-book-labs/logstash/bin/log/output.log”
codec => line { format => "custom format: %{message}"}
}
}

Statsd
output {
statsd {
host => "statsd.example.org"
count => {
"http.bytes" => "%{bytes}"
}
}
}

MODULE V-2
440

Logstash Complete Example

MODULE V-2
441

Kibana

Kibana is a data visualization and exploration tool used for log and time-series analytics,
application monitoring, and operational intelligence use cases

Dashboard offers various interactive diagrams, geospatial data, and graphs to visualize

complex queries

It can be used for search, view, and interact with data stored in Elasticsearch directories

It helps users to perform advanced data analysis and visualize their data in a variety of

tables, charts, and maps.

Integrates different methods for performing searches on data

MODULE V-2
442

Kibana

MODULE V-2
443

ELK

Centralized logging can be useful when attempting to identify problems with servers or

applications

ELK is a collection of three open-source tools Elasticsearch, Logstash and Kibana.

Elasticsearch is a NoSQL database, Logstash is the data collection pipeline tool and Kibana

is a data visualization platform

Netflix, LinkedIn, Tripware, Medium all are using ELK stack for their business

ELK works best when logs from various Apps of an enterprise converge into a single ELK

instance

MODULE V-2
444

Quizz

1. Which of the following tasks can Logstash fulfill without 2. Which sections can exist in a Logstash configuration
using other components of the Elastic Stack? file

Receive log data from remote systems Output

Aggregate log data over a period of time filter
Store log data persistently Forward
Process log data to extract information Input
Forward log data to other services Generate

MODULE V-2
445

References/Modern Software Development

https://www.amazon.fr//dp/B0B5PK5P9V/
https://www.lpi.org/blog/2018/01/16/devops-tools-introduction-02-modern-software-development
https://www.tutorialspoint.com/sdlc/sdlc_quick_guide.htm
https://martinfowler.com/bliki/DevOpsCulture.html
https://www.tutorialspoint.com/sdlc/sdlc_quick_guide.htm
https://about.gitlab.com/topics/devops/
https://orangematter.solarwinds.com/2022/03/21/what-is-devops/
https://razorops.com/blog/benefits-of-devops/
https://www.guru99.com/agile-vs-devops.html
https://www.manutan.com/blog/fr/lexique/api-definition-et-application-dans-les-achats
https://slideplayer.com/slide/15761190/
https://restfulapi.net/rest-architectural-constraints/
https://reflectoring.io/complete-guide-to-csrf/
https://portswigger.net/web-security/csrf/tokens
https://www.wallarm.com/what/what-is-cross-site-request-forgery
https://portswigger.net/web-security/csrf
https://developer.mozilla.org/fr/docs/Web/HTTP/CORS
https://bunny.net/academy/http/what-are-cross-origin-resource-sharing-cors-headers/
https://www.ionos.fr/digitalguide/sites-internet/developpement-web/cross-origin-resource-sharing/
https://www.tutorialspoint.https://www.tutorialspoint.com/software_architecture_design/introduction.htm
https://medium.com/@concisesoftware/whats-the-difference-between-software-architecture-and-design-b705c2584631
https://www.slideshare.net/arslantumbin/software-architecture-vs-design
https://www.atlassian.com/microservices/microservices-architecture/microservices-vs-monolith
https://www.bmc.com/blogs/microservices-architecture/
https://www.techtarget.com/searchapparchitecture/definition/service-oriented-architecture-SOA
https://www.infoworld.com/article/2071889/what-is-service-oriented-architecture.html
https://subscription.packtpub.com/book/application-development/9781789133608/1/ch01lvl1sec04/service-oriented-architecture-soa
https://www.geeksforgeeks.org/service-oriented-architecture/
https://www.geeksforgeeks.org/service-oriented-architecture/
https://microservices.io/
https://smartbear.com/solutions/microservices/
https://anceret-matthieu.fr/2021/07/larchitecture-micro-services/
https://www.datadoghq.com/knowledge-center/serverless-architecture/
https://www.okta.com/identity-101/serverless-computing/
https://rubygarage.org/blog/monolith-soa-microservices-serverless
446

References/ Standard Components and Platforms

https://www.bridge-global.com/blog/mutable-vs-immutable-infrastructure/
https://k21academy.com/terraform-iac/why-terraform-not-chef-ansible-puppet-cloudformation/
https://www.hpe.com/us/en/what-is/data-storage.html
https://qumulo.com/blog/block-storage-vs-object-storage-vs-file-storage/
https://www.techtarget.com/searchstorage/tip/Object-storage-vs-file-storage-for-cloud-applications
https://www.bigdataframework.org/data-types-structured-vs-unstructured-data/
https://www.researchgate.net/figure/CAP-theorem-with-databases-that-choose-CA-CP-and-AP_fig1_282519669
https://www.bmc.com/blogs/cap-theorem/
https://www.bmc.com/blogs/acid-atomic-consistent-isolated-durable/
https://hevodata.com/learn/relational-database-vs-nosql/
https://www.tutorialspoint.com/Types-of-databases
http://talimi.se/database/
https://www.researchgate.net/figure/SQL-vs-MongoDB-terms_fig4_340622952
http://développeurweb.com/sql-vs-nosql-avantages-et-inconvenients/
https://www.cloudamqp.com/blog/what-is-message-queuing.html
https://blog.devgenius.io/everything-about-distributed-message-queue-ae6597d84b36
https://dachou.github.io/2018/09/28/cloud-service-models.html
http://bigdatariding.blogspot.com/2013/10/cloud-computing-types-of-cloud.html
447

References/ Source Code Management

https://www.lpi.org/blog/2018/01/30/devops-tools-introduction-04-source-code-management
https://www.geeksforgeeks.org/version-control-systems/
https://medium.com/@vemulasrinivas2505/version-control-systems-74375eb48961
https://medium.com/version-control-system/types-of-version-control-system-766a6b656088
https://devopsbuzz.com/centralized-vs-distributed-version-control-systems/
https://medium.com/@derya.cortuk/version-control-software-comparison-git-mercurial-cvs-svn-21b2a71226e4
https://gite.lirmm.fr/common-docs/doc-git/-/wikis/presentation/initiation_git_gitlab.pdf
https://www.slideshare.net/KishrorKumar/git-46566815
https://slideplayer.fr/slide/15375846/
https://www.tutorialspoint.com/git/git_life_cycle.htm
https://medium.com/analytics-vidhya/git-most-frequently-used-commands-9df9f200c235
https://git-scm.com/docs/git-revert
https://www.miximum.fr/blog/enfin-comprendre-git/
https://fr.slideserve.com/sakina/git-branching-powerpoint-ppt-presentation
https://blog.developer.atlassian.com/pull-request-merge-strategies-the-great-debate/
448

References/ CI/CD
https://www.lpi.org/blog/2018/02/06/devops-tools-introduction-05-continuous-delivery
https://www.slideshare.net/stevemac/introduction-to-cicd
https://medium.com/edureka/continuous-integration-615325cfeeac
https://aws.amazon.com/devops/continuous-integration/
https://www.guru99.com/software-testing-introduction-importance.html
https://www.leewayhertz.com/software-testing-process/
https://quodem.com/en/blog/what-is-software-testing-outsourcing/
https://www.leewayhertz.com/software-testing-process/
https://vivadifferences.com/difference-between-functional-and-non-functional-testing/
https://www.gcreddy.com/tag/types-of-test-tools
https://aws.amazon.com/devops/continuous-delivery/?nc1=h_ls
https://www.spiceworks.com/tech/devops/articles/cicd-vs-devops/
https://1902software.com/blog/automated-deployment/
https://developer.cisco.com/learning/labs/jenkins-lab-01-intro/
https://www.gocd.org/2017/07/25/blue-green-deployments.html
https://www.gocd.org/2017/08/15/canary-releases/
https://www.census.gov/fedcasic/fc2018/ppt/6CVazquez.pdf
https://formation.cloud-gfi-nord.fr/assets/integrationContinue/CI-Jenkins.pptx
https://subscription.packtpub.com/book/application-development/9781784390891/3/ch03lvl1sec20/the-jenkins-user-interface
https://blog.devgenius.io/what-is-jenkins-pipeline-and-jenkinsfile-96f30f3a29c
https://formation.cloud-gfi-nord.fr/assets/integrationContinue/CI-Jenkins.pptx
https://www.edureka.co/blog/jenkins-master-and-slave-architecture-a-complete-guide/
https://www.jenkins.io/doc/book/pipeline/syntax/
449

References/ Virtual Machine Deployment

Practical DevOps Tools, Gilbert Fongan Toussido
https://www.slideshare.net/NikunjDhameliya1/virtual-machine-69002899
https://techgenix.com/linux-virtual-machine-vm/
https://slideplayer.com/slide/13686844/
https://www.javatpoint.com/what-is-vagrant
https://www.agilequalitymadeeasy.com/post/vagrant-crash-course-development-environments-made-easy
https://medium.com/happy-giraffe/vagrant-box-up-and-go-88f5ff5d09d1
https://mariadb.com/kb/en/creating-a-vagrantfile/
https://slideplayer.com/slide/13686844/
450

References/ Cloud Deployment

https://www.lpi.org/blog/2018/01/23/devops-tools-introduction-03-cloud-components-and-platforms
https://cloudcomputingtypes.com/
https://www.academia.edu/40026388/Cloud_Deployment_Model
https://www.spiceworks.com/tech/cloud/articles/what-is-private-cloud-storage/
https://medium.com/@IDMdatasecurity/types-of-cloud-services-b54e5b574f6
https://stackshare.io/stackups/cloud-foundry-vs-openstack
https://www.cloudfoundry.org/
https://cloudinit.readthedocs.io/en/latest/
https://www.c-sharpcorner.com/article/what-is-cloud-init-and-boot-diagnostic-on-azure-vm/
451

References/ System Image Creation

https://www.packer.io/intro/use-cases
https://learn.hashicorp.com/tutorials/packer/get-started-install-cli
https://www.talentica.com/blogs/automation-of-ami-creation-using-packer/
https://devopscube.com/packer-tutorial-for-beginners/
https://lzone.de/ch
https://developer.hashicorp.com/packer/docs/commands/buildeat-sheet/pa
452

References/Container Usage
https://www.lpi.org/blog/2018/02/13/devops-tools-introduction-06-container-basics
https://www.slideteam.net/introduction-to-dockers-and-containers-powerpoint-presentation-slides.html#images-3
https://www.citrix.com/fr-fr/solutions/app-delivery-and-security/what-is-containerization.html
https://github.com/kaan-keskin/introduction-to-docker/blob/main/Introduction.md
https://kaushal28.github.io/Docker-Basics/
https://www.slideteam.net/introduction-to-dockers-and-containers-powerpoint-presentation-slides.html
https://kaushal28.github.io/Docker-Basics/
https://medium.com/@mccode/using-semantic-versioning-for-docker-image-tags-dfde8be06699
https://github.com/kaan-keskin/introduction-to-docker/blob/main/ContainerizingAnApp.md
https://kapeli.com/cheat_sheets/Dockerfile.docset/Contents/Resources/Documents/index
https://iceburn.medium.com/dockerfile-cheat-sheet-9f52aa4a99b3
https://docs.docker.com/engine/reference/commandline/build/
https://aboullaite.me/multi-stage-docker-java/
https://www.padok.fr/en/blog/docker-image-multi-staging
https://github.com/dockersamples/atsea-sample-shop-app/blob/master/app/Dockerfile
https://blog.octo.com/en/docker-registry-first-steps/
https://phoenixnap.com/kb/list-of-docker-commands-cheat-sheet
453

References/ Container Deployment and Orchestration

https://www.lpi.org/blog/2018/02/20/devops-tools-introduction-07-container-orchestration
https://docs.docker.com/compose/
https://www.baeldung.com/ops/docker-compose
https://k21academy.com/docker-kubernetes/docker-compose/
https://net2.com/how-to-install-docker-compose-on-ubuntu-20-04/
https://github.com/nigelpoulton/counter-app/blob/master/docker-compose.yml
https://faizanbashir.me/practical-introduction-to-docker-compose-d34e79c4c2b6
https://docs.docker.com/compose/compose-file/compose-file-v3/
https://github.com/kaan-keskin/introduction-to-docker/blob/main/ContainerNetworking.md
https://docs.docker.com/engine/tutorials/networkingcontainers/
https://www.section.io/engineering-education/introduction-to-docker-swarm-in-container-orchestration/
https://pt.slideshare.net/FawadKhaliq/microservices-architectures-with-docker-swarm-etcd-kuryr-and-neutron/6
https://www.section.io/engineering-education/introduction-to-docker-swarm-in-container-orchestration/
https://neo4j.com/blog/neo4j-container-orchestration-kubernetes-docker-swarm-mesos/
https://github.com/kaan-keskin/introduction-to-docker/blob/main/DockerSwarm.md
https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/
https://www.slideshare.net/thomasch/docker-distributed-application-bundle-stack-overview
https://buildvirtual.net/how-to-use-docker-stack-to-deploy-docker-containers/
https://www.slideshare.net/rajdeep/introduction-to-kubernetes
https://www.slideteam.net/understanding-kubernetes-architecture-with-diagrams-complete-deck.html
https://kodekloud.com/blog/kubernetes-features-for-beginner/
https://www.waytoeasylearn.com/learn/kubernetes-features/
https://github.com/gopal1409/KubernetesDocker
https://kubernetes.io/docs/reference/kubectl/cheatsheet/
https://phoenixnap.com/kb/kubernetes-objects
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
https://iximiuz.com/en/posts/kubernetes-api-structure-and-terminology/
https://kubernetes.io/docs/tutorials/kubernetes-basics/explore/explore-intro/
https://blog.ineat-group.com/2019/10/presentation-de-kubernetes/
https://blog.learncodeonline.in/kubernetes-core-concepts-services
https://www.learnitguide.net/2020/05/kubernetes-services-explained-examples.html
https://collabnix.com/3-node-kubernetes-cluster-on-bare-metal-system-in-5-minutes/
https://medium.com/the-programmer/working-with-deployments-roll-back-and-rolling-updates-bb4299488e34
454

References/Container Infrastructure
https://www.lpi.org/blog/2018/02/27/devops-tools-introduction-08-container-infrastructure
https://github.com/docker/machine
https://devopscube.com/docker-machine-tutorial-getting-started-guide/
https://medium.com/@cnadeau_/docker-machine-basic-examples-7d1ef640779b
https://lucanuscervus-notes.readthedocs.io/en/latest/Virtualization/Docker/Docker%20Machine/
https://docs.ionos.com/docker-machine-driver-1/v/docker-machine-driver/usage/commands
https://docs.docker.com/desktop/
455

References/ Ansible
https://www.lpi.org/blog/2018/03/13/devops-tools-introduction-10-ansible
https://datascientest.com/ansible
https://www.ansible.com/overview/how-ansible-works
https://spacelift.io/blog/ansible-vs-terraform
https://docs.rockylinux.org/books/learning_ansible/01-basic/
https://dev.to/rahulku48837211/ansible-architecture-and-setup-2355
https://www.fita.in/devops-tutorial/
https://www.openvirtualization.pro/red-hat-ansible-automation-architecture-and-features/
https://www.fita.in/devops-tutorial/
https://www.fita.in/devops-tutorial/
Installation Guide — Ansible Documentation
https://docs.ansible.com/ansible/latest/reference_appendices/general_precedence.html#general-precedence-rules
https://www.openvirtualization.pro/ansible-part-i-basics-and-inventory/
https://tekneed.com/creating-and-managing-ansible-configuration-file-linux/
https://www.c-sharpcorner.com/article/getting-started-with-ansible-part-3/
https://www.ssh.com/academy/ssh/copy-id
https://linuxhint.com/use-ssh-copy-id-command/
https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#intro-inventory
https://docs.ansible.com/ansible/latest/user_guide/intro_adhoc.html
https://www.javatpoint.com/ansible-playbooks
https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html
https://digitalvarys.com/how-to-write-an-ansible-playbook/
https://www.slideshare.net/knoldus/introduction-to-ansible-81369741
https://www.bogotobogo.com/DevOps/Ansible/Ansible_SettingUp_Webservers_Nginx_Install_Env_Configure_Deploy_App.php
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_conditionals.html
https://www.ansible.com/overview/how-ansible-works
https://linuxhint.com/ansible-with_item/
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_loops.html
https://ericsysmin.com/2019/06/20/how-to-loop-blocks-of-code-in-ansible/
https://www.toptechskills.com/ansible-tutorials-courses/ansible-include-import-variables-tutorial-examples/
https://www.dasblinkenlichten.com/ansible-roles-and-variables/
https://galaxy.ansible.com/docs/contributing/creating_role.html
https://www.redhat.com/sysadmin/intro-ansible-tower
https://developer.arubanetworks.com/aruba-aoscx/docs/ansible-tower-overview
456

References/ Other Configuration Management Tools

https://www.lpi.org/blog/2018/03/20/devops-tools-introduction-11-other-configuration-management-tools
https://www.devopsschool.com/blog/what-is-chef-tools-benefits-of-using-chef-tools/
https://github.com/Tikam02/DevOps-Guide/blob/master/Infrastructure-provisioning/Chef/chef-commands.md
https://www.tutorialspoint.com/chef/chef_overview.htm
https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-environments-in-chef-to-control-server-configurations
https://docs.chef.io/data_bags/
https://www.devopsschool.com/blog/what-is-chef-tools-benefits-of-using-chef-tools/
https://www.linode.com/docs/guides/install-a-chef-server-workstation-on-ubuntu-18-04/
https://nabeelvalley.co.za/docs/automation/manage-node-with-server/
https://severalnines.com/blog/how-automate-daily-devops-database-tasks-chef/
https://docs.chef.io/cookbooks/
https://medium.com/edureka/chef-tutorial-8205607f4564
https://docs.chef.io/chef_solo/
https://docs.chef.io/workstation/knife/
https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-environments-in-chef-to-control-server-configurations
https://www.techtarget.com/searchitoperations/tip/What-is-the-Puppet-configuration-management-tool-and-how-does-it-work
https://www.javatpoint.com/puppet-architecture
https://github.com/Tikam02/DevOps-Guide/blob/master/Infrastructure-provisioning/Puppet/puppet-commands.md
457

References/ IT Operations and Monitoring

https://www.lpi.org/blog/2018/03/27/devops-tools-introduction-12-it-operations-and-monitoring
https://www.crowdstrike.com/cybersecurity-101/observability/devops-monitoring/
https://www.tigera.io/learn/guides/prometheus-monitoring/
https://k21academy.com/docker-kubernetes/prometheus-grafana-monitoring/
https://samirbehara.com/2019/05/30/cloud-native-monitoring-with-prometheus/
https://sensu.io/blog/introduction-to-prometheus-monitoring
https://prometheus.io/docs/instrumenting/exporters/
458

References/ Log Management and Analysis

https://www.lpi.org/blog/2018/04/03/devops-tools-introduction-13-log-management-and-analysis
https://www.mezmo.com/learn-log-management/what-is-log-
analysis#:~:text=Log%20analysis%20also%20provides%20significant,map%20patterns%20of%20user%20behavior.
https://devops.com/business-value-of-log-analysis/
https://www.guru99.com/elk-stack-tutorial.html
https://www.yobitel.com/single-post/2019/12/03/elk-kubernetes-stack-in-cloud-native-way-scope-about-retail-industry
https://www.velotio.com/engineering-blog/elasticsearch-101-fundamentals-core-concepts
https://www.oreilly.com/library/view/learning-elastic-stack/9781787281868/5a399bbb-a9d6-4338-814c-3058e1fa4e3f.xhtml
https://www.bmc.com/blogs/logstash-using-data-pipeline/
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-statsd.html
459

THANKS!
Any questions?
You can find me at [email protected]