Table of Contents

1
preface
-;;pter 1: Introducing DevOps and Continuous Delivery 7
Introducing DevOps 7
How fast is fast? 9
The Agile wheel of wheels 10
Beware the cargo cult Agile fallacy 11
oevOps and ITIL 13
summary 13
Chapter 2: A View from Orbit 15
The DevOps process and CD - an overview 15
The developers 16
The Revision Control System 18
The build server 19
The artifact repository 20
Package managers 21
Test environments 21
Staging/production 22
Release management 23
Scrum, Kanban, and the delivery pipeline 24
Wrapping up- a complete example 25
Identifying bottlenecks 26
Summary 26

Chapter 3: How DevOps Affects Architecture 27

Introducing software architecture 27
The monolithic scenario 28
The Twelve Factors 29
Factor 1 - code base 29
Factor 2 - dependencies 30
Factor 3 - config 30
Factor 4 - backing services 30
Factor 5 - build, release, run 30
Factor 6 - processes 30
Factor 7 - port binding 31
Factor 8 - concurrency 31
Factor 9 - disposability 31
Factor 10 - dev/prod parity 31
Factor 11 - logs 32
Table of Contents

Factor 12 _ admin processes

Architecture rules of thumb 32
The separation of concerns 32
The principle of cohesion 32
Coupling 32
Back to the monolithic scenario 33
A practical example 33
Three-tier systems J4
The presentation tier J4
The logic tier 35
The data tier 35
Handling database migrations 35
Rolling upgrades 36
Hello world in Liquibase 37
The changelog file 38
The pom.xml file 38
39
Manual installation
40
Microservices
42
Interlude - Conway's law 43
How to keep service interfaces forward compatible
44
Microservices and the data tier 45
DevOps, architecture, and resilience 45
Summary 47
Chapter 4: Everything is Code 49
The need for source code control 50
The history of source code management 50
Roles and code 51
Which source code management system? 52
A word about source code management system migrations 53
Choosing a branching strategy 54
Branching problem areas 56
Artifact version naming 57
Choosing a client 58
Setting up a basic Git server 59
Shared authentication 60
Hosted Git servers 61
Large binary files 62
Trying out different Git server implementations 63
Docker intermission 63
Gerrit 64
65
Installing the g_it-review package
65
The value of history revisionism
The pull request model 68

[ii]
 69
Gitlab 71
sum mar y
Chapter 5: Buil din g t he Cod e
n
Why do we buil d cod e?
The man y face s of buil d sys tem s
,.,
1-J

f!
The Jen kins b uild serv er ff}
Man agin g b ulld dep end enc ies
HD
The fina l artif act ..,,
Cheat ing w ith FPM ij~
Con tinu ous Inte grat ion
a3
Con tinu ous Deli very ~
Jen kins plug ins M
The hos t serv er 85
Buil d slav es 86
Soft war e on the hos t 57
Trig gers 87
Job cha inin g and buil d pipe line s ea
A look at the Jen kins files yste m layo ut ~
Buil d serv ers and Infr astr uctu re as Cod e 00
Building by depe nden cy orde r 02
Build pha ses ~
Alte rnat ive buil d serv ers (l3
Coll atin g qua lity mea sure s 94
Abo ut buil d stat us visu aliz atio n 9d
Tak ing buil d erro rs seri ous ly 95
Rob ustn ess S5
Sum mar y
Chapter 6: Tes ting the Cod e
Man ual test ing
-:,

18
Pros and con s with test a utom atio n I ~C
Unit test ing 10 l
xUn it in gen eral and JUn it in part icul ar 10:1
A JUni t exam ple t i.J
Moc king I '.J:j
Tes t cov erag e 1114
Auto mat ed inte gra tion test ing ((!5
Dock er in auto mate d testi ng i06
Arquillian L06
Per form anc e test ing 107
Auto mat ed acc epta nce test ing 100
Auto mat ed GUI test ing

[ iii J
Table of Contents ~
Integrating Selenium tests in Jenk ins 110
JavaScript testing . . 111
Testing backend integratio n poin ts 112
Test-driven development 114
REPL-driven development . 115
A complete test automation ~ce~ ano 115
Manually testing our web apphcatron 116
Running the automated test 118
Finding a bug
118
Test walkthrough . . 118
Handling tricky dependencies with Dock er 122
124
Summary
125
Chapter 7: Deploying the Code
Why are there so many deployment syst ems ? 126
Configuring the base OS
127
127
Describing clusters
128
Delivering packages to a system
130
Virtualization stacks
132
Executing code on the client
132
A note about the exercises
133
The Puppet master and Puppet agents
134
Ansible
138
Deploying with Chef
140
Deploying with SaltStack
142
Salt versus Ansible versus Puppet execution mod els
142
Vagrant
145
Deploying with Docker
145
Comparison tables
146
Cloud solutions
146
AWS
147
Azure
148
Summary
149
Chapter 8: Monitoring the Code
149
Nagios
156
Munin
160
Ganglia
164
Graphite
Log handling 166
167
Client-side logging libraries
The ELK stack 170
Summary 172

[iv]
~

d
- Tablt of Co11tr,rl:s

Chapter 9: ls~ue Tracking f73

What are issue trackers used for? 't73
some examples of workflowa and h1eue 1 175
What do we need from an laaue tracker? 1?7
Problems with issue tracker prollferatlon 180
All the trackers 180
Bugzilla 181
Trac 18&
Redmine 196
The Gitlab issue tracker 202
Jira 200
Summary 20a
Chapter 10: The Internet of Things and DevOps 200
Introducing the loT and DevOps 2rS
The future of the loT according to the market 21J
Machine-to-machine communication 215
loT deployment affects software architecture 218
loT deployment security 216
Okay, but what about DevOps and the loT again? 217
A hands-on lab with an loT device for DevOps 219
Summary 225
Other Books You May Enj o y '227
Index 231

[ V]
-

Introd ucing DevOps and

Continuous Delivery
Welcome to Practical OevOps!

n,e first chapter of this book will deal with the background of OevOps and setting the
scene for how DevOps fits into the wider world of Agile systems development.
An important part of OevOps is being able to explain to your coworkers what OevOps is
and what it isn't.
The faster you can get everyone aboard-the DevOps train, the sooner you will be able to
perform the actual technical implementation!
In this chapter, we will cover the following topics:
• Introducing OevOps
• How fast is fast?
• The Agile wheel of wheels
• The cargo cult Agile fallacy
• OevOps and ITIL

Introducing DevOps
DevOps is, by definition, a field that spans several disciplines. It is a field that is very
practical and hands-on, but, at the same time, you must understand both the tecfmic~l
background and the non-technical cultural asp ects. This book covers both the practical and
soft skills required for a best-of-b reed DevOps implementation in your organization.
/11l ro1l11ci11g Dt'P0p , 1111d C011li11 11011c; DelrPa y

11,c word DrvOp s ic; a combin,,tion of lhc _w ordc; rfrt1L'iop111_c11I and ~pernlio11. This ~ordp
lay
a\rcad y serves to t?,ivc w, t1 hint uf tlw b,1i;11. nature o( the 1dc,1 behind DevOps. It 1s
a
practice where the 1.n\\,ihtm1linn lwt w,'l'n d1H1.•1l' nl di "iclplin c!, of c;;oftwarc develo
pment ic.;
cncour.\gcct .
11,c mi~ii, nl tlw wo,d l),•vOp::- ,md tlw l'tHly d,1 y•, nf llw Ik vOpH movcm C'nt ran be
tracked
rt\lh1..·r prccb dy. Pi'l\tilk lkbob I!,~, hoflwt,n' dt•vclo1wr ,md com,u lt,,nt with 1·xpcri
cnce in
mat'\v tidd~ within n. I le was frustrated with llw di vide lwlWl'cn dt·vtlc1perq ,ind
oper..1lion~ personnel. l le tried gelling people interested in the problem dt n mforencc't,
bu t
there wasn't much interest inHiall y.
ln 2009, there was a well-re ceived talk at the O'Reilly Velocity Confe rence: JO+
Deploy., pt'r
Dny: Dev nnrl Ops Cooperation at Flickr. Patrick then decide d to organize an event in Ghent,
Belgium, called DevOpsDays. This time, there was much interest, and the conler
ence was a
success. The name DevOpsDays struck a chord, and the conlerence has becom
e a recurring
event. DevOpsDays was abbreviated to DevOp s in conversation s on Twitter and
various
internet forums.

The DevO ps movement has its roots in Agile software development principles.
The Agile
Manifesto was written in 2001 by a number of individuals wanting to improve
the then
current status quo of system development and find new ways of working in the
software
development industry. The following is an excerpt from the Agile Manifesto, the
now
classic text, which is available on the web at http : //agil eman ifesto .org / :

"Individuals and interactions over processes and tools

Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more."
In light of this, DevOps can be said to relate to the first principle, Individuals and
interaction:5
over processes and tools.
1his may be seen as a fairly obvious beneficial way to work- why do we even have
to state
this obvious fact? Well, if you have ever worked in any large organization, you will
know
that the opposite principle seems to be in operation instead. Walls between differe
nt parts
of an organization tend to form easily, even in smaller organizations, where at first
it would
appear to be impossible for such walls to form.

l 8l
 Chapter 1

th nd indiv idual s are very

DevO p s, en, te s to emph asize that inter actio ns between
th ng these inter actio ns happ en
impo rtant , and at te~o l~gy may possibly assist in maki
th coun ter-in tuitiv e, given that
and ~ear d~w ~ e walls ~sid e organ izatio ns. This may seem
, but my opin ion is that any
the first prme1ple favor s inter actio n amon g peop le over tools
erly, they can facilitate all
tool can h~ve sever al ~ffects when used . If we use the tools prop
of the desir ed prop ertie s of an Agile work place .
repor t bugs. Quit e often ,
A very simp le exam ple may be the choice of syste ms used to
syste ms to hand le tasks and
developm ent team s and quali ty assur ance team s use diffe rent
furth er separ ates them when
bugs. This creat es unne cessa ry friction betw een th.e team s and
The opera tions team may, in
they shou ld really be focusing on work ing together instead.
the organ izatio n's serve rs.
turn, use a third syste m to hand le reque sts for deplo ymen t to
imme diate ly recog nize all
An engin eer with a DevO ps mind set, on the other hand , will
rties. It shou ld be possi ble for
three syste ms as being work flow syste ms with similar prope
perh aps twea ked to gene rate
everyone in the three diffe rent team s to use the same system,
d be smal ler main tenan ce
different view s for the diffe rent roles. A furth er benefit woul
costs, since three syste ms are repla ced by one.
Deliv ery (CD). Simp ly put,
Another core goal of DevO ps is autom ation and Continuous
hum an interaction, wher e true
autom ating repet itive and tedio us tasks leaves more time for
value can be creat ed.

How fast is fast?

to consi der the time to mark et
The turna roun d for DevO ps processes must be fast. We need
in the smal ler persp ectiv e.
in the large r persp ectiv e, and simp ly stay focused on our tasks
This line of thou ght is also held by the CD movement.
CD are in fact diffe rent
As with many thing s Agile, many of the ideas in DevO ps and
ntion betw een the two
name s of the same basic concepts. There really isn't any conte
concepts; they are two sides of the same coin.
, more efficient, and more
DevO ps engin eers work on maki ng enter prise processes faster
ved when ever possible.
reliable. Repetitive manu al labor, whic h is error-prone, is remo
DevO ps imple ment ation s.
It's easy, howe ver, to lose track of the goal when work ing with
keep track of deliv ering
Doing nothi ng faster is of no use to anyo ne. Instead, we must
increased busin ess value .

[9]
Introduci11g DevOps and Conti11uous DelivenJ

For instan ce, incre ased comm unica tion betw een roles in the
organ izatio n has clear value .
Yotrr prod uct owne rs may be wond ering how the deve lopm
ent proce ss is going and are
eager to have a look. In this situat ion, it is usefu l to be able
to deliv er incre ment al
impr ovem ents of code to the test envir onme nts quick ly and
effici ently . In the test
envir onme nts, the invol ved stake holde rs, such as prod uct
owne rs and, of cours e, the
quali ty assur ance team s, can follow the progr ess of the deve
lopm ent proce ss.
Anot her way to look at it is this: if you ever feel yours elf losin
g focus becau se of needl ess
waiti ng, some thing is wron g with your proce sses or your
toolin g. If you find yours elf
watc hing video s of robot s shoo ting ballo ons durin g comp
ile time, your comp ile times are
too long!
The same is true for team s idlin g while waiti ng for deplo
ys and so on. This idlin g is, of
cours e, even more expe nsive than that of a singl e indiv idual
.
Whil e robot -shoo ting pract ice video s are fun, softw are deve
lopm ent is inspi ring too! We
shou ld help focus our creat ive poten tial by elimi natin g unne
cessa ry overh ead.

Th e Ag ile wh ee l of wheels
Ther e are sever al diffe rent cycle s in Agile deve lopm ent,
from the Portf olio level throu gh to
the Scru m and Kanb an cycle s and down to the Cont inuou
s Integ ratio n (Cl) cycle. The
emph asis on whic h caden ce work happ ens in is a bit diffe
rent depe ndin g on whic h Agile
fram ewor k you are work ing with. Kanb an emph asize s the
24-ho ur cycle , whic h is popu lar
in oper ation s team s. Scru m cycle s can be two to four week
s and are often used by
deve lopm ent team s using the Scru m Agile proce ss. Long
er cycle s are also comm on and are
calle d Prog ram Incre ment s (PI), whic h span sever al Scru
m Sprin t cycle s, in the Scale d
Agil e Fram ewor k (SAFe®):

~ • ..._ ♦ Portfolio
• ~m \
I (,I k~_6J@
,.
_n\ ~ ■
-. 1
•

~~~.-. •
'~ ~ ~r&.
I I '

edit/compile/debug

The Agile wheel of wheels

[ 101
 Chapter I

DevOps must be able to support all these cycles. This is qui te nt1turnl given the centra l
theme of DevOps: cooperation bclwcen d isciplines in nn Agile org<1nizt1tion.

The most obvious and measurably concrete benefits of DevOps occur in the c.; hortcr cycles,
which in turn makes the longer cycles more effi cient. Tak,, rnrt' of llw pfln11ieB, and the pounds
will take care of themselves, as the old adage goes.

Here are some examples of when DevOps can benefit Agi le cycles:

• Deployment systems, maintained by DevOps engineers, make the deliveries at

the end of Scrum cycles faster and more effi cient. These can take place with a
periodicity of two to four weeks.
• In organizations where deployments are done mostly by hand, the time to
deploy can be several days. Organizations that have these inefficient dep loyment
processes will benefit greatly from a DevOps mindset.
• The Kanban cycle is 24 hours, and it's therefore obvious that the deployment
cycle needs to be much faster than that if we are to succeed with Kanban.
• A well-designed DevOps CD pipeline can deploy code from being committed to
the code repository to production in a matter of minutes, depending on the size
of the change.

Beware the cargo cult Agile fallacy

Richard Feynman was awarded the Nobel prize for his work in the field of quantum
physics in 1965. He noticed a common behavior among scientists, in which they went
though all the motions of science but missed some central, vital ingredient of the scientific
process. He called this behavior cargo cult science, since it was reminiscent of the cargo
cults in the Melanesian South Sea islands. These cargo cults were formed during the World
War II when the islanders watched great planes land with useful cargo. After the war
stopped, the cargo also stopped coming. The islanders started simulating landing strips,
doing everything just as they had observed the American military do, in order for the
planes to land.

[ 11]
 r
~.
I
-.
/11 /rorl11 ci11g DevOps n11rl Co11tim1011s Delivery

· -----
We are not working in an Agile or DevOps-oriented manner simply ~ecause we have a free
morning where we drink coffee and chat about the weather. We don t have ~ DevOps
pipeline just because we have a Puppet implementation that only the operations team
knows anything about.
It is very important that we keep track of ou r goals and co~ti~uously question ~het~er we
are doing the right thing and are still on the right track. This. 1s cent~al to ~ll Agile thinking.
It is, however, something that is mani festly very hard to do m practice. It 1s easy to wind up
as followers of the cargo cults.
When constructing deployment pipelines, for examp~e, keep in_mind why we are building
them in the first place. The goal is to allow people to interact ~1th new s~stems faster and
with less work. This, in turn, helps people with different roles interact with one another
more efficiently and with less turnaround.
If, on the other hand, we build a pipeline that only helps only one group of people achieve
their goals, for instance, the operations personnel, we have failed to achieve our basic goal.

While this is not an exact science, it pays to bear in mind that Agile cycles, such as the
Sprint cycle in the Scrum Agile method, normally have a method to deal with this situation.
In Scrum, this is called the Sprint Retrospective, where the team gets together and discusses
what went well and what could have gone better during the sprint. Spend some time here
to make sure you are doing the right thing in your daily work.

A common problem here is that the output from the Sprint Retrospective isn't really acted
upon. This, in turn, may be caused by the unfortunate fact that the identified problems
were really caused by some other part of the organization that you don't communicate well
with. Therefore, these problems come up again and again in the retrospectives and are
never remedied.

If you recognize that your team is in this situation, you will benefit from the DevOps
approach since it emphasizes cooperation between roles in the organization.

To summarize, try to use the mechanisms provided in the Agile methods in your methods
themselves. If you are using Scrum, use the Sprint Retrospective mechanism to capture
potential improvements. This being said, don't take the methods as gospel. Find out what
works for you.

l 12 l

_.....11111
 Chapter 1

DevOps and ITIL

This section explains how DevOps and other ways of working coexis t and fit together as a
larger whole.

DevOps fits well together with m any frameworks for Agile or Lean enterprises. SAFe®
specifically mentions DevOps. There is nearly neve r any disagreem ent be tween proponents
of different Agile practices and DevOps since DevOps originated in the Agile environment.
The story is a bit different with the Information Technology Infrastructure Library (ITIL),
though.

ITIL, which was formerly known as the Information Technology Infrastructure Library, is a
practice used by many large and mature organizations.

ITIL is a large framework that formalizes many aspects of the software life cycle. While
DevOps and CD hold the view that the changesets we deliver to production should be
small and happen often, at first glance, ITIL would appear to hold the opposite view. It
should be noted that this isn't really true. Legacy systems are quite often monolithic, and in
these cases, you need a process such as ITIL to manage the complex changes often
associated with large monolithic systems.

If you are working in a large organization, the likelihood that you are working with such
large monolithic legacy systems is very high.

In any case, many of the practices described in ITIL translate directly into corresponding
DevOps practices. ITIL prescribes a configuration management system and a configuration
management database. These types of systems are also integral to DevOps, and several of
them will be described in this book.

Summary
This chapter presented a brief overview of the background of the DevOps movement. We
discussed the history of DevOps and its roots in development and operations, as well as in
the Agile movement. We also took a look at how ITIL and DevOps might coexist in larger
organizations. The cargo cult anti-pattern was explored, and we discussed how to avoid it.
You should now be able to answer the question of where DevOps fits into a larger Agile
context and the different cycles of Agile development.

We will gradually move towards more technical and hands-on subjects. The next chapter
will present an overview of what the technical systems w e tend to focus on in DevOps look
like.

[ 13]