DevOps

Culture, Automation, Measure and Sharing

It’s tough out there …
Software delivery challenges
Enter Text
Agile Development
DevOps
Enter Text
 What is DevOps?
DevOps is a culture, movement or practice that emphasizes the collaboration and
communication of both software developers and other IT professionals while enabling
organizations to deliver applications or changes of all types – including new
features, config changes, infra changes, bug fixes and experiments - more quickly into
production to better connect with customers while simultaneously reducing both
cost and risk.

DevOps is an understood set of practices and cultural values that has been proven to
help organizations of all sizes improve their software release cycles, software quality,
security, and ability to get rapid feedback on product development.
 DevOps explained
• The old view of operations
• “Dev” side being the “makers” and
• “Ops” side being the “people that deal with the creation after its birth”
• The realization of the harm that has been done in the industry of those two being treated as siloed concerns
is the core driver behind DevOps.
• DevOps can be interpreted as an outgrowth of Agile
• Agile software development prescribes close collaboration of customers, product management, developers,
and (sometimes) QA to fill in the gaps and rapidly iterate towards a better product.
• DevOps says “yes, but service delivery and how the app and systems interact are a fundamental part of the
value proposition to the client as well, and so the product team needs to include those concerns as a top level
item.”
• DevOps at the conceptual level is simply extending Agile principles beyond the boundaries of
“the code” to include systems and operations.
 Origins of DevOps
• Took place around 2009 during the convergence of numerous adjacent and
mutually reinforcing movements:
• 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr presentation.
• The “Infrastructure as code" movement (Mark Burgess and Luke Kanies)
• The "Agile infrastructure" movement (Andrew Shafer)
• The Agile system administration movement (Patrick DeBois)
• The Lean Startup movement (Eric Ries).
• The continuous integration and release movement (Jez Humble).
• The widespread availability of cloud and platform as a service (PaaS)
Principles behind DevOps
 The Three Ways
• Flow
• Feedback
• Continuous Experimentation and Learning
 The First Way : Flow
• Creating the left to right business value flow (from business to customer).
• Emphasizes the performance of the entire system.
• The focus is on all business value streams that are enabled by IT.
• It begins when requirements are identified (e.g., by the business or IT), are built in Development, and then
transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.
• The outcomes of putting the First Way into practice include
• Never passing a known defect to downstream work centers,
• Never allowing local optimization to create global degradation
• Always seeking to increase flow, and
• Always seeking to achieve profound understanding of the system
 The Second Way : Feedback
• Creating the right to left feedback loops.
• Shorten and amplify feedback loops so necessary corrections can be
continually made.
• The outcomes of the Second Way include
• Understanding and responding to all customers, internal and external,
• Shortening and amplifying all feedback loops, and
• Embedding knowledge where we need it.
 The Third Way : Continuous Experimentation

• Creating a culture that fosters two things:

• Continual experimentation, taking risks and learning from failure; and
• Understanding that repetition and practice is the prerequisite to mastery.
• The outcomes of the Third Way include
• Allocating time for the improvement of daily work,
• Creating rituals that reward the team for taking risks, and
• Fault injection
DevOps

1 Plan 4 Monitor + Learn

Enter Text
Development Production

2 Develop + Test 3 Release

 Plan
It starts with an idea – and a plan
how to turn this idea into reality … Project starts

Manage work

Enter Text
Develop + Test 1

Track progress Plan

 Develop + Test
Once the iteration starts, developers
turn great ideas into features … 2

Write Code

Enter Text
Unit Testing

Version Control

Build

Build Verification

Release
 Release
When all tests pass, the build is deployed to testing
environments for each stage in the release process

Cloud
Load Testing

Enter Text Integration testing

environment
Staging
environment

3 Monitor + Learn

Automated functional Pre-production

testing environment environment
 Monitor + Learn
Learn and understand how users use your app, how it reacts
and quickly fix issues and bugs
Plan the next iteration

Feedback

Enter Text
Monitor

4
 List of DevOps Practices
• Infrastructure as Code (IaC) • Availability Monitoring
• Capacity Management
• Continuous Integration • Change/Configuration Management
• Automated Testing • Feature Flags
• Automated Environment De-Provisioning
• Continuous Deployment • Self Service Environments
• Release Management • Automated Recovery (Rollback & Roll-Forward)

• App Performance Monitoring • Hypothesis Driven Development

• Testing in Production/AB testing

• Load Testing & Auto-Scale • Fault Injection/Monkey testing

• Usage Monitoring / User Telemetry

http://www.itproguy.com/devops-practices/
 Benefits
• Faster time to market
• Reduced IT waste
• Better quality product
• Competitive advantage
• Higher customer satisfaction
• Reduced cost of development etc.
 Faster time to market
• High performers are doing thousands of deployments per day.
• Amazon has gone on record that they are routinely performing over 23,000 deploys per day while
preserving –
• Reliability
• Stability
• Security
• High deployment rate = Business value
• How quickly the organization can go from idea to delivering value to customer
• How many experiments the organization can perform simultaneously
• High deployment rate = Rapid and constant experimentation
 Reduced IT waste
• Enormous waste in IT value stream
• Long lead times
• Poor hand-offs
• Unplanned work
• Rework
Deploy frequency
 Case Study : HP Laserjet Firmware
• 400 people distributed across 3 continents
• 10m+ lines of code
• 2 Software releases per year
• Majority of time spent on porting code to new product

Source: A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware (Agile Software Development Series)
 Problems
• 5% percent of dev time spent writing new features
• 10 to 15% of development time spent integrating code into
mainline
• 6 weeks to learn whether the checked in code worked
• One week to determine whether it integrated successfully
• Another day to reach the "integration build"
• Another day was spent running acceptance tests
• 6 weeks to complete full integration testing
Source: A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware (Agile Software Development Series)
 Improvements
• Architecture changes: Common code base.
Eliminated separate branches for all products,
putting all LaserJet models into trunk.
• Automated testing: Automated unit,
acceptance, and integration tests, which would
continually run against trunk.
• Stopping the line - All work stopped when
someone broke the build or the test suite.
Source: A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware (Agile Software Development Series)
 Results
• Overall development costs reduced
by ~40%
• Programs under development
increased by ~140%
• Development costs per program
reduced by 78%
• Resources driving innovation
increased by 5x
Source: A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware (Agile Software Development Series)
Thanks to practices such as Continuous integration, Continuous Delivery and
DevOps, high performers are often now doing thousands of deployments daily.

Gene Kim, Author of “The DevOps Handbook”

“You will probably write more code for test automation than you will code for
running your product. And that is not bad.”
Gary Gruver, president of Practical Large Scale Agile
“If there’s anything that all horses(enterprise IT orgs) hate, it’s hearing stories
about unicorns(DevOps shops like Amazon, Google, FB, Etsy etc), which is
strange, because horses and unicorns are probably the same species. Unicorns
are just horses with horns.”
DevOps using VSTS/TFS
2) Code Repository 3) Build 4) Test 5. Deploy to
Cloud/Onpremise

Contoso App

1. Developers
6. Monitor and Improve

Azure
Two reasons that motivate people to ‘do’ anything:
•they're trying to make something better or to gain something (the carrot)
•they’re trying to avoid pain (the stick)

The Carrot: What can DevOps make better?

1. Increase performance of your teams

Based on statistics from the 2017 State of DevOps report (authored by Jez Humble, Gene Kim and Nicole Forsgren), we know that teams practicing DevOps experience:
•24x faster recovery from failures
•3x lower change failure rate
•22% less time spent on unplanned work and rework
•50% less time remediating security issues.

2. Become attractive workplaces to people you’d like to hire

The State of Devops report also points out that employees in high-performing teams were 2.2 times more likely to recommend their organizations as a great place to work.
Note: If you are interested in the science behind the report, you can check out the video Sciencing the Crap Out of DevOps and the SSRN paper on 'The Role of Continuous Delivery in
IT and Organizational Performance'.
The Stick: What if you don’t “do DevOps”
If the benefits alone are not enough to convince your audience, then try pointing out the threats of not practicing DevOps.
1. Get overtaken by the competition
Survival is optional. No one has to change.
I like to use this quote to describe the current situation. If you don’t do DevOps, that’s fine. But in all probability, your competitors will do it, get faster, and beat you.
2. Slow recovery from inevitable security issues
Heartbleed is a great example of this. According to their survey, when this SSL issue came out, the average recovery was around 10 days. There were a lot of sites that were fixed in 20
minutes because they had solid continuous delivery pipelines and were able to run through their entire process to get to product very quickly. Symantec did a survey on vulnerabilities by
year, and the number of vulnerabilities grows every year.
3. Pay the high cost of human errors
Manual deployments is a game of russian roulette. In 2012, trading firm Knight Capital, used a manual process to deploy their software. They had eight servers to deploy to, but on one
occasion, the technician did not copy the new code to one of the eight. This company, with nearly $400 million in assets, went bankrupt in 45 minutes because of a failed deployment.
More recently, an employee at Amazon typed in a command and unintentionally brought down a whole lot of servers and a huge part of the Internet with it. It still took them (Amazon!)
four and a half hours to recover.
Here are some numbers about the cost of downtime which you can use to convince your business stakeholder about DevOps. According to Business Insider, during AWS' four-hour
disruption downtime, S&P 500 companies lost 150 million, US financial services lost 160 million, but Apple, Walmart, Newegg, Best Buy, Costco, and surprisingly Amazon/Zappos were
not affected by the outage. This is because they had processes in place and were able to switch data centers.
Common misconceptions
Before we get to the strategies to implement change, I want to highlight what I see are the most common misconceptions about implementing new ideas.
These are highly opinionated, but I will explain why I believe so.
Ideas are judged 100% on merit
Those of us that have been in technology for a while know that there are a lot of successful products in the market that might not have been the best idea.
Ideas actually aren't judged 100% on merit. If they were, betamax would've beat VHS. Good ideas need to go together with implementation to create a
lasting impact. You can not just do a session to introduce it. Implementing change means that there's work to be done and it’s an ongoing process.
Top down approach is effective for changes
It’s very common for companies to attempt top-down culture change. A quick web search would turn up many examples of organizations who “went agile”
and failed. Quite often a deeper dive reveals that they never actually changed behavior.
I saw a group several years ago who decided to switch from waterfall to agile methods. They went through fairly extensive training in things like user stories.
For those that aren’t familiar, most successful agile organizations use stories as “conversation starters”, not detailed requirements.
At least one group in this organization took their massive waterfall requirements document, split it on paragraph markers and then proudly presented a
spreadsheet with hundreds of smaller pieces of text they called user stories.
They got compliance, they created user stories. There was no commitment to real change.
The lesson I learned was that top-down change seldom works. You can’t just say, "Okay. CEO, CIO, CFO, CTO said we're gonna do the DevOps", and
expect results. . Pauly Comtois from Hearst Media did a great talk how middle managers are instrumental in propagating change.
Patterns for gaining acceptance
As an engineer, I'm a fan of patterns - things that are repeatable. I am going to introduce you to some patterns from Linda Rising and Mary Lynn Manns’
book Fearless Change, which I personally, found very useful.
1. Become an evangelist
A technology evangelist is a person who builds a critical mass of support for a given technology and then establishes it as a technical standard. If you want
to drive a DevOps cultural change, you need to go around to teams and get support from different groups of people. It’s an ongoing process, which means
that you are never done.
2. Tailor your message
It’s important that you communicate differently depending on who you are talking to. For example, It’s very unlikely that your Chief Financial Officer cares
how many times a day you can deploy. It’s very likely that they do care if you can quickly address issues which are blocking your customers from buying
your products and services.
3. Lunch and learn
Make lunch and learns effective by being creative about the food & the format, making it interactive, having clear takeaways and promote the session. This
is another place where you should tailor the message. “Come learn how we’re working to respond faster to security risks” is better than “check out our new
deployment tool”.
4. Leave hints
• “If Agile software development was the opening act to a great performance,
Continuous Delivery is the headliner.”
Forrester Research 2013
• Jez Humble explained that DevOps is “a cross-disciplinary community of practice dedicated to the study of building,
evolving and operating rapidly-changing resilient systems at scale.”

• Automate environment management.

• Integrate continuously.
• Automate deployments.
• Automate testing with APIs.
• Make data-driven release decisions.
• Reduce size of releases.
• Eliminate handoffs and wait time.
• Drive better results with feedback.
 Continuous Delivery
• Is a journey
• Isn’t just a technical shift, its a cultural one.

• The ability to create new, high-quality software applications and bring them to
market more quickly is the “X factor” that defines industry leaders.

• Remember it is not the strongest of the species that survive, nor the most
intelligent, but those most responsive to change. (Charles Darwin)
Get started with Continuous
Delivery
• Start small(It is like a marathon - Go slow, follow a routine and be
consistent)
• Get buy-in from devs and ops
• Implement Continuous Integration
• Reinforce best practices, and automate the deploy-to-production pipeline
• Adopt automation across the delivery process
• Build on your successes
 Deployment pipeline
• Assembly and delivery line
 Case Study : Microsoft Dev Division
• Over seven years, Microsoft Developer Division (DevDiv) embraced Agile.
• Achieved a 15x reduction in technical debt through solid engineering practices
• Trained everyone on Scrum, multidisciplinary teams and product ownership across
the division.
• Significantly focused on the flow of value to our customers.
• By the time they shipped VS2010, the product line achieved a level of customer
recognition that was unparalleled.
• DevOps Values – Fundamental DevOps values are effectively captured in the Agile Manifesto – with
perhaps one slight emendation to focus on the overall service or software fully delivered to the
customer instead of simply “working software.”
• DevOps Practices –Specific techniques used as part of implementing the above concepts and
processes. Continuous integration and continuous deployment, “Give your developers a pager and
put them on call,” using configuration management, metrics and monitoring schemes, a toolchain
approach to tooling… Even using virtualization and cloud computing is a common practice used to
accelerate change in the modern infrastructure world.
• DevOps Tools – Tools you’d use in the commission of these principles. In the DevOps world there’s
been an explosion of tools in release (jenkins, travis, teamcity), configuration management (puppet,
chef, ansible, cfengine), orchestration (zookeeper, noah, mesos), monitoring, virtualization and
containerization (AWS, OpenStack, vagrant, docker) and many more. While, as with Agile, it’s
incorrect to say a tool is “a DevOps tool” in the sense that it will magically bring you DevOps, there
are certainly specific tools being developed with the express goal of facilitating the above principles,
methods, and practices, and a holistic understanding of DevOps should incorporate this layer.
 DevOps
• Part of a DevOps culture is learning from usage.
• A tacit assumption of Agile was that the Product Owner was omniscient and could groom the
backlog correctly.
• In contrast, when you run a high-reliability service, you can observe how customers are actually
using its capabilities in near real-time.
• Release frequently
• Experiment with improvements, measure, and ask customers how they perceive the changes.
• The data you collect becomes the basis for the next set of improvements you do.
• In this way, a DevOps product backlog is really a set of hypotheses that become experiments in
the running software and allow a cycle of continuous feedback.
“The good thing about cloud and mobile is you can develop very quickly. You
can scale fast if it works or just try something else if it doesn’t.”
Pascal Eymery, VP of strategy and businessdevelopment customer support at Airbus
As you prepare to build your business case, identify the area or areas of greatest
potential impact for your company, and go from there.
Ido Ben Moshe, VP at Zend
“The most valuable thing you do as a software developer is make that
connection between creation and use.”
Chris Hilton, Lead consultant at ThoughtWorks
“We’re in sales. Our job is to sell DevOps inside the organization and unless I
have the reporting and the data, I’ll get questions every week from CIOs that I
can’t answer.”

Chris Nowak, senior vice president of DevOps for Bank of America