#145

Get More Refcardz! Visit refcardz.com

CONTENTS INCLUDE: The Deployment Pipeline Benets Conguration Management Continuous Integration Testing Hot Tips and more...
n n n n n n

brought to you by...

Continuous Delivery
Patterns and Antipatterns in the Software Lifecycle
By Paul M. Duvall
F  eedback: Team members learn of problems as soon as they occur so that issues are xed as soon as possible. C  ontinually Deploy: Through a fully automated process, you can deploy and release any version of the software to any environment. (1) In the Deployment Pipeline diagram above, all of the patterns are shown in context. There are some patterns that span multiple stages of the pipeline, so I chose the stage where its most predominately used.
BENEFITS

ABOUT CONTINUOUS DELIVERY

With Continuous Delivery (CD), teams continuously deliver new versions of software to production by decreasing the cycle time between an idea and usable software through the automation of the entire delivery system: build, deployment, test, and release. CD is enabled through the Deployment Pipeline, which encompasses a collection of patterns described in this Refcard. CD is concerned with how all the moving parts t together: conguration management, automated testing, continuous integration and deployment, data management, environment management, and release management. (1)
THE DEPLOYMENT PIPELINE

E  mpowering Teams: Because the deployment pipeline is a pull system, testers, developers, operations, and others can self service the application version into an environment of their choice. R  educing Errors: Ensuring the correct version, conguration, database schema, etc. are applied the same way every time through automation. L  owering Stress: Through push-button releases to production and Rehearsing Deployments, a release becomes commonplace without the typical stress. D  eployment Flexibility: Instantiate a new environment or conguration by making a few changes to the automated delivery system.  Practice makes Perfect: Through the deployment pipeline, the nal deployment into production is being rehearsed every single time the software is deployed to any target environments. (1)

Continuous Delivery

The purpose of the deployment pipeline is threefold: V  isibility: All aspects of the delivery system - building, deploying, testing, and releasing are visible to all team members promoting collaboration.

DZone, Inc.

www.dzone.com

Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

CONFIGURATION MANAGEMENT

Short-Lived Branches (1)

Pattern Anti-patterns Branches must be short lived ideally less than a few days and never more than an iteration. Branches that last more than an iteration. Branches by product feature that live past a release.

Conguration Management is the process by which all artifacts relevant to your project, and the relationships between them, are stored, retrieved, uniquely identied, and modied. (1)
Note: Each pattern is cited with a number in parentheses that corresponds to the source in the References section.

Single Command Environment (1)

Pattern Check out the projects version-control repository and run a single command to build and deploy the application to any accessible environment, including the local development. Forcing the developer to dene and congure environment variables. Making the developer install numerous tools in order for the build/deployment to work.

Congurable Third-Party Software (1)

Pattern Anti-patterns

Evaluate and use third-party software that can be easily congured, deployed, and automated. Procuring software that cannot be externally congured. Software without an API or commandline interface that forces teams to use the GUI only.

Anti-patterns

Conguration Catalog (1)

Pattern Maintain a catalog of all options for each application, how to change these options and storage locations for each application. Automatically create this catalog as part of the build process.

Single Path to Production (1)

Pattern Conguration management of the entire system source, conguration, environment and data. Any change can be tied back to a single revision in the version-control system. Parts of system are not versioned. Inability to get back to a previously congured software system.

Anti-patterns

Anti-patterns

Conguration options are not documented. The catalog of applications and other assets is tribal knowledge.

CONTINUOUS INTEGRATION (CI)

Mainline (3)
Pattern Anti-patterns Minimize merging and keep the number of active code lines manageable by developing on a mainline.

Build Threshold (5)

Pattern

Multiple branches per project.

Anti-patterns

Fail a build when a project rule is violated such as architectural breaches, slow tests, and coding standard violations. Manual code reviews. Learning of code quality issues later in the development cycle.

Merge Daily (1)

Pattern Anti-patterns Changes committed to the mainline are applied to each branch on at least a daily basis.

Merging every iteration once a week or less often than once a day.

Commit Often (6)

Pattern Each team member checks in regularly to trunk - at least once a day but preferably after each task to trigger the CI system. Source les are committed less frequently than daily due to the number of changes from the developer.

Protected Conguration (5) ,(1)

Pattern Store conguration information in secure remotely accessible locations such as a database, directory, or registry. Open text passwords and/or single machine or share.

Anti-patterns

Continuous Feedback (6)

Pattern Anti-patterns

Anti-patterns

Send automated feedback from CI system to all Cross-Functional Team members.

Notications are not sent; notications are ignored; CI system spams everyone with information they cannot use.

Repository (3) , (1)

Pattern All source les - executable code, conguration, host environment, and data - are committed to a versioncontrol repository. Pattern Anti-patterns

Continuous Integration (6)

Building and testing software with every change committed to a projects version control repository.

Anti-patterns

Some les are checked in, others, such as environment conguration or data changes, are not. Binaries that can be recreated through the build and deployment process are checked in.

Scheduled builds, nightly builds, building periodically, building exclusively on developers machines, not building at all.

DZone, Inc.

www.dzone.com

Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Stop the Line (5) , (1) , (4), (12)

Pattern

Parallel Tests (1)

Pattern Anti-patterns Run multiple tests in parallel across hardware instances to decrease the time in running tests.

Fix software delivery errors as soon as they occur; stop the line. No one checks in on a broken build as the x becomes the highest priority. Builds stay broken for long periods of time, thus preventing developers from checking out functioning code.

Anti-patterns

Running tests on one machine or instance. Running dependent tests that cannot be run in parallel.

Stub Systems (1)

Pattern

Independent Build (6)

Pattern Write build scripts that are decoupled from IDEs. These build scripts are executed by a CI system so that software is built at every change. Anti-patterns

Use stubs to simulate external systems to reduce deployment complexity. Manually installing and conguring interdependent systems for Commit Stage build and deployment.

Anti-patterns

Automated build relies on IDE settings. Builds are unable to be run from the command line.

DEPLOYMENT PIPELINE

Visible Dashboards
Pattern Provide large visible displays that aggregate information from your delivery system to provide highquality feedback to the Cross-Functional Team in real time.

Deployment Pipeline (1)

Pattern

A deployment pipeline is an automated implementation of your applications build, deploy, test, and release process. Deployments require human intervention (other than approval or clicking a button). Deployments are not production ready.

Anti-patterns

Anti-patterns

Email-only alerts or not publicizing the feedback to the entire team.

Pattern

Value-Stream Map (4)

TESTING
Create a map illustrating the process from check in to the version-control system to the software release to identify process bottlenecks.

Automate Tests
Pattern Automate the verication and validation of software to include unit, component, capacity, functional, and deployment tests Manual testing of units, components, deployment, and other types of tests.

Anti-patterns

Separately dened processes and views of the checkin to release process.

Anti-patterns

BUILD AND DEPLOYMENT SCRIPTING

Dependency Management (5)

Unit- Automating tests without any dependencies. Component- Automating tests with dependencies to other components and heavyweight dependencies such as the database or le system. Deployment- Automating tests to verify the deployment and conguration were successful. Sometimes referred to as a smoke tests. Functional- Automating tests to verify the behavior of the software from a users perspective. Capacity- Automating load and performance testing in nearproduction conditions.
Anti-patterns Pattern

Centralize all dependent libraries to reduce bloat, classpath problems, and repetition of the same dependent libraries and transitive dependencies from project to project. Multiple copies of the same binary dependencies in each and every project. Redening the same information for each project. Classpath hell!

Common Language (1)

Pattern

As a team, agree upon a common scripting language - such as Perl, Ruby, or Python - so that any team member can apply changes to the Single Delivery System. Each team uses a different language making it difcult for anyone to modify the delivery system reducing cross-functional team effectiveness.

Isolate Test Data (1)

Pattern

Anti-patterns

Use transactions for database-dependent tests (e.g., component tests) and roll back the transaction when done. Use a small subset of data to effectively test behavior. Using a copy of production data for Commit Stage tests. Running tests against a shared database.

Externalize Conguration (5)

Pattern Changes between environments are captured as conguration information. All variable values are externalized from the application conguration into build/deployment-time properties.

Anti-patterns

Anti-patterns

Hardcoding values inside the source code or per target environment.

DZone, Inc.

www.dzone.com

Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Fail Fast (6)

Pattern

Blue-Green Deployments (1)

Pattern

Fail the build as soon as possible. Design scripts so that processes that commonly fail run rst. These processes should be run as part of the Commit Stage.
Common build mistakes are not uncovered until late in the deployment process. Anti-patterns

Deploy software to a non-production environment (call it blue) while production continues to run. Once its deployed and warmed up, switch production (green) to non-production and blue to green simultaneously.
Production is taken down while the new release is applied to production instance(s).

Anti-patterns

Fast Builds (6)

Pattern The Commit Build provides feedback on common build problems as quickly as possible - usually in under 10 minutes.

Anti-patterns

Throwing everything into the commit stage process, such as running every type of automated static analysis tool or running load tests such that feedback is delayed.

Dark Launching (11)

Pattern Anti-patterns

Scripted Deployment (5)

Pattern

Launch a new application or features when it affects the least amount of users. Software is deployed regardless of number of active users.

All deployment processes are written in a script, checked in to the version-control system, and run as part of the Single Delivery System. Deployment documentation is used instead of automation. Manual deployments or partially manual deployments. Using GUI to perform a deployment.

Anti-patterns

Rollback Release (5)

Pattern Anti-patterns

Provide an automated single command rollback of changes after an unsuccessful deployment. Manually undoing changes applied in a recent deployment. Shutting down production instances while changes are undone.

Unied Deployment (5)

Pattern The same deployment script is used for each deployment. The Protected Conguration per environment - is variable but managed.

Self-Service Deployment (1)

Pattern

Anti-patterns

Different deployment script for each target environment or even for a specic machine. Manual conguration after deployment for each target environment.

Any Cross-Functional Team member selects the version and environment to deploy the latest working software.
Deployments released to team are at specied intervals by the Build Team. Testing can only be performed in a shared state without isolation from others.

Anti-patterns

DEPLOYING AND RELEASING APPLICATIONS

INFRASTRUCTURE AND ENVIRONMENTS

Binary Integrity (5)

Pattern

Build your binaries once, while deploying the binaries to multiple target environments, as necessary. Software is built in every stage of the deployment pipeline.

Automate Provisioning (1)

Pattern

Anti-patterns

Automate the process of conguring your environment to include networks, external services, and infrastructure.
Congured instances are works of art requiring team members to perform partially or fully manual steps to provision them.

Anti-patterns

Canary Release
Pattern

Release software to production for a small subset of users (e.g. , 10%) to get feedback prior to a complete rollout. Software is released to all users at once.

Behavior-Driven Monitoring (1)

Pattern

Anti-patterns

Automate tests to verify the behavior of the infrastructure. Continually run these tests to provide near real-time alerting. No real-time alerting or monitoring. System conguration is written without tests.

Anti-patterns

DZone, Inc.

www.dzone.com

Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Immune System (9)

Pattern Deploy software one instance at a time while conducting Behavior-Driven Monitoring. If an error is detected during the incremental deployment, a Rollback Release is initiated to revert changes.

Scripted Database (7)

Pattern Anti-patterns

Script all database actions as part of the build process. Using data export/import to apply data changes. Manually applying schema and data changes to the database.

Anti-patterns

Non-incremental deployments without monitoring.

Lockdown Environments (1)

Pattern Lock down shared environments from unauthorized external and internal usage, including operations staff. All changes are versioned and applied through automation. The Wild West: any authorized user can access shared environments and apply manual conguration changes, putting the environment in an unknown state leading to deployment errors.

INCREMENTAL DEVELOPMENT

Branch by Abstraction (2)

Pattern Instead of using version-control branches, create an abstraction layer that handles both an old and new implementation. Remove the old implementation. Branching using the version-control system leading to branch proliferation and difcult merging. Feature branching.

Anti-patterns

Anti-patterns

Production-Like Environments (1)

Pattern Anti-patterns

Target environments are as similar to production as possible.

Environments are production like only weeks or days before a release. Environments are manually congured and controlled.

Toggle Features (10)

Pattern Anti-patterns Deploy new features or services to production but limit access dynamically for testing purposes.

Waiting until a feature is fully complete before committing the source code.

Transient Environments
Pattern

COLLABORATION

Utilizing the Automate Provisioning, Scripted Deployment and Scripted Database patterns, any environment should be capable of terminating and launching at will.
Environments are xed to DEV, QA or other predetermined environments.

Delivery Retrospective (1)

Pattern

Anti-patterns

For each iteration, hold a retrospective meeting where everybody on the Cross-Functional Team discusses how to improve the delivery process for the next iteration.
Waiting until an error occurs during a deployment for Dev and Ops to collaborate. Having Dev and Ops work separately.

Anti-patterns

DATA

Database Sandbox (7)

Pattern Create a lightweight version of your database using the Isolate Test Data pattern. Each developer uses this lightweight DML to populate his local database sandboxes to expedite test execution.

Cross-Functional Teams (1)

Pattern

Everybody is responsible for the delivery process. Any person on the Cross-Functional Team can modify any part of the delivery system. Siloed teams: Development, Testing, and Operations have their own scripts and processes and are not part of the same team.

Anti-patterns

Shared database. Developers and testers are unable to make data changes without it potentially adversely affecting other team members immediately.

Anti-patterns

Decouple Database (1)

Pattern Ensure your application is backward and forward compatible with your database so you can deploy each independently Application code data are not capable of being deployed separately.

Amazon.com has an interesting take on this approach. They call it You build it, you run it. Developers take the software theyve written all the way to production.

Root-Cause Analysis (1)

Pattern

Anti-patterns

Learn the root cause of a delivery problem by asking why of each answer and symptom until discovering the root cause. Accepting the symptom as the root cause of the problem.

Anti-patterns

Database Upgrade (7)

Pattern Anti-patterns

Use scripts to apply incremental changes in each target environment to a database schema and data. Manually applying database and data changes in each target environment.

TOOLS

This is meant to be an illustrative list, not an exhaustive list, to give you an idea of the types of tools and some of the vendors that help to enable effective Continuous Delivery. The Java, .NET and Ruby platforms are represented. The tools that span categories have been
|

DZone, Inc.

www.dzone.com

Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

assigned to the most appropriate category or duplicated when necessary.

Category
Conguration Management Continuous Integration

Components and Dependencies Collaboration

Ivy, Archiva, Nexus, Artifactory, Bundler Mingle, Greenhopper, JIRA

Example Software Tools

Subversion (SVN), git, Perforce, PassPack, PasswordSafe, ESCAPE, CongGen Bamboo, Jenkins, AntHill Pro, Go, TeamCity, TFS 2010, Electric Commander. Supporting tools: , Doxygen, Grand, GraphViz, JavaDoc, NDoc, SchemaSpy, UmlGraph, CheckStyle, Clover, Cobertura, FindBugs, FxCop, JavaNCSS, JDepend, PMD, Sonar, Simian Twist , AntUnit, Cucumber, DbUnit, webrat, easyb, Fitnesse, JMeter, JUnit, NBehave, SoapUI, Selenium, RSpec, SauceLabs Go, AntHill Pro Ant, AntContrib, NAnt, MSBuild, Buildr, Gant, Gradle, make, Maven, Rake, Java Secure Channel, ControlTier, Altiris, Capistrano, Fabric, Func AWS EC2, AWS S3, Windows Azure, Google App Engine, AWS Elastic Beanstalk, Heroku, Capistrano, Cobbler, BMC Bladelogic, CFEngine, IBM Tivoli Provisioning Manager, Puppet, Chef, Bcfg2, AWS Cloud Formation, Windows Azure AppFabric, rPath, JEOS, BoxGrinder, CLIP, Eucalyptus, AppLogic, CloudKick, CloudWatch, Nagios, Zabbix, Zenoss Hibernate, MySQL, Liquibase, Oracle, PostgreSQL, SQL Server, SimpleDB, SQL Azure, Ant, MongoDB, dbdeploy

REFERENCES
1. Jez Humble and David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, Addison Wesley Professional, 2010 2. Paul Hammant and www.continuousdelivery.com 3. Stephen P. Berczuk and Brad Appleton, Software Conguration Management Patterns., Addison Wesley Professional, 2003 4. Mary and Tom Poppendieck, Leading Lean Software Development, Addison Wesley, 2009 5. Paul M. Duvall, Continuous integration. Patterns and Antipatterns, DZone refcard #84, 2010 http://bit.ly/l8rfVS 6. Paul M. Duvall, Continuous integration. Improving Software Quality and Reducing Risk, Addison Wesley, 2007 7. Scott W. Ambler and Pramodkumar J. Saladage, Refactoring Databases. Evolutionary Database Design, Addison Wesley, 2006. 8. Paul M. Duvall, IBM developerWorks series Automation for the people http://ibm.co/iwwvPX 9. IMVU: http://bit.ly/jhqP5f 10. Martin Fowler and Facebook: http://on.fb.me/miBrOM 11. Facebook Engineering: http://on.fb.me/miBrOM 12. Paul Julius, Enterprise Continuous Integration Maturity Model, http://bit.ly/m7h5vC

Testing

Deployment Pipeline Build and Deployment Scripting Infrastructure and Environments

Data

ABOUT THE AUTHOR

Paul M. Duvall is the CTO of Stelligent, (http://stelligent. com/) a company that helps organizations with Continuous Integration and Delivery, Cloud Migrations and Elastic Operations. Paul is the author of many books including the Jolt-award winning book Continuous Integration: Improving Software Quality and Reducing Risk (http://amzn.to/cibook) and the recently published Startup@Cloud: 33 tips for running your company using cloud-based software (http://amzn.to/startupatcloud). He is passionate about the cloud and automation and blogs at http://devopscloud.com/.

RECOMMENDED BOOK

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation The book has two themes: automation and collaboration. Delivering software of any complexity involves people with a bunch of skills: testers, developers, and sysadmin / operations personnel. The last group of people often gets left out of the process of delivering software until the end, and even testers are often not as heavily involved in the development process as they should be.

DZone, Inc. 140 Preston Executive Dr. Suite 100 Cary, NC 27513 DZone communities deliver over 6 million pages each month to more than 3.3 million software developers, architects and decision makers. DZone offers something for everyone, including news, tutorials, cheat sheets, blogs, feature articles, source code and more. DZone is a developers dream, says PC Magazine.
Copyright 2011 DZone, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

888.678.0399 919.678.0300 $7.95 Refcardz Feedback Welcome [email protected] Sponsorship Opportunities [email protected]

Version 1.0