DevOps on AWS

Deep Dive on Continuous Delivery

and the AWS Developer Tools
Matthew Trescot, Manager, Solutions Architecture

July 2016

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why are we
here today?
© Mike Gifford
https://secure.flickr.com/photos/mgifford/4525333972
https://creativecommons.org/licenses/by-nc-sa/2.0/legalcode
Software moves faster today

Software creation and distribution is

easier and faster than ever
• Startups can now take on giants with little to
no funding ahead of time
• Getting your software into the hands of
millions is a download away
• Your ability to move fast is paramount to your
ability to fight off disruption
The software delivery model has drastically changed

Old software delivery model New software delivery model

What tools do you need to move fast?

Releasing software in this new software-driven world

requires a number of things
• Tools to manage the flow of your software development
release process
• Tools to properly test and inspect your code for defects
and potential issues
• Tools to deploy your applications
First, we need to understand
a little bit about software
release processes

© Steve Jurvetson
https://www.flickr.com/photos/jurvetson/5201796697/
https://creativecommons.org/licenses/by-nc-sa/2.0/legalcode
Release processes have four major phases

Source Build Test Production

• Check-in • Compile code • Integration • Deployment

source code • Unit tests tests with to production
such as .java • Style checkers other systems environments
files. • Code metrics • Load testing
• Peer review • Create • UI tests
new code container • Penetration
images testing
Release processes levels

Source Build Test Production

Continuous integration

Continuous delivery

Continuous deployment
Release Processes levels

Source Build Test Production

Our focus today
Continuous integration

Continuous delivery

Continuous deployment
 Continuous delivery benefits

Automate the software Improve developer Find and address Deliver updates faster
release process productivity bugs quickly
A look back at
development
at Amazon
© Craig Morey
https://secure.flickr.com/photos/pixelthing/15806918992/
https://creativecommons.org/licenses/by-nc-sa/2.0/legalcode
Development transformation at Amazon: 2001–2009

2001 2009

monolithic microservices + 2 pizza teams

application + teams
 Things went much
better under this
model and teams
were developing
features faster than
ever, but we felt that
we could still improve
In 2009, we
ran a study to
find out where
inefficiencies
might still exist
We were just waiting

Write Build Deploy Deploy

code Wait code Wait to test Wait to prod
We were just waiting.

Write Build Deploy Deploy

code Wait code Wait to test Wait to prod

Mins Days Mins Days Mins Days Mins

We were just waiting
Weeks

Write Build Deploy Deploy

code Wait code Wait to test Wait to prod

Mins Days Mins Days Mins Days Mins

We were just waiting
Weeks

Write Build Deploy Deploy

code Wait code Wait to test Wait to prod

Mins Days Mins Days Mins Days Mins

We built tools to
automate our software
release process
© Lindsey G
https://secure.flickr.com/photos/lindseygee/5894617854/
https://creativecommons.org/licenses/by/2.0/legalcode
 Automated actions and
transitions; from check-
Pipelines in to production

Development benefits
• Faster
• Safer
• Consistent and
standardized
• Visualization of the
process
This has continued to work out really well

In 2014
• Thousands of service teams across Amazon
• Building microservices
• Practicing continuous delivery
• Many environments (staging, beta, production)

50 million deployments
We continue to survey our
software developers every year
and in 2014 results found only
one development tool or service
could be correlated statistically
with happier developers:

Our pipelines service!

continuous delivery
==
happier developers!
© Barbara M.
https://www.flickr.com/photos/cannnela/4614340819/
https://creativecommons.org/licenses/by/2.0/legalcode
AWS CodePipeline
Continuous delivery service for fast and
reliable application updates

Model and visualize your software release

process

Builds, tests, and deploys your code every time

there is a code change

Integrates with third-party tools and AWS

 AWS CodePipeline benefits

Configurable workflow Easy to integrate Improved quality

Rapid delivery Get started fast

 CodePipeline
MyApplication
Source
Source
GitHub

Build
Stage
JenkinsOnEC2 Pipeline
Jenkins Action

Transition
Deploy
JavaApp
AWS Elastic Beanstalk
 CodePipeline
MyApplication
Source
Source
GitHub

Build
JenkinsOnEC2 NotifyDevelopers
Jenkins AWS Lambda

Parallel actions
Deploy
JavaApp
AWS Elastic Beanstalk
 CodePipeline
MyApplication
Source
Source
GitHub

Build
JenkinsOnEC2 NotifyDevelopers
Jenkins AWS Lambda
Sequential actions
TestAPI
Runscope

Deploy
JavaApp
AWS Elastic Beanstalk
 CodePipeline
MyApplication
Source 1. Get changes

Source
GitHub

3. Poll for job

Build 5. Get source artifact S3
4. Acknowledge job
JenkinsOnEC2 7. Put success Source artifact
Jenkins 6. Store build artifact
Amazon EC2 instance
S3
Build artifact
8. Retrieve build artifact
Deploy
JavaApp 9. Deploy build artifact Java app
AWS Elastic Beanstalk
Elastic Beanstalk
Web container
We have a strong partner list, and it’s growing
Source Build Test Deploy
AWS service integrations
Source Invoke logic Deploy

Amazon S3 AWS Lambda AWS CodeDeploy

AWS CodeCommit AWS Elastic Beanstalk
AWS OpsWorks
Building your application
development release
pipeline
© Seattle Municipal Archives
https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/
https://creativecommons.org/licenses/by/2.0/legalcode
DEMO!
Build and test your
application

© Spencer Cooper
https://secure.flickr.com/photos/spenceyc/7481166880
https://creativecommons.org/licenses/by/2.0/legalcode
Building your code
“Building” code typically refers to languages that
require compiled binaries
• .NET languages: C#, F#, VB.net, etc
• Java and JVM languages: Java, Scala,
JRuby
• Go
• iOS languages: Swift, Objective-C
We also refer to the process of creating Docker
container images as “building” the image EC2
No building required!

Many languages don’t require building; these

are considered interpreted languages
• PHP
• Ruby
• Python
• Node.js EC2

You can just deploy your code!

Testing your code

Testing is both a science and an art form!

Goals for testing your code
• Want to confirm desired functionality
• Catch programming syntax errors
• Standardize code patterns and format
• Reduce bugs due to undesired application
usage and logic failures
• Make applications more secure
Deploying your
applications
© Simon Q
https://secure.flickr.com/photos/simononly/15386966677
https://creativecommons.org/licenses/by/2.0/legalcode
AWS CodeDeploy
Automates code deployments to any instance

Handles the complexity of updating your

applications

Avoid downtime during application deployment

Deploy to Amazon EC2 or on-premises servers

in any language and on any operating system

Integrates with third-party tools and AWS

 appspec.yml Example
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html
permissions:
- object: /var/www/html
pattern: “*.html”
owner: root
group: root
mode: 755
hooks:
ApplicationStop:
- location: scripts/deregister_from_elb.sh
BeforeInstall:
- location: scripts/install_dependencies.sh
ApplicationStart:
- location: scripts/start_httpd.sh
ValidateService:
- location: scripts/test_site.sh
- location: scripts/register_with_elb.sh
 appspec.yml Example
version: 0.0
os: linux
files: • Send application files to one
- source: / directory and configuration
destination: /var/www/html
permissions:
files to another
- object: /var/www/html
pattern: “*.html” • Set specific permissions on
owner: root
group: root
specific directories and files
mode: 755
hooks:
ApplicationStop:
- location: scripts/deregister_from_elb.sh
• Remove/add instance to
BeforeInstall: Elastic Load Balancing
- location: scripts/install_dependencies.sh • Install dependency packages
ApplicationStart: • Start Apache
- location: scripts/start_httpd.sh
ValidateService: • Confirm successful deploy
- location: scripts/test_site.sh • More!
- location: scripts/register_with_elb.sh
Choose deployment speed and group
Dev Deployment group

one at a time
v2 v1 v1 v1 v1 v1 Agent Agent

OR
half at a time Prod Deployment group

v2 v2 v2 v1 v1 v1
Agent Agent Agent

all at once
v2 v2 v2 v2 v2 v2 Agent Agent Agent
Launching to
production

https://www.flickr.com/photos/spacex/16510243060/
Launching to production
After you’ve built and tested your code and
hopefully gone through a few preproduction
deploys, its time for the real thing!
You’ll want think about the following
• Impact to customers
• Impact to infrastructure
• Impact to business

How can we track these and communicate

deployments?
Extend AWS CodePipeline using custom actions

Mobile testing Update tickets Provision resources

Update dashboards Send notifications Security scan

 CodePipeline With custom actions,
MyApplication the job worker drives the interaction
Source
Source
between AWS CodePipeline
GitHub and other applications or services

Build
JenkinsOnEC2
Jenkins

Deploy
1. Poll for job Amazon
EC2
JavaApp
Action 2. Acknowledge job 3. Perform job
CustomBeanstalk
Elastic action
4. Put success
Job worker

JavaApp
Elastic Beanstalk
 CodePipeline With AWS Lambda-based actions,
MyApplication AWS CodePipeline
Source
Source
drives the integration with Lambda,
GitHub which then connects with other
applications or services
Build
JenkinsOnEC2
Jenkins

Deploy
1. Invoke Lambda function
MyAction
2. Perform job
AWS Lambda 3. PutJobSuccessResult w/
continuation token
AWS
JavaApp 4. Invoke Lambda function w/ Lambda
Elastic Beanstalk continuation token
5. PutJobSuccessResult #3 and #4 repeat until no continuation
token is sent, signaling the action has
been completed (#5)
What extension method should I use?

Lambda Custom action

Short-running tasks are easy to build Can perform any type of workload

Long-running tasks need more work Control over links displayed in console

Node.js, Python, and Java support Any language support

Runs on AWS Can run on premises
No servers to provision or manage Requires compute resources
Manual approvals – new!
You can add a manual approval at the point where you want the
pipeline to stop running until someone approves or rejects the
revision in progress

• Pipeline stops executing when it has reached the point at which you
set the approval action

• Pipeline execution resumes only when the action has been approved

• Approval action managed with AWS Identity and Access

Management (IAM) permissions

• Notify approvers in several ways including email, SMS, webhooks,

and more
FIN, ACK
We’ve seen a quick run through today of the benefits of
continuous delivery on our software release process
• Continuous integration (build/test) helps shrink our
feedback loop greatly
• We can get our software out in front of our users much
more rapidly
• By moving faster we can actually ensure better quality
• CodePipeline allows for integration with almost any
service or tool you can think of!
• Plus visualization of what’s going on!
Try it out today
Test out CodePipeline and spin up a full continuous
delivery pipeline using the starter kit

bit.ly/AWSCodeStarterKit
 But wait, there’s more!
Resources to learn more
• Continuous integration: https://aws.amazon.com/devops/continuous-integration/
• Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/

• CodePipeline
• https://aws.amazon.com/codepipeline/
• https://aws.amazon.com/documentation/codepipeline/
• CodeDeploy
• https://aws.amazon.com/codedeploy/
• https://aws.amazon.com/documentation/codedeploy/
• https://github.com/awslabs/aws-codedeploy-samples
• Code services starter kit: http://bit.ly/AWSCodeStarterKit
Thank You!