NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.

com
AWS CloudFormation
• CloudFormation is a declarative way of outlining your AWS
Infrastructure, for any resources (most of them are supported)
• For example, within a CloudFormation template, you say:
• I want a security group
• I want two EC2 instances using this security group
• I want two Elastic IPs for these EC2 instances
• I want an S3 bucket
• I want a load balancer (ELB) in front of these EC2 instances

• Then CloudFormation creates those for you, in the right order, with the
exact configuration that you specify

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Template Example
Infrastructure Composer

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Benefits of AWS CloudFormation (1/2)
• Infrastructure as code
• No resources are manually created, which is excellent for control
• The code can be version controlled for example using Git
• Changes to the infrastructure are reviewed through code

• Cost
• Each resources within the stack is tagged with an identifier so you can easily see how
much a stack costs you
• You can estimate the costs of your resources using the CloudFormation template
• Savings strategy: In Dev, you could automation deletion of templates at 5 PM and
recreated at 8 AM, safely

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Benefits of AWS CloudFormation (2/2)
• Productivity
• Ability to destroy and re-create an infrastructure on the cloud on the fly
• Automated generation of Diagram for your templates!
• Declarative programming (no need to figure out ordering and orchestration)

• Separation of concern: create many stacks for many apps, and many layers. Ex:
• VPC stacks
• Network stacks
• App stacks

• Don’t re-invent the wheel

• Leverage existing templates on the web!
• Leverage the documentation

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
How CloudFormation Works
• Templates must be uploaded in S3 and then referenced in
CloudFormation
• To update a template, we can’t edit previous ones. We have to re-
upload a new version of the template to AWS
• Stacks are identified by a name
• Deleting a stack deletes every single artifact that was created by
CloudFormation.

upload reference create Create

AWS Resources
Template S3 bucket Stack
AWS CloudFormation

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Deploying CloudFormation Templates
• Manual way
• Editing templates in Infrastructure Composer or code editor Template
• Using the console to input parameters, etc… create
Infrastructure
• We’ll mostly do this way in the course for learning Composer / Code
purposes Editor Stack

• Automated way CloudFormation Console

• Editing templates in a YAML file
• Using the AWS CLI (Command Line Interface) to deploy
the templates, or using a Continuous Delivery (CD) tool create-stack

• Recommended way when you fully want to automate

your flow AWS CLI CloudFormation

Template
© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Building Blocks
• Template’s Components
• AWSTemplateFormatVersion – identifies the capabilities of the template “2010-09-09”
• Description – comments about the template
• Resources (MANDATORY) – your AWS resources declared in the template
• Parameters – the dynamic inputs for your template
• Mappings – the static variables for your template
• Outputs – references to what has been created
• Conditionals – list of conditions to perform resource creation

• Template’s Helpers
• References
• Functions

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Introductory Example
• We’re going to create a simple EC2 instance Port: 22
Port: 80
• And we’re going to add security group to it
• For now, forget about the code syntax
• We’ll look at the structure of the files later
WebServer

WebServerSecurityGroup

• We’ll see how in no-time, we are able to get started

with CloudFormation!

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
YAML Crash Course
• YAML and JSON are the languages you can
use for CloudFormation
• JSON is horrible for CF
• YAML is great in so many ways
• Let’s learn a bit about it!

• Key value Pairs

• Nested objects
• Support Arrays
• Multi line strings
• Can include comments!
© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Resources
• Resources are the core of your CloudFormation template (MANDATORY)
• They represent the different AWS Components that will be created and
configured
• Resources are declared and can reference each other

• AWS figures out creation, updates and deletes of resources for us

• There are over 700 types of resources (!)
• Resource types identifiers are of the form:
service-provider::service-name::data-type-name

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
How do I find
Resources documentation?
• I can’t teach you all the 700+ resources, but I can teach you how to
learn how to use them
• All the resources can be found here:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/a
ws-template-resource-type-ref.html
• Then, we just read the docs J
• Example here (for an EC2 instance):
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/a
ws-resource-ec2-instance.html

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Analysis of CloudFormation Template
• Going back to the example of the introductory lecture, let’s learn why it
was written this way.
• Relevant documentation can be found here:
• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-
resource-ec2-instance.html
• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-
resource-ec2-securitygroup.html
• http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-
resource-ec2-eip.html

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Resources FAQ
• Can I create a dynamic number of resources?
ØYes, you can by using CloudFormation Macros and Transform
ØIt is not in the scope of this course

• Is every AWS Service supported?

ØAlmost. Only a select few niches are not there yet
ØYou can work around that using CloudFormation Custom Resources

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Parameters
User

• Parameters are a way to provide inputs to your upload template

AWS CloudFormation template +
provide parameter values
• They’re important to know about if:
• You want to reuse your templates across the company
Parameter1: value1
• Some inputs can not be determined ahead of time + Parameter2: value2
…
• Parameters are extremely powerful, controlled, CF template Parameters
and can prevent errors from happening in your
create stack
templates, thanks to types

AWS CloudFormation

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
When should you use a Parameter?

• Ask yourself this:

• Is this CloudFormation resource configuration likely to change in the future?
• If so, make it a parameter

• You won’t have to re-upload a template to change its content J

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Parameters Settings
• Parameters can be controlled by all these settings:
• Type: • Description
• String • ConstraintDescription (String)
• Number
• CommaDelimitedList • Min/MaxLength
• List<Number> • Min/MaxValue
• AWS-Specific Parameter (to • Default
help catch invalid values – match
against existing values in the AWS • AllowedValues (array)
account)
• List<AWS-Specific Parameter>
• AllowedPattern (regex)
• SSM Parameter (get parameter • NoEcho (Boolean)
value from SSM Parameter store)

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Parameters Example
AllowedValues NoEcho

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
How to Reference a Parameter?

• The Fn::Ref function can be leveraged to reference parameters

• Parameters can be used anywhere in a template
• The shorthand for this in YAML is !Ref
• The function can also reference other elements within the template

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Pseudo Parameters
• AWS offers us Pseudo Parameters in any CloudFormation template
• These can be used at any time and are enabled by default
• Important pseudo parameters:
Reference Value Example Returned Value
AWS::AccountId 123456789012
AWS::Region us-east-1
AWS::StackId arn:aws:cloudformation:us-east-1:123456789012:stack/MyStack/1c2fa620-982a-
11e3-aff7-50e2416294e0
AWS::StackName MyStack
AWS::NotificationARNs [arn:aws:sns:us-east-1:123456789012:MyTopic]
AWS::NoValue Doesn’t return a value

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Mappings
• Mappings are fixed variables within your CloudFormation template
• They’re very handy to differentiate between different environments
(dev vs prod), regions (AWS regions), AMI types…
• All the values are hardcoded within the template

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Accessing Mapping Values (Fn::FindInMap)
• We use Fn::FindInMap to return a named value from a specific key
• !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]

Mappings work great for AMIs

Because AMIs are region-specific!

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
When would you use Mappings vs. Parameters?
• Mappings are great when you know in advance all the values that can be
taken and that they can be deduced from variables such as
• Region
• Availability Zone
• AWS Account
• Environment (dev vs prod)
• etc…
• They allow safer control over the template

• Use parameters when the values are really user specific

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Outputs
• The Outputs section declares optional outputs values
that we can import into other stacks (if you export Network Stack
them first)!
• You can also view the outputs in the AWS Console Exported Output
or in using the AWS CLI VPC ID: vpc-2f09a348

• They’re very useful for example if you define a

network CloudFormation, and output the variables reference
such as VPC ID and your Subnet IDs
• It’s the best way to perform some collaboration cross
stack, as you let expert handle their own part of the
stack Application Stack

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Outputs
• Creating a SSH Security Group as part of one template
• We create an output that references that security group

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Outputs Cross-Stack
Reference
• We then create a second template that leverages that security group
• For this, we use the Fn::Impor tValue function
• You can’t delete the underlying stack until all the references are deleted

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Conditions
User
• Conditions are used to control the creation of
resources or outputs based on a condition
Parameters Parameters
• Conditions can be whatever you want them to
Environment: Environment:
be, but common ones are: dev prod

• Environment (dev / test / prod) template

• AWS Region Development Production

• Any parameter value

• Each condition can reference another condition,
parameter value or mapping
EC2 Instance EC2 Instance

EBS Volume

Dev Stack Prod Stack

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
How to define a Condition

• The logical ID is for you to choose. It’s how you name condition
• The intrinsic function (logical) can be any of the following:
• Fn::And
• Fn::Equals
• Fn::If
• Fn::Not
• Fn::Or

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
How to use a Condition
• Conditions can be applied to resources / outputs / etc…

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Intrinsic Functions Blue = must know

• Ref • Fn::Base64
• Fn::GetAtt • Fn::Cidr
• Fn::FindInMap • Fn::GetAZs
• Fn::Impor tValue • Fn::Select
• Fn::Join
• Fn::Split
• Fn::Sub
• Fn::Transform
• Fn::ForEach
• Fn::ToJsonString • Fn::Length
• Condition Functions (Fn::If, Fn::Not, Fn::Equals, etc…)

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Intrinsic Functions – Fn::Ref
• The Fn::Ref function can be leveraged to reference
• Parameters – returns the value of the parameter
• Resources – returns the physical ID of the underlying resource (e.g., EC2 ID)
• The shorthand for this in YAML is !Ref

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Intrinsic Functions – Fn::GetAtt
• Attributes are attached to any resources you create
• To know the attributes of your resources, the best place to look at is
the documentation
• Example: the AZ of an EC2 instance!

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Intrinsic Functions – Fn::FindInMap
• We use Fn::FindInMap to return a named value from a specific key
• !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Intrinsic Functions – Fn::ImportValue
• Import values that are exported in other stacks
• For this, we use the Fn::Impor tValue function

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Intrinsic Functions – Fn::Base64
• Convert String to it’s Base64 representation

• Example: pass encoded data to EC2 Instance’s UserData property

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Intrinsic Functions – Condition Functions

• The logical ID is for you to choose. It’s how you name condition
• The intrinsic function (logical) can be any of the following:
• Fn::And
• Fn::Equals
• Fn::If
• Fn::Not
• Fn::Or

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Rollbacks
• Stack Creation Fails:
• Default: everything rolls back (gets deleted). We can look at the log
• Option to disable rollback and troubleshoot what happened

• Stack Update Fails:

• The stack automatically rolls back to the previous known working state
• Ability to see in the log what happened and error messages

• Rollback Failure? Fix resources manually then issue

ContinueUpdateRollback API from Console
• Or from the CLI using continue-update-rollback API call

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Service Role Permissions
- cloudformation:* User
- iam:PassRole
• IAM role that allows CloudFormation to
create/update/delete stack resources on your
behalf Template

• Give ability to users to create/update/delete the

stack resources even if they don’t have Service Role
permissions to work with the resources in the - s3:*Bucket
stack
CloudFormation
• Use cases:
• You want to achieve the least privilege principle
• But you don’t want to give the user all the required
permissions to create the stack resources
• User must have iam:PassRole permissions S3 bucket
Stack
© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation Capabilities
• CAPABILITY_NAMED_IAM and CAPABILITY_IAM
• Necessary to enable when you CloudFormation template is creating or updating IAM
resources (IAM User, Role, Group, Policy, Access Keys, Instance Profile…)
• Specify CAPABILITY_NAMED_IAM if the resources are named

• CAPABILITY_AUTO_EXPAND
• Necessary when your CloudFormation template includes Macros or Nested Stacks
(stacks within stacks) to perform dynamic transformations
• You’re acknowledging that your template may change before deploying

• InsufficientCapabilitiesException
• Exception that will be thrown by CloudFormation if the capabilities haven’t been
acknowledged when deploying a template (security measure)

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – DeletionPolicy Delete
• DeletionPolicy:
• Control what happens when the
CloudFormation template is
deleted or when a resource is
removed from a CloudFormation
template
• Extra safety measure to preserve
and backup resources
• Default DeletionPolicy=Delete
• ⚠ Delete won’t work on an S3 ⚠
bucket if the bucket is not empty

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – DeletionPolicy Retain
• DeletionPolicy=Retain:
• Specify on resources to preserve
in case of CloudFormation
deletes
• Works with any resources

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – DeletionPolicy Snapshot
• DeletionPolicy=Snapshot
• Create one final snapshot before
deleting the resource
• Examples of supported resources:
• EBS Volume, ElastiCache Cluster,
ElastiCache ReplicationGroup
• RDS DBInstance, RDS DBCluster,
Redshift Cluster, Neptune DBCluster,
DocumentDB DBCluster

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Stack Policies
• During a CloudFormation Stack update, all
update actions are allowed on all resources
(default)

• A Stack Policy is a JSON document that

defines the update actions that are allowed
on specific resources during Stack updates
• Protect resources from unintentional updates
• When you set a Stack Policy, all resources in
the Stack are protected by default
• Specify an explicit ALLOW for the resources Allow updates on all resources
you want to be allowed to be updated except the ProductionDatabase

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Termination Protection
• To prevent accidental deletes of CloudFormation Stacks, use
TerminationProtection

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Custom Resources
• Used to
• define resources not yet supported by CloudFormation
• define custom provisioning logic for resources can that be outside of
CloudFormation (on-premises resources, 3rd party resources…)
• have custom scripts run during create / update / delete through Lambda
functions (running a Lambda function to empty an S3 bucket before being
deleted)
• Defined in the template using
AWS::CloudFormation::CustomResource or
Custom::MyCustomResourceTypeName (recommended)
• Backed by a Lambda function (most common) or an SNS topic

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
How to define a Custom Resource?
• ServiceToken specifies where CloudFormation sends requests to, such
as Lambda ARN or SNS ARN (required & must be in the same region)
• Input data parameters (optional)

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Use Case – Delete content from an S3 bucket
User
• You can’t delete a non-empty delete stack
S3 bucket
• To delete a non-empty S3 CloudFormation
bucket, you must first delete all
the objects inside it
• We can use a custom resource Stack

to empty an S3 bucket before

it gets deleted by Custom resource Lambda function
CloudFormation empty bucket

S3 bucket

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Custom Resources – How does it work?
Resource Provider
Template with
custom resource
request contains
S3 pre-signed URL for response

create/update/delete AWS Lambda API calls

Whatever you want
OR
Template developer CloudFormation

Amazon SNS

listens upload JSON response to S3

Using S3 pre-signed URL

Amazon S3

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Hands-On: Lambda-backed Custom Resource
User
• You can’t delete a non-empty S3 bucket
delete stack
• To delete a non-empty S3 bucket, you
must first delete all the objects inside it CloudFormation

• We’ll create a custom resource with AWS

Stack
Lambda that will be used to empty an S3
bucket before deleting it
Custom resource Lambda function

empty bucket
• Let’s create our first Custom Resource!

S3 bucket
© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Dynamic References
Template
• Reference external values stored in Systems Manager create/update
Parameter Store and Secrets Manager within
CloudFormation templates
• CloudFormation retrieves the value of the specified CloudFormation
reference during create/update/delete operations
get value
• For example: retrieve RDS DB Instance master password result
(reference-key)
from Secrets Manager
• Supports
• ssm – for plaintext values stored in SSM Parameter Store
SSM Parameter Store
• ssm-secure – for secure strings stored in SSM Parameter Store
• secretsmanager – for secret values stored in Secrets Manager
‘{{resolve:service-name:reference-key}}’
Secrets Manager
© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Dynamic References
SSM SSM Secure
{{resolve:ssm:parameter-name:version}} {{resolve:ssm-secure:parameter-name:version}}

Secrets Manager
{{resolve:secretsmanager:secret-id:secret-string:json-key:version-stage:version-id}}

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation, Secrets Manager & RDS
Option 1 – ManageMasterUserPassword
• ManageMasterUserPassword – creates admin secret implicitly
• RDS, Aurora will manage the secret in Secrets Manager and its rotation

Secrets
Manager
create Secret
create Stack

User CloudFormation
RDS
create DB and
configure
Username/Password

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation, Secrets Manager & RDS
Option 2 – Dynamic Reference
1. secret is generated 3. link the secret to
RDS DB instance (for rotadon)

2. Reference secret in
RDS DB instance

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
User Data in EC2 for CloudFormation
• We can have user data at EC2 instance launch through the console
• Let’s learn how to write the same EC2 user-data script in our
CloudFormation template

• The impor tant thing to pass is the entire script through the function
Fn::Base64
• Good to know, user data script log is in /var/log/cloud-init-output.log

• Let’s see how to do this in CloudFormation!

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
The Problems with EC2 User Data
• What if we want to have a very large instance configuration?
• What if we want to evolve the state of the EC2 instance without terminating
it and creating a new one?
• How do we make EC2 user-data more readable?
• How do we know or signal that our EC2 user-data script completed
successfully?

• Enter CloudFormation Helper Scripts!

• Python scripts, that come directly on Amazon Linux AMIs, or can be installed using yum
or dnf on non-Amazon Linux AMIs
• cfn-init, cfn-signal, cfn-get-metadata, cfn-hup

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
AWS::CloudFormation::Init
• A config contains the following and is executed in
that order
• Packages: used to download and install pre-packaged
apps and components on Linux/Windows (ex. MySQL,
PHP, etc…)
• Groups: define user groups
• Users: define users, and which group they belong to
• Sources: download files and archives and place them on
the EC2 instance
• Files: create files on the EC2 instance, using inline or can
be pulled from a URL
• Commands: run a series of commands
• Services: launch a list of sysvinit

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – cfn-init
CloudFormation
• Used to retrieve and interpret the resource
metadata, installing packages, creating files and
starting services
• With the cfn-init script, it helps make complex

retrieve init data

EC2 configurations readable

launch
• The EC2 instance will query the CloudFormation
service to get init data
• AWS::CloudFormation::Init must be in the
Metadata of a resource
• Logs go to /var/log/cfn-init.log
run cfn-init
EC2 Instance

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – cfn-signal & Wait Conditions
CloudFormation
• We still don’t know how to tell CloudFormation that the
EC2 instance got properly configured after a cfn-init Wait Condition
• For this, we can use the cfn-signal script!
• We run cfn-signal right after cfn-init
• Tell CloudFormation service that the resource creation success/fail

retrieve init data

to keep on going or fail

signal from
cfn-signal
launch
• We need to define WaitCondition:
• Block the template until it receives a signal from cfn-signal
• We attach a CreationPolicy (also works on EC2, ASG)
• We can define a Count > 1 (in case you need more than 1
signal)

run cfn-init
EC2 Instance

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Wait Condition Didn't Receive the Required
Number of Signals from an Amazon EC2 Instance
• Ensure that the AMI you're using has the AWS CloudFormation helper scripts
installed. If the AMI doesn't include the helper scripts, you can also download them
to your instance
• Verify that the cfn-init & cfn-signal command was successfully run on the instance.
You can view logs, such as /var/log/cloud-init.log or /var/log/cfn-init.log, to help you
debug the instance launch
• You can retrieve the logs by logging in to your instance, but you must disable
rollback on failure or else AWS CloudFormation deletes the instance after your
stack fails to create
• Verify that the instance has a connection to the Internet. If the instance is in a VPC,
the instance should be able to connect to the Internet through a NAT device if it's is
in a private subnet or through an Internet gateway if it's in a public subnet
• For example, run: curl -I https://aws.amazon.com

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Nested Stacks
Stack A (root)
• Nested stacks are stacks as part of other stacks
• They allow you to isolate repeated patterns / Stack B
common components in separate stacks and call
them from other stacks Stack D
• Example:
• Load Balancer configuration that is re-used
• Security Group that is re-used Stack C
Stack E Stack F
• Nested stacks are considered best practice
• To update a nested stack, always update the parent
(root stack)
• Nested stacks can have nested stacks themselves!

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Cross Stacks vs. Nested Stacks
App 1 Stack
• Cross Stacks
• Helpful when stacks have different lifecycles
App 2 Stack
• Use Outputs Export and Fn::ImportValue
• When you need to pass export values to VPC Stack
many stacks (VPC Id…)
App 3 Stack

• Nested Stacks
• Helpful when components must be re-used
• Example: re-use how to properly configure
an Application Load Balancer
• The nested stack only is important to the RDS Stack ASG Stack RDS Stack ASG Stack
higher-level stack (it’s not shared)

ELB Stack ELB Stack

App 1 Stack App 2 Stack

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – DependsOn
• Specify that the creation of a specific resource
follows another
• When added to a resource, that resource is
created only after the creation of the resource
specified in the DependsOn attribute
• Applied automatically when using !Ref and
!GetAtt
• Use with any resource

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – StackSets
• Create, update, or delete stacks across Administrator account
multiple accounts and regions with a
single operation/template Create StackSet

• Administrator account to create StackSet

Template
StackSets
• Target accounts to create, update,
delete stack instances from StackSets
Region 1 Region 2
• When you update a stack
set, all associated stack instances are
updated throughout all accounts and Target account A Target account B Target account A Target account C
regions
• Can be applied into all accounts of an
AWS organizations Stack Instance Stack Instance Stack Instance Stack Instance

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – StackSets Permission Models
Administrator account

• Self-managed Permissions AWSCloudFormationStackSetAdministrationRole

• Create the IAM roles (with established trusted
relationship) in both administrator and target accounts Trust relationship

• Deploy to any target account in which you have

Target accounts
permissions to create IAM role
AWSCloudFormationStackSetExecutionRole
• Service-managed Permissions
• Deploy to accounts managed by AWS Organizations
• StackSets create the IAM roles on your behalf (enable
trusted access with AWS Organizations)
• Must enable all features in AWS Organizations Management account

• Ability to deploy to accounts added to your

organization in the future (Automatic Deployments) Target
accounts

AWS Organizations
© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
StackSets with AWS Organizations
• Ability to automatically deploy Stack instances to new Accounts in an
Organization

• Can delegate StackSets administration to member accounts in AWS

Organization
• Trusted access with AWS Organizations must be enabled before
delegated administrators can deploy to accounts managed by
Organizations

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
StackSets with AWS Organizations
AWS Organization

(Delegated) Administrator Account

StackSet

manage

OU (Prod) OU (Dev)

Member Account - A Member Account - B Member Account - C New Account

Stack Instance Stack Instance Stack Instance Stack Instance

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Hands-On: StackSets
• We’ll use StackSets to enable AWS Administrator/Target account

Config across AWS regions with a single Create StackSet

click
Template StackSet

enable
• Let’s see how this works!
Region 1 Region 2 Region 3

AWS Config AWS Config AWS Config

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Troubleshooting
• DELETE_FAILED
• Some resources must be emptied before deleting, such as S3 buckets
• Use Custom Resources with Lambda functions to automate some actions
• Security Groups cannot be deleted until all EC2 instances in the group are gone
• Think about using DeletionPolicy=Retain to skip deletions

• UPDATE_ROLLBACK_FAILED
• Can be caused by resources changed outside of CloudFormation, insufficient
permissions, Auto Scaling Group that doesn’t receive enough signals…
• Manually fix the error and then ContinueUpdateRollback

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – StackSet Troubleshooting
• A stack operation failed, and the stack instance status is OUTDATED.
• Insufficient permissions in a target account for creating resources that are
specified in your template.
• The template could be trying to create global resources that must be unique but
aren't, such as S3 buckets
• The administrator account does not have a trust relationship with the target
account
• Reached a limit or a quota in the target account (too many resources)

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – ChangeSets
• When you update a stack, you need to know what changes will happen
before it applying them for greater confidence
• ChangeSets won’t say if the update will be successful
• For Nested Stacks, you see the changes across all stacks

create view execute

Original Stack Change Set Change Set AWS CloudFormadon

create additional
Change Sets (optional)

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
cfn-hup
• Can be used to tell your EC2 instance to look for
Metadata changes every 15 minutes and apply the
Metadata configuration again AWS CloudFormation
• It’s very powerful but you really need to try it out

changes every 15 mins

retrieve init data
to understand how it works

launch

check Metadata
• It relies on a cfn-hup configuration, see
/etc/cfn/cfn-hup.conf and /etc/cfn/hooks.d/cfn-
auto-reloader.conf EC2 Instance
re-run configuration
if changes detected

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
CloudFormation – Drift
• CloudFormation allows you to create
infrastructure SSHSecurityGroup
Type Protocol/Port Source

• But it doesn’t protect you against manual SSH TCP/22 10.0.0.115/32

configuration changes Stack

so g
on s i n
le
• How do we know if our resources have drifted?

2C yu
EC odif
m
Type Protocol/Port Source

SSH TCP/22 0.0.0.0/0

• We can use CloudFormation Drift!
• Detect drift on an entire stack or on individual Compare

resources within a stack

Template Drifted!
CloudFormation Drift

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
StackSet Drift Detection
Administrator account
• Performs drift detection on the stack associated with
each stack instance in the StackSet
• If the current state of a resource in a stack varies from
the expected state: StackSet
• The stack considered drifted
• And the stack instance that the stack associated with
considered drifted
• And the StackSet is considered drifted
Target account
• Drift detection identifies unmanaged changes (outside
CloudFormation)
Stack Instance
• Changes made through CloudFormation to a stack
directly (not at the StackSet level), aren’t considered Stack
drifted modify through
• You can stop drift detection on a StackSet EC2 console
EC2 instance

User

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
AWS Service Catalog
• Users that are new to AWS have too many options, and may create
stacks that are not compliant / in line with the rest of the organization

• Some users just want a quick self-service por tal to launch a set of
authorized products pre-defined by admins

• Includes: virtual machines, databases, storage options, etc…

• Enter AWS Service Catalog!

© Stephane Maarek
 NOT FOR DISTRIBUTION © Stephane Maarek www.datacumulus.com
Service Catalog diagram
Product Portfolio Control
ADMIN TASKS

CloudFormation Collecdon of Products IAM Permissions to

Templates Access Portfolios
USER TASKS

Product List Provisioned Products

launch

Authorized by IAM Ready to use

Properly Configured
Properly Tagged
© Stephane Maarek