Immutable Deployments with AWS CloudFormation and AWS Lambda
27 likes•45,509 views
This document describes an immutable infrastructure approach using AWS Lambda and CloudFormation. Key points: - Infrastructure is defined as code using CloudFormation templates for reproducibility and versioning. - Lambda functions are used to provision resources, configure settings, run tests, and clean up resources to enforce immutability. - A pipeline handles building AMIs, deploying stacks, testing, updating DNS, and deleting old stacks in an automated and repeatable way.
![var r = require('cfn-response');
exports.handler = function (event, context) {
[…]
var res = {};
if (event.RequestType == 'Create') {
res.Password = randomPassword(20);
}
r.send(event, context, r.SUCCESS, res);
};](https://image.slidesharecdn.com/2016-03-17advancedawsmeetup-immutabledeployments-160318055040/85/Immutable-Deployments-with-AWS-CloudFormation-and-AWS-Lambda-29-320.jpg)
!["IndexerDb": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
[...]
"MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]},
[...]
}
},
"GenerateDbPassword": {
"Type": "Custom::PasswordGenerator",
"Properties": {
"ServiceToken": {"Ref": "PasswordGeneratorArn"}
}
},](https://image.slidesharecdn.com/2016-03-17advancedawsmeetup-immutabledeployments-160318055040/85/Immutable-Deployments-with-AWS-CloudFormation-and-AWS-Lambda-30-320.jpg)
!["CountInstances": {
"Type": "Custom::InstanceCounter",
"Properties": {
"ServiceToken": {"Ref": "InstanceCounter"},
"AutoScalingGroupTags": [
{"Key": "Environment", "Value": "prod"},
{"Key": "Type", "Value": "Frontend"}
],
"Min": 1, "Max": 10, "Factor": "1.5"
}
},
"FrontendAsg": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
[...]
"DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]},
"Tags": [
{"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true},
{"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true}
]
}
},](https://image.slidesharecdn.com/2016-03-17advancedawsmeetup-immutabledeployments-160318055040/85/Immutable-Deployments-with-AWS-CloudFormation-and-AWS-Lambda-37-320.jpg)
!["UpdateR53": {
"DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"],
"Type": "Custom::Route53Update",
"Properties": {
"ServiceToken": {"Ref": "R53Updater"},
"Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]},
"HostedZoneId": {"Ref": "HostedZoneId"},
"AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]},
"AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]},
"Comment": "Updated via CloudFormation/Lambda"
}
},
"DeleteStacks": {
"Condition": "DeleteOldStacks",
"DependsOn": ["UpdateR53"],
"Type": "Custom::StackDeleter",
"Properties": {
"ServiceToken": {"Ref": "StackDeleter"},
"TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"},
"ExceptStackName": {"Ref": "AWS::StackName"}
}
}](https://image.slidesharecdn.com/2016-03-17advancedawsmeetup-immutabledeployments-160318055040/85/Immutable-Deployments-with-AWS-CloudFormation-and-AWS-Lambda-40-320.jpg)
![$repository = new AwsInspectorModelElbRepository();
$dns = $repository->findElbsByTags([
'Environment' => 'deploy',
'Build' => 554,
'Type' => 'Frontend’
])->getFirst()->getDNSName();](https://image.slidesharecdn.com/2016-03-17advancedawsmeetup-immutabledeployments-160318055040/85/Immutable-Deployments-with-AWS-CloudFormation-and-AWS-Lambda-53-320.jpg)
![> bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build
filter by tag
Please select an instance
[0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477)
[1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477)
[2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477)
[3] i-705ad42f (Type: Worker; Environment: prod; Build: 476)
>
• will take jump hosts into
account (ProxyCommand)
• auto-detects your local
(encrypted) private keys
• multiplexed ssh connections
• run commands directly](https://image.slidesharecdn.com/2016-03-17advancedawsmeetup-immutabledeployments-160318055040/85/Immutable-Deployments-with-AWS-CloudFormation-and-AWS-Lambda-54-320.jpg)
Immutable Deployments with AWS CloudFormation and AWS Lambda
- 3. & aoepeople!
- 4. E-Commerce: Magento CMS: TYPO3 Portals: ZF, Symfony,… Mobile Searchperience: ElasticSearch 250+ people world-wide (in 8 locations) Global Enterprise Projects Infrastructure: AWS
- 6. /i(m)ˈmyo͞ odəb(ə)l/ adjective unchanging over time or unable to be changed. “disposable” “ephemeral”
- 7. Pet Cattle
- 15. Static Resources BuildVPC Build
- 17. Build Bucket IAM Setup VPC Static Resources Build Manual Setup separate CloudFormation Stacks use resources from underlying stacks as input parameters Lambda functions, Monitoring,…
- 18. Build Bucket IAM Setup Environment A (e.g. prod) Environment B (e.g. stage) VPC VPC Static Resources Static Resources Build BuildBuild Build Build X Build X+1 Build X Build X+1 Manual Setup different scopes (build, environment, account, global…)
- 19. Build Bucket IAM Setup Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. dev) IAM Setup VPC VPC VPC VPC Static Resources Static Resources Static Resources Static Resources Build BuildBuild Build Build Build Build Build Build X Build X+1 Build X Build X+1 Build X Build X+1 Build X Build X+1 Manual Setup Manual Setup Build Bucket Access
- 20. Private subnets Public subnets ElastiCache (Redis) with replication groups for cache and sessions RDS (multi-az) with DB subnet group Bastion server s3: media storage* Route 53: DNS configuration CloudFront distribution SSL Certificates Security group for Varnish servers Security group for Magento servers Security group for Load Balancer Static Resources
- 21. Build Auto Scaling group Auto Scaling group Elastic Load Balancer Auto- Scaling Group Launch Configurati on Scaling Policy
- 22. Build Auto Scaling group Auto Scaling group Elastic Load Balancer Auto- Scaling Group Launch Configurati on Scaling Policy Auto Scaling group Auto Scaling group
- 23. +1 Baking AMIs
- 25. Keep it simple…!
- 26. ✔ ✔ (✔)
- 27. the “last mile” most underestimated CFN feature!
- 29. var r = require('cfn-response'); exports.handler = function (event, context) { […] var res = {}; if (event.RequestType == 'Create') { res.Password = randomPassword(20); } r.send(event, context, r.SUCCESS, res); };
- 30. "IndexerDb": { "Type": "AWS::RDS::DBInstance", "Properties": { [...] "MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]}, [...] } }, "GenerateDbPassword": { "Type": "Custom::PasswordGenerator", "Properties": { "ServiceToken": {"Ref": "PasswordGeneratorArn"} } },
- 32. • launch some ASGs (set DesiredCapacity) • create database passwords • tag all resources (incl. ElastiCache!) • restore database and media files • (one-time) install scripts (db migrations,…) • detect Varnish backends • wait for healthy backends in the ELBs • run infrastructure tests • cache warming • update Route53 records sets • delete old stacks • …
- 34. DesiredCapacity = Number of Instances in current ASG x 1.2 better safe than sorry…
- 35. DesiredCapacity = Number of Instances in current ASG x 1.2
- 36. DesiredCapacity = Number of Instances in current ASG x 1.2 ScalingPolicy takes care…
- 37. "CountInstances": { "Type": "Custom::InstanceCounter", "Properties": { "ServiceToken": {"Ref": "InstanceCounter"}, "AutoScalingGroupTags": [ {"Key": "Environment", "Value": "prod"}, {"Key": "Type", "Value": "Frontend"} ], "Min": 1, "Max": 10, "Factor": "1.5" } }, "FrontendAsg": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { [...] "DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]}, "Tags": [ {"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true}, {"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true} ] } },
- 38. Update Route53 record set
- 39. Update Route53 record set Delete old stacks
- 40. "UpdateR53": { "DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"], "Type": "Custom::Route53Update", "Properties": { "ServiceToken": {"Ref": "R53Updater"}, "Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]}, "HostedZoneId": {"Ref": "HostedZoneId"}, "AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]}, "AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]}, "Comment": "Updated via CloudFormation/Lambda" } }, "DeleteStacks": { "Condition": "DeleteOldStacks", "DependsOn": ["UpdateR53"], "Type": "Custom::StackDeleter", "Properties": { "ServiceToken": {"Ref": "StackDeleter"}, "TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"}, "ExceptStackName": {"Ref": "AWS::StackName"} } }
- 43. Have fun filling out 47 form fields! :)
- 44. aoepeople/stackformation command-line tool (Symfony console, uses AWS SDK for PHP) integrates nicely into your CI (Jenknis,…)
- 45. pre-process Userdata Lambda JS “Template” superset of CFN template (JSON) Parameterslookup from other stacks, env vars,… Template Parameter Values create/ update Stack policies, Tags,…
- 46. Stack magento-stage-build-5 Stack magento-stage-build-6 Stack magento-prod-build-6 Stack magento-prod-build-5 CloudFormation Template merge & pre-process “CloudFormation+X” Template(s) + Dynamic Parameters + Stack Policies + Behavior + Tags … Blueprint magento-{env:ENVIRONMENT}-build-{env:BUILD} Blueprint magento-{env:ENVIRONMENT}-setup Stack magento-stage-setup Stack magento-prod-setup
- 47. blueprints: - stackname: 'magento-{env:BUILD}' template: 'magento.template' stackPolicy: 'policy.json' OnFailure: 'DO_NOTHING' parameters: Build: '{env:BUILD}' KeyPair: '{var:KeyPair}' VPC: '{resource:setupstack:VPC}' Subnet: '{resource:setupstack:Subnet}' InstanceSg: '{resource:setupstack:InstanceSg}' InstanceProfile: '{output:setupstack:InstanceProfile}' BootAmi: 'ami-06116566' tags: Environment: 'prod' Build: '{env:BUILD}' enforce “immutability” by denying updates!
- 49. so we can integrate this into our CI pipeline…
- 50. •
- 52. aoepeople/awsinspector command-line tool (Symfony console, uses AWS SDK for PHP) Domain models for PHP
- 53. $repository = new AwsInspectorModelElbRepository(); $dns = $repository->findElbsByTags([ 'Environment' => 'deploy', 'Build' => 554, 'Type' => 'Frontend’ ])->getFirst()->getDNSName();
- 54. > bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build filter by tag Please select an instance [0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477) [1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477) [2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477) [3] i-705ad42f (Type: Worker; Environment: prod; Build: 476) > • will take jump hosts into account (ProxyCommand) • auto-detects your local (encrypted) private keys • multiplexed ssh connections • run commands directly
- 56. logins, orders,… deployments, scaling activity,… JMeter response time, error rate,… CPU, load, network I/O,…
- 60. … “Ready for testing” (WaitCondition) setup terminate instances and wait • Create Security allowing access to all IPs and attach to UAT’s ELB environment (incl. restoring db+media snapshot from prod) environment* Delete self* *via Lambda custom resource Pushing samples in near real-time via custom JMeter backend listener “Testing Completed” (WaitCondition) JMeter test CloudFormation Stack GrafanaElasticSearch (Service) CloudWatch Lambda find relevant resources for the current deployment (via tags), collect metrics, and push them to ElasticSearch
- 61. CloudFormation Stack • Install JMeter • Download testcase from S3 • Signal “Ready” and wait • Create SG allowing access to all IPs and attach to UAT’s ELB Auto-Scaling Group of Load Generator Instances Run Stress Test2. Time timeout 3600 jmeter –n –t testcase.jmx Delete test environment* Delete self* 3. 4. *via Lambda custom resource … Spin up test environment (incl. restoring db+media snapshot from prod) 1. CloudFormation Stack setup terminate instances JMeter test Pushing samples in near real-time via custom JMeter backend listener “Ready for testing” (WaitCondition) “Testing Completed” (WaitCondition)
- 66. Provisioning “Ready for baking” (WaitCondition) EC2 AMI Baker (Lambda) Node.js SDK AWS CLI “Baking Completed” (WaitCondition) shutdown (-> terminate) ec2.createImage aws ec2 wait image-available AMI Baker (Lambda) Node.js SDK ec2.deleteImage Custom CloudFormation Resource Time
- 68. •
- 69. Follow me on twitter! My blog