Amazon EC2 Cookbook - Sample Chapter
Amazon EC2 Cookbook - Sample Chapter
ee
$ 34.99 US
22.99 UK
P U B L I S H I N G
pl
e
Aurobindo Sarkar
Sekhar Reddy
Sa
Q u i c k
a n s w e r s
t o
c o m m o n
p r o b l e m s
Sekhar Reddy
Aurobindo Sarkar
P U B L I S H I N G
Aurobindo Sarkar is actively working with several start-ups in the role of CTO/technical
director. With a career spanning more than 22 years, he has consulted at some of the leading
organizations in the the US, the UK, and Canada. He specializes in software-as-a-service product
development, cloud computing, big data analytics, and machine learning. His domain expertise
is in financial services, media, public sector, mobile gaming, and automotive sectors. Aurobindo
has been actively working with technology startups for over 5 years now. As a member of the
top leadership team at various startups, he has mentored several founders and CxOs, provided
technology advisory services, developed cloud strategy, product roadmaps, and set up large
engineering teams. Aurobindo has an MS (computer science) from New York University, M.Tech
(management) from Indian Institute of Science, and B.Tech (engineering) from IIT Delhi.
Preface
With the increasing interest in leveraging cloud infrastructure around the world, AWS Cloud
from Amazon offers a cutting-edge platform to architecture, build, and deploy web-scale cloud
applications. The variety of services and features available from AWS can reduce the overall
infrastructure costs and accelerate the development process for both large enterprises and
startups alike. In such an environment, it is imperative for developers to be able to set up the
required infrastructure and effectively use various cloud services provided by AWS. In addition,
they also should be able to effectively secure access to their production environments and
deploy and monitor their applications.
Amazon EC2 Cookbook will serve as a handy reference to developers building production
applications or cloud-based products. It will be a trusted desktop reference book that you
reach out to first, or refer to often, to find solutions to specific AWS development-related
requirements and issues. If you have a specific task to be completed, then we expect you to
jump straight to the appropriate recipe in the book. By working through the steps in a specific
recipe, you can quickly accomplish the typical tasks and issues related to the infrastructure,
development, and deployment of an enterprise-grade AWS Cloud application.
Preface
Chapter 5, Monitoring Amazon EC2 Instances, contains recipes for monitoring your EC2
instances using AWS CloudWatch. It will also cover a related topicautoscaling.
Chapter 6, Using AWS Data Services, contains recipes for using various AWS relational and
NoSQL data services in AWS applications.
Chapter 7, Accessing Other AWS Services, contains recipes for accessing key AWS services
(other than AWS data services). These services include Route 53, Amazon S3, AWS SES,
AWS SNS, and AWS SQS.
Chapter 8, Deploying AWS Applications, talks about the recipes for AWS application
deployments using Docker containers, Chef cookbooks, and Puppet recipes.
Selecting and
Configuring Amazon
EC2 Instances
In this chapter, we will cover recipes for:
Creating an instance with multiple NIC cards and a static private IP address
Introduction
You need to ask yourself several questions in order to choose the right AWS EC2 instance for
meeting your requirements. These include: What is the primary purpose of the EC2 instance
being provisioned? What is the duration of your need for a particular machine? Do you need
high performance storage? Should you go for dedicated or shared tenancy? Will the machine
be used for compute-intensive or memory-intensive processing? What are the scalability,
availability, and security requirements? What are your networking requirements? There are
several options available for each of these parameters, and we will describe them in our recipes
for making the right choices. For low latency, you can host your application in the AWS region
nearest to the end user. Each AWS region is a separate geographic area, and has multiple
isolated locations called availability zones. These availability zones are individual data centers in
each region. They are used to deploy fault-tolerant and highly available applications. The latency
between these availability zones is very low. If something goes wrong in an availability zone, then
it does not affect the systems in another availability zone.
On-demand instances: These instances are billed on an hourly basis and no upfront
payments are required. Applications with unpredictable workloads or short-duration
requirements are best handled using on-demand instances. This is the default
purchasing option in AWS.
Spot instances: There are no upfront costs for provisioning spot instances, and
the costs are typically much lower than the on-demand instances. The provisioning
is done through a bidding process. If you lose the bid, you will not get the EC2
instances. Usually, applications that are viable only at very low compute prices
are a good use case for using spot instances.
Chapter 1
There are several AWS EC2 instance families available for different types of application
workloads. These include general purpose, memory optimized, compute optimized, storage
optimized, and GPU instances. Choosing the right instance type is a key decision in provisioning
EC2 instances.
Refer to http://aws.amazon.com/ec2/instance-types/ for
descriptions and typical use cases for each of these EC2 instance types.
We recommend that you start with a minimum required instance type that meets your
requirements. In many cases, choosing a general-purpose EC2 instance is a good starting
point. You can then load test your application on this instance for overall performance and
stability. If your applications are not meeting your performance objectives on the current
instance type, you can easily upgrade the size or choose a more specialized instance type,
though this process does require a reboot of your instance. This approach can help you
optimize your instance sizes and types.
To achieve high performance or meet compliance requirements or to just avoid noisy
neighbors, the type of tenancy chosen is a critical decision. On AWS, there are two types
of tenancy, dedicated and shared. In the case of dedicated tenancy, AWS provisions your
instance on dedicated hardware. These instances are isolated from instances created using
the shared tenancy option and instances created by other tenants. Tenancy can be configured
at the instance level or at the VPC level. Once the option is selected, changing the tenancy
type (instance or VPC level) is not allowed. There are cost implications of using dedicated
tenancy versus shared tenancy.
In addition, if we want to set the Provisioned IOPS parameter, then we have to use the EBSoptimized instance types. Amazon EBS-optimized instances deliver dedicated throughput
to Amazon EBS, with options ranging between 500 Mbps and 2,000 Mbps (depending on
the instance type selected). EBS-optimized flag provides dedicated and more consistent link
between EC2 and EBS. EBS optimized EC2 instances also allocate dedicated bandwidth to
its attached volumes.
How to do it
In this recipe, we will create and launch an EC2 instance.
1. After you log in to the AWS console, choose Services, and then select EC2 from the
list of AWS services. At this stage, the EC2 Dashboard will appear, then perform the
following operations:
1. Press the Launch Instance button.
Chapter 1
3. Choose EBS-Optimized Available instance type in the Choose an Instance Type
wizard to avail this performance benefit:
5. If you don't want to customize any further then review and launch the instance.
How to do it
In the following two sections, we list the set of instructions required to accomplish this on
Linux and Windows/Mac platforms.
Chapter 1
You can get the AWS credentials from AWS management portal by following these steps:
1. Log in to the AWS management portal using your AWS username and password.
2. Select account name from top menu at the right corner in the console.
3. Select security credentials.
4. Click on access keys (access key ID and secret access key).
5. Click on the Create New Access Key button.
6. Click on Download Key File, which will download the file. If you do not download the
key file now, you will not be able to retrieve your secret access key again.
7.
2. Start by installing Python. Download the compressed TAR archive file from the Python
site, and then install it using the commands listed below. The following steps target
the apt-based Linux distributions:
$ sudo apt-get install gcc
$ wget https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
$ tar -zxvf Python-2.7.8.tgz
$ cd Python-2.7.8
$ ./configure
$ make
$ sudo make install
6. Then install AWS CLI. If you have already installed awscli, you can upgrade the
installation using the upgrade option.
$ sudo pip install awscli
7.
2. Start by installing Python. Download the installer from the following URL and install
Python by using that installer: https://www.python.org/downloads/.
3. Check your Python installation:
$ python help
4. Before installing pip, check whether pip is already installed on your machine or not
by using the following command. If pip is already installed on your machine, skip to
the awscli installation step.
$ pip help
5. In the next step, we install pip. Download and run the installation script from
https://bootstrap.pypa.io/get-pip.py. After that, run the following
command:
$ python get-pip.py
Chapter 1
6. Install AWS CLI. If you have already installed awscli, you can upgrade the
installation using the upgrade option.
$ pip install awscli
7.
Next, we configure AWS CLI. Execute the following command from the
command prompt.
$ aws configure
This command will then prompt you for the AWSAccessKey ID, AWSSecretKey,
default AWS region, and default output format.
8. Check the installation by getting the regions list:
$ aws ec2 describe-regions
If you created your AWS account after 2013-12-04, then only EC2-VPC is supported. In this
case, a default VPC is created in each AWS region. Therefore, unless you create your own VPC
and specify it when you launch an instance, your instances are launched in your default VPC.
If you created your AWS account before 2013-03-18, then both EC2-Classic and EC2-VPC are
supported in the regions you used before, and only EC2-VPC in regions that you didn't use.
In this case, a default VPC is created in each region in which you haven't created any AWS
resources. Therefore, unless you create your own VPC and specify it when you launch an
instance in a region (that you haven't used before), the instance is launched in your default
VPC for that region. However, if you launch an instance in a region that you've used before,
the instance is launched in EC2-Classic.
In this recipe, we will launch EC2 instances using EC2-Classic and EC2-VPC.
Getting started
Before we launch the EC2 instances, we need the image ID.
Run the following command to get the list of images. We can apply the filter to identify a
specific image. Record the image ID for later use:
$ aws ec2 describe-images
--filter [Filter]
By executing the following command, you obtain the image ID of a 64-bit version of Ubuntu
12.04 image:
$ aws ec2 describe-images
--filter
"Name=virtualization-type,Values=paravirtual"
"Name=root-device-type,Values=ebs" "Name=architecture,Values=x86_64"
"Name=name,Values=ubuntu/images/ebs/ubuntu-precise-12.04-amd64server-20130204"
How to do it
We will see the EC2 instances being launched, one by one:
10
Chapter 1
The parameters used in this command are described as follows:
The following command will create a micro instance in EC2-Classic (in the Singapore region):
$ aws ec2 run-instances
--image-id ami-7e2c612c
--count 1
--instance-type t1.micro
--key-name WebServerKeyPair
--security-group-ids sg-ad70b8c9
Here, SubnetId specifies the subnet where you want to launch your instance.
Next, run the following command to create a micro instance in EC2-VPC (in the
Singapore region):
$ aws ec2 run-instances
--image-id ami-7e2c612c
--count 1
--instance-type t1.micro
--key-name WebServerKeyPair
--security-group-ids sg-ad70b8c8
--subnet-id subnet-aed11acb
11
See also
The Configuring security groups and Creating an EC2 key pair recipes
How to do it
For allocating EIP addresses, perform the following steps:
1. Run the following command to allocate the EIP:
$ aws ec2 allocate-address
--domain [Domain]
You have to specify whether domain is standard or VPC. Record the allocation
ID for further use.
Domain value indicates whether the EIP address is used with instances in
EC2-Classic (standard) or instances in a EC2-VPC (VPC).
2. Next, run the following command to create the EIP in VPC:
$ aws ec2 allocate-address --domain vpc
3. Then, run the following command to associate the EIP to the Elastic Network
Interface (ENI):
$ aws ec2 associate-address
--network-interface-id [NetworkInterfaceId]
--allocation-id [AllocationId]
You need to provide the network interface ID of the ENI and allocation ID of the EIP
you obtained in step 1. If you don't specify the private IP address, then the Elastic
IP address is associated with the primary IP address.
12
Chapter 1
The parameters used in this command are described here:
See also
The Creating an instance with multiple NIC cards and a static private IP address recipe
13
How to do it
Creating an instance with multiple NIC cards requires us to create a network interface, attach
it to an instance, and finally associate the EIP to the ENI.
network interface
security groups
Syntax:
PrivateIpAddress=string,Primary=boolean
2. Next, run the following command to create the ENI with private IP addresses
10.0.0.26 and 10.0.0.27:
$ aws ec2 create-network-interface
--subnet-id subnet-aed11acb
--groups sg-ad70b8c8
--private-ip-addresses PrivateIpAddress=10.0.0.26,Primary=true Pri
vateIpAddress=10.0.0.27,Primary=false
Chapter 1
The parameters used in this command are described as follows:
[DeviceIndex]: This parameter provides the index of the device for the network
interface attachment
Then, run the following command to attach the ENI to the EC2 instance:
$ aws ec2 attach-network-interface
--network-interface-id eni-5c88f739
--instance-id i-2e7dace3
--device-index 1
[AllocationId]: This gives the allocation ID of EIP, which is required for EC2-VPC
See also
Encryption is also possible while using the EBS volumes. Encryption is done for data at rest,
data in transit, and disk I/O. Using encrypted EBS volumes have a minor effect on I/O latency,
but the performance remains the same. To encrypt EBS volume, you just need to select the
Encrypt this volume checkbox when creating EBS volume from AWS console. In this recipe,
we list the commands for creating an EBS volume, and then attaching it to an EC2 instance.
16
Chapter 1
How to do it
Run the following command to list the availability zones in a selected region. If the command is
run in the ap-southeast-1 region, you get the list of availability zones in the Singapore region.
$ aws ec2 describe-availability-zones
[VolumeType]: This gives the volume type. This can be gp2 for General
Purpose (SSD) volumes, io1 for Provisioned IOPS (SSD) volumes, or standard
for Magnetic volumes.
[IOPS]: This is only valid for Provisioned IOPS (SSD) volumes. This parameter
Use the following command to create a 90 GiB Provisioned IOPS (SSD) volume with 1000
Provisioned IOPS in availability zone ap-southeast-1b:
$ aws ec2 create-volume
--availability-zone ap-southeast-1b
--volume-type io1
--iops 1000
--size 90
17
[Device]: This one is used to mention the device name to expose to the instance
(for example, /dev/sdh or xvdh)
Run the following command to attach the EBS volume to an EC2 instance as /dev/sdf:
$ aws ec2 attach-volume
--volume-id vol-64e54f6a
--instance-id i-2e7dace3
--device /dev/sdf
How to do it
Using the create-tags command, you can create tags for one or more AWS resources.
18
Chapter 1
[Resources]: This parameter is used to provide the IDs of one or more resources
to tag
Syntax:
Key=KeyName,Value=ValueToAssign
The following command creates the Name and Group tag with its associated value for the EC2
instance (i-2e7dace3):
$ aws ec2 create-tags
--resources i-2e7dace3
--tags
Key=Name,Value=Tomcat Key=Group,Value='FronEnd Server Group'
How to do it
In this recipe, we first list the commands for creating a security group for EC2-Classic and EC2VPC. Then, we see how to create inbound and outbound rules. Finally, we list the command for
adding the security group to an instance.
Next, run the following command to create a security group with the
WebServerSecurityGroup name in EC2-Classic:
$ aws ec2 create-security-group
--group-name WebServerSecurityGroup
--description "Web Server Security Group"
20
Chapter 1
The parameters used in this command are described as follows:
Next, run the following command to create the inbound rule that allows SSH traffic from
IP address 123.252.223.114 in the security group (sg-c6b873a3):
$ aws ec2 authorize-security-group-ingress
--group-id sg-c6b873a3
--protocol tcp
--port 22
--cidr 123.252.223.114/32
21
Then, run the following command to create the outbound rule that allows MySQL traffic from
your instance to IP address 123.252.223.114 in the security group (sg-c6b873a3):
$ aws ec2 authorize-security-group-egress
--group-id sg-c6b873a3
--protocol tcp
--port 3866
--cidr 123.252.223.114/24
[SecurityGroupIds]: This option provides the IDs of one or more security groups
22
Chapter 1
Then, run the following command to add the security groups sg-c6b873a3 and sg-ccb873a9
to EC2 instance i-2e7dace3:
$ aws ec2 modify-instance-attribute
--instance-id i-2e7dace3
--groups sg-c6b873a3 sg-ccb873a9
How to do it
Here, we list the commands to create a key pair and then launching the EC2 instance (using
the key pair).
23
2. After executing the create-key-pair command, copy the entire output key into file
including the following lines:
----BEGIN RSA PRIVATE KEY--------END RSA PRIVATE KEY-----
How to do it
In order to group EC2 instances using placement groups, first we create a placement group,
and then add our EC2 instances in it.
24
Chapter 1
Here, the GroupName parameter specifies a name for the placement group and the
Strategy parameter specifies the placement strategy.
Next, run the following command to create a placement group with the name
WebServerGroup:
$ aws ec2 create-placement-group
--group-name WebServerGroup
--strategy cluster
[ImageId]: This gives the ID of the image from which you want to create the
EC2 instance
[KeyPairName]: This parameter provides the key pair name for the authentication
[SubnetId]: This option provides the ID of the subnet where you want to launch
your instance
25
Syntax:
--placement AvailabilityZone=value,GroupName=value,Tenancy=value
Next, execute the following command to launch a c3.large EC2 instance in the
WebServerGroup placement group:
$ aws ec2 run-instances
--image-id ami-7e2c612c
--count 1
--instance-type c3.large
--key-name WebServerKeyPair
--security-group-ids sg-ad70b8c8
--subnet-id subnet-aed11acb
--placement GroupName= WebServerGroup
26
Chapter 1
How to do it
Here, we list the commands for creating an ELB, configuring the same for performing health
checks, and finally associating specific EC2 instances with it.
[SubnetIds]: This option gives a list of subnet IDs in your VPC to attach to
your load balancer. You can get a list of subnet IDs by running the aws ec2
describe-subnets command.
27
Syntax:
Target=HTTP:8080/index.html,Interval=30,UnhealthyThreshold=
2,HealthyThreshold=2,Timeout=3
The following command will add the health check configuration to an ELB. The ELB checks
the instance health at <URL>:8080/index.html. ELB health check interval is set to 30
seconds. UnhealthyThreshold specifies the number of consecutive unsuccessful URL
probes before the ELB changes the instance health status to unhealthy. HealthyThreshold
specifies the number of consecutive successful URL probes before ELB changes the instance
health status to healthy.
$ aws elb configure-health-check
--load-balancer-name WebLoadBalancer
--health-check Target=HTTP:8080/index.html,Interval=30,UnhealthyThreshold
=2,HealthyThreshold=2,Timeout=3
[Instances]: This option gives a list of instances for the load balancer
28
Chapter 1
The following command will add ELB to EC2 instances with IDs i-d3ff2c1e and
i-2e7dace3.
$ aws elb register-instances-with-load-balancer
--load-balancer-name WebLoadBalancer
--instances i-d3ff2c1e i-2e7dace3
How to do it
Here, we list the commands required for configuring high availability across two different
regions using Route 53:
1. Create an instance in the first region. Before launching the EC2 instance, create the
required VPC, subnets, key pairs, and security groups in this region.
$ aws ec2 run-instances
--image-id [ImageId]
--count [InstanceCount]
--instance-type [InstanceType]
--key-name [KeyPairName]
--security-group-ids [SecurityGroupIds]
--subnet-id [SubnetId]
29
to create
authentication
[SubnetId]: This parameter provides the ID of the subnet where you want
30
Chapter 1
The parameters used in this command are described as follows:
4. Create health checks for previously created instances in the first region by performing
the following steps:
1. First create a virginiahc.json file with the following JSON. The IP
address used is the public IP address of EC2 instance.
{
"IPAddress":"54.173.200.169",
"Port":8080,
"Type":"HTTP",
"ResourcePath":"/index.html",
"RequestInterval":30,
"FailureThreshold":3
}
Syntax:
file://virginiahc.json
31
5. Create health checks for previously created instances in second region by performing
the following steps:
1. Create a second singaporehc.json file with the following JSON. The IP
address used is the public IP address of EC2 instance.
{
"IPAddress":"54.169.85.163",
"Port":8080,
"Type":"HTTP",
"ResourcePath":"/index.html",
"RequestInterval":30,
"FailureThreshold":3
}
Syntax:
file:// singaporehc.json
3. Create health check by running the following command. Record the health
check ID for further usage.
$ aws route53 create-health-check
--caller-reference 2014-11-29-17:04
--health-check-config file://singaporehc.json
32
Chapter 1
6. Add a primary and secondary record set to the Route 53-hosted zone by performing
the following steps:
1. Create a recordset.json file with the following JSON. In primary record
set, replace health check ID and IP address with first region health check
ID and EC2 public IP address accordingly. In secondary record set, replace
health check ID and IP address with second region health check ID and EC2
public IP address accordingly.
{
"Comment":"CreatingRecordSet",
"Changes":[
{
"Action":"CREATE",
"ResourceRecordSet":{
"Name":"DNS Domain Name",
"Type":"A",
"SetIdentifier":"PrimaryRecordSet",
"Failover":"PRIMARY",
"TTL":300,
"ResourceRecords":[
{
"Value":"54.173.200.169"
}
],
"HealthCheckId":"<your first region's
health check id>"
}
},
{
"Action":"CREATE",
"ResourceRecordSet":{
"Name":" DNS Domain Name",
"Type":"A",
"SetIdentifier":"SecondaryRecordSet",
"Failover":"SECONDARY",
"TTL":300,
"ResourceRecords":[
{
"Value":"54.169.85.163"
}
],
"HealthCheckId":"<your second region's
health check id>"
}
}
]
}
33
Syntax:
file://recordset.json
3. Add the record set to the hosted zone by running the following command:
$ aws route53 change-resource-record-sets
--hosted-zone-id Z3DYG8V5Z07JP8
--change-batch file://recordset.json
7.
Test the failover configuration by stopping the server in the primary region. You can stop
your first region EC2 instance by running the aws ec2 stop-instances command.
How to do it
Here we list the commands for creating AMIs for offering them to other users on
AWS Marketplace.
Chapter 1
--instance-id [InstanceId]
--name [Name]
--description [Description]
The following command creates an image of the EC2 instance with ID i-2e7dace3:
$ aws ec2 create-image
--instance-id i-2e7dace3
--name "WebServerImage"
--description "Image of web server"
Syntax:
"{\"Add\": [{\"Group\":\"all\"}]}"
35
www.PacktPub.com
Stay Connected: