Module 5

Cloud Platforms in Industry

Syllabus
Cloud Platforms in Industry: Amazon Web Services, Google AppEngine, Microsoft
Azure, Cloud Applications: Scientific Applications, Business and Consumer
Applications.
Introduction
Cloud computing allows end users and developers to use large distributed
computing systems.

It is possible due to infrastructure management software and distributed

computing platforms.

Cloud platforms provide on-demand compute, storage, and other advanced

services.

These services help in building enterprise cloud applications or extending

industrial applications.

There are multiple cloud computing platforms available, each offering different
types of services.
A cloud system can be developed using:
● A single technology/vendor or
● A combination of multiple technologies/vendors.
Amazon web services
● Amazon Web Services (AWS) is a cloud platform that helps develop flexible
applications.
● It provides scalable infrastructure, messaging, and data storage
solutions.
● AWS can be accessed using SOAP or RESTful web service interfaces.
● It offers a web-based console for managing resources, monitoring usage,
and tracking costs on a pay-as-you-go basis.
Key AWS Services

1. Core Services (Compute & Storage)

● Amazon EC2 (Elastic Compute Cloud): Provides raw computing power.
● Amazon S3 (Simple Storage Service): Offers scalable cloud storage.

2. Additional Compute & Scaling Services

● Elastic MapReduce (EMR): Processes big data using Hadoop.
● AutoScaling: Adjusts computing power based on demand.

3. Data Management Services

● Elastic Block Store (EBS): Stores persistent data.
● Amazon SimpleDB: A NoSQL database for lightweight applications.
● Amazon RDS (Relational Database Service): Managed relational database
service.
● Amazon ElastiCache: Provides caching for faster data access.
4. Networking Services
● Amazon VPC (Virtual Private Cloud): Creates isolated cloud networks.
● Elastic Load Balancing: Distributes traffic to multiple servers.
● Amazon Route 53: Manages domain name system (DNS).
● Amazon Direct Connect: Establishes private network connections to AWS.
5. Communication Services
● Amazon SQS (Simple Queue Service): Manages message queues between
applications.
● Amazon SNS (Simple Notification Service): Sends notifications.
● Amazon SES (Simple Email Service): Sends bulk emails reliably.
6. Other Advanced Services
● Amazon CloudFront: Content delivery network for faster web content
access.
● Amazon CloudWatch: Monitors AWS services and resources.
● Amazon Elastic Beanstalk & CloudFormation: Simplifies application
deployment.
Compute services
● Compute services are the core of cloud computing systems.
● The main compute service in AWS is Amazon EC2 (Elastic Compute
Cloud).
● EC2 provides an Infrastructure as a Service (IaaS) solution.
● It serves as a reference model for similar cloud services from other vendors.
● EC2 allows users to deploy virtual machines (VMs) called instances.
Instances are based on preconfigured images that include:
● An operating system
● A software stack

Users can customize instances by choosing:

● Memory size
● Number of processors
● Storage capacity

After deployment, users receive credentials to remotely access the instance.

Users can then further configure the instance or install additional software as
needed.
Amazon Machine Images (AMIs)
● Amazon Machine Images (AMIs) are templates used to create virtual
machines in AWS.
● AMIs are stored in Amazon S3 and have a unique identifier (e.g., ami-
xx.xxxx).
● Each AMI includes a predefined operating system and file system layout.
● AMIs contain:
○ Amazon Ramdisk Image (ARI, id: ari-yyyyyy)
○ Amazon Kernel Image (AKI, id: aki-zzzzzz)
AMIs can be created in two ways:
● From scratch (manually setting up the OS and configurations).
● By bundling an existing EC2 instance (modifying an instance and saving it
as an AMI).

A common approach is:

● Create an EC2 instance from an AMI.
● Log in and install the necessary software.
● Convert it into a new AMI using AWS tools.
Once created, an AMI is stored in an S3 bucket.

Users can choose to:

● Keep the AMI private.
● Share it with other AWS users.

AMIs can have an associated product code, allowing the owner to earn
revenue whenever someone uses the AMI to launch an EC2 instance.
EC2 Instances (Virtual Machines in AWS)
EC2 instances are virtual machines created using Amazon Machine Images
(AMIs) as templates.

Users can customize instances by selecting:

● Number of cores (processors).
● Computing power (measured in EC2 Compute Units - ECUs).
● Installed memory (RAM).

ECUs (EC2 Compute Units) measure the computing power of a virtual core.
● One ECU is equal to a 1.0–1.2 GHz 2007 Opteron or Xeon processor.
● ECUs help AWS keep performance consistent even when hardware is
upgraded over time.
 1. Instead of buying hardware, users rent computing power, making cloud
computing more flexible and scalable.

Major Categories of EC2 Instances

AWS offers six major categories of EC2 instances. Here are three key types:

2. Standard Instances:
○ Suitable for most applications.
○ Comes in three levels, each offering higher computing power,
storage, and memory.
3. Micro Instances:
○ Designed for low-power applications.
○ Can handle occasional CPU bursts for unexpected workload surges.
○ Ideal for small web applications with limited traffic.
High-Memory Instances:
● Designed for applications needing large memory capacity.
● Suitable for high-traffic, three-tier web applications.
● Available in three levels, with memory increasing proportionally to CPU
power.

High-CPU Instances:
● Designed for compute-intensive applications.
● Available in two configurations, where computing power increases more
than memory.
● Suitable for tasks requiring high processing power but not much memory.
Cluster Compute Instances:
● Designed for virtual cluster services.
● Features high CPU power, large memory, and extremely high I/O &
network performance.
● Best for High-Performance Computing (HPC) applications that need fast
processing and data transfer.

Cluster GPU Instances are designed for applications that require high-
performance graphics processing.

These instances come with Graphics Processing Units (GPUs) along with:
● High computing power
● Large memory
● Extremely high I/O and network performance
They are ideal for graphic-intensive tasks such as:
● Rendering clusters (e.g., animations, simulations).
● High-Performance Computing (HPC) applications.

Since GPUs can be used for general-purpose computing (GPGPU), users can
harness extra computing power for complex calculations beyond graphics.
EC2 Environment (Virtual Machine Setup in AWS)
EC2 instances run in a virtual environment that provides essential services
such as:
● Allocating IP addresses
● Attaching storage volumes
● Configuring security settings (access control, network connectivity)

IP Addressing in EC2:
● By default, instances get an internal IP address (for communication within
the EC2 network).
● Instances can access the Internet as clients using this internal IP.
● To make an instance publicly accessible, an Elastic IP (EIP) can be assigned.
● Elastic IPs are flexible: they can be reassigned to different instances over
time, allowing failover capabilities.
Domain Naming System (DNS) in EC2:
● Each instance gets a domain name (e.g., ec2-xxx-xxx-xxx.compute-
x.amazonaws.com).
● The xxx-xxx-xxx part represents the external IP address.
● The compute-x part indicates the AWS availability zone where the instance
is deployed.

AWS Availability Zones & Pricing:

● There are currently five major availability zones with different pricing:
○ United States: Virginia, Northern California
○ Europe: Ireland
○ Asia-Pacific: Singapore, Tokyo
Advanced Compute Services in AWS
Amazon Web Services (AWS) offers advanced services on top of EC2 instances and
AMIs to simplify application deployment and execution.

1. AWS CloudFormation

● Extends the simple EC2 deployment model.

● Uses JSON-formatted templates to define the resources needed for an
application.
● Helps in linking EC2 instances together and defining dependencies among
them.
● Allows easy integration of EC2 instances with other AWS services like:
○ S3 (Storage)
○ SimpleDB (Database)
○ SQS, SNS (Messaging Services)
○ Route 53 (DNS Management)
○ Elastic Beanstalk (Application Deployment)
AWS Elastic Beanstalk
● A simplified way to package and deploy applications on AWS Cloud.
● Automates instance provisioning and application deployment.
● Currently supports Web applications built using Java/Tomcat.
● Developers can package their Java Web App as a WAR file and deploy it
easily using Beanstalk.
Amazon Elastic MapReduce (EMR) & AWS Storage Services

Amazon Elastic MapReduce (EMR)

● Provides a cloud computing platform for MapReduce applications.
● Uses Hadoop as the MapReduce engine.
● Runs on EC2 instances and uses Amazon S3 for data storage.
● Helps process large-scale datasets efficiently in a distributed environment.
AWS Storage Services

AWS offers various storage and information management services.

Amazon Simple Storage Service (S3)

● A distributed object storage service.
● Stores data in different formats and is highly scalable.
● Two key components:
○ Buckets → Virtual containers that store objects.
○ Objects → Actual stored data (can have metadata for tagging and
additional information)
Amazon S3
Simple Storage with REST Interface
● Amazon S3 acts as a distributed file system but with key differences for
efficiency.
● Accessed through a REST API.

Two-Level Storage Hierarchy

● Storage is organized into "buckets", which cannot be partitioned further.
● No directories, but logical groupings can be simulated using object names.

Immutable Objects
● Once stored, objects cannot be modified, renamed, or relocated.
● To update an object, it must be deleted and re-uploaded.
Eventual Consistency
● S3 uses replication for redundancy, meaning changes take time to reflect
globally.
● Large objects may experience delays in availability across different regions.

Occasional Request Failures

● Due to distributed infrastructure, requests may sometimes fail but are
designed to be retried.
Resource naming
Amazon S3 URL Structure
● Amazon S3 provides three ways to address a bucket and its objects using a
REST interface and URLs.

Bucket Addressing Forms

● Canonical Form:
○ URL format: http://s3.amazonaws.com/bucket_name
○ Bucket name is part of the path in the URL.
○ Fewer restrictions on allowed characters in the name.
Subdomain Form:

● URL format: http://bucket_name.s3.amazonaws.com/

● This is the preferred method, especially for better compatibility across Amazon's
servers.
● Bucket names must meet these rules:
○ Between 3 and 63 characters long
○ Can only include letters, numbers, periods, and dashes
○ Start with a letter or number
○ Contain at least one letter
○ No fragment between periods starting or ending with a dash, or empty
segments.

Virtual Hosting Form:

● URL format: http://bucket-name.com/

● Custom URLs through DNS configuration (CNAME record pointing to the
subdomain form).
Object Addressing
● Objects in S3 are part of a bucket and always referred to in the URL:

● Canonical Form:

○ URL format: http://s3.amazonaws.com/bucket_name/object_name

● Subdomain Form:

○ URL format: http://bucket-name.s3.amazonaws.com/object_name

● Virtual Hosting Form:

○ URL format: http://bucket-name.com/object_name

Object-Specific Information
● Certain parameters can be added to the URL to access specific details about
objects:
○ Object ACL (Access Control List):
■ URL format:
http://s3.amazonaws.com/bucket_name/object_name?acl
○ Bucket Server Logging:
■ URL format: http://s3.amazonaws.com/bucket_name?logging

Metadata
● Object metadata cannot be directly accessed through a URL.
● Metadata is manipulated by adding attributes to the request in the URL, not
part of the identifier.
Buckets
● A bucket is a container for storing objects (files) in Amazon S3.
● Think of it like a virtual drive or storage space.
● Buckets are top-level elements, meaning they cannot be nested or have sub-
buckets.

Geographic Location & Replication

● Every bucket is created in a specific geographic location (data center region).

● Buckets are replicated for fault tolerance (to prevent data loss) and better content
delivery.
● You can choose the location when creating the bucket, but by default, it’s
created in Amazon's U.S. data centers.
● All objects in the bucket are stored in the same availability zone as the bucket.
Creating a Bucket

● To create a bucket, send a PUT request to: http://s3.amazonaws.com/

● You must provide:
○ The bucket name
○ Optionally, you can specify the availability zone (location).

Listing Bucket Contents

● To view the content inside a bucket, send a GET request specifying the bucket name.

Bucket Name Limitations

● Once created, a bucket’s name cannot be changed.

● A bucket cannot be relocated.
● If you need to change its name or location, you must delete the old bucket and create
a new one.
Deleting a Bucket
● To delete a bucket, send a DELETE request.
● The deletion will only succeed if the bucket is empty. You must remove all
objects inside the bucket before deleting it.

Objects and metadata

● An object is the content you store in Amazon S3, like files or data.
● Each object is identified by a unique name within the bucket where it is
stored.
● The object name can be up to 1,024 bytes in UTF-8 encoding and allows
almost any character, including characters normally used as path separators.
● Since buckets don't support nesting, directories can be emulated by naming
objects in a certain way (e.g., using slashes / in the name).
Creating an Object

● To store an object, you send a PUT request that specifies:

○ The object name
○ The bucket name
○ The contents of the object
○ Optional additional properties (like metadata)
● The maximum object size is 5 GB.
● Once created, an object cannot be:
● Modified
● Renamed
● Moved to another bucket
Retrieving and Deleting Objects

● To retrieve an object, you send a GET request.

● To delete an object, you send a DELETE request.

Object Metadata

● Metadata are key-value pairs associated with an object.

○ System-defined metadata: Managed by S3 to control how it interacts with the
object.
○ User-defined metadata: Custom metadata meaningful to the user (e.g., tags,
descriptions).
● Maximum metadata size: 2 KB per key-value pair.
● Metadata can be retrieved in two ways:
○ GET request (gets both the object content and metadata).
○ HEAD request (gets only the metadata without the content)
Advanced features
Server Access Logging

● What it is: Server access logging tracks detailed information about the requests
made to a bucket and its objects (e.g., who accessed the objects, when, and from
where).
● How to enable: By default, this feature is off. You can turn it on by sending a PUT
request to the bucket URI (with /logging at the end).
● Required information:
○ The request must include an XML file specifying:
■ The target bucket where logs will be saved.
■ The file name prefix for the log files.
● Retrieve logging configuration: To view the current logging settings, send a
GET request to the same bucket URI.
Integration with BitTorrent
● What it is: S3 objects can be exposed to the BitTorrent file-sharing
network. This means files stored in S3 can be downloaded using the
BitTorrent protocol.
● How to enable: You can expose an S3 object to BitTorrent by adding torrent
to the end of the object’s URI (e.g.,
http://s3.amazonaws.com/bucket_name/object_name?torrent).
● Permission required: For users to download the object via BitTorrent, the
Access Control Policy (ACP) of the object must allow read permission to
everyone.
Amazon elastic block store
● Amazon Elastic Block Store (EBS) provides persistent storage for EC2 instances.
● It allows data to be stored even after the instance is stopped or terminated.

Storage & Access

● Each EBS volume can have up to 1 TB of storage.

● They are accessed through a block device interface, which means they can be
formatted as needed (raw storage, file system, etc.).

Data Persistence & Backup

● EBS volumes persist data even if the instance is stopped.

● Their content is stored in Amazon S3 for durability.
● Snapshots (backups) can be taken and stored incrementally.
Performance & Availability

● EBS volumes usually reside in the same availability zone as the EC2 instance
using them, ensuring high performance.
● Volumes from different availability zones can also be connected if needed.

Usage & Modifications

● Volumes are lazily loaded, meaning data is fetched in the background as needed.
● Multiple EBS volumes can be attached to a single EC2 instance.
● A volume can be created from a snapshot and its size modified (if the file system
supports resizing).

Limitations

● A volume image cannot be shared among multiple instances.

● However, separate active volumes can be created from a snapshot and used
independently.
Amazon ElastiCache
● ElastiCache is an in-memory caching service based on a cluster of EC2
instances.
● It provides fast data access for other EC2 instances.

Compatibility & Migration

● It supports a Memcached-compatible protocol, meaning existing applications

using Memcached can migrate to ElastiCache without modification.

How It Works

● Runs caching software on EC2 instances.

● Accessible through Web services.
● Can be dynamically resized based on demand.
Management & Reliability

● Automatic patching, failure detection, and recovery ensure the cache remains
available without user intervention.
● Users only need to adjust cluster size when necessary.

Pricing

● Follows the EC2 pricing model with a small additional cost for the caching
service.
● Different instance types are available based on performance needs.
Structured Storage Solutions in AWS
Many enterprise applications use databases to store and analyze data.
Traditionally, Relational Database Management Systems (RDBMS) have been
widely used, but more scalable and lightweight alternatives are now available.

AWS offers three structured storage solutions:

1. Preconfigured EC2 AMIs

2. Amazon Relational Database Service (RDS)
3. Amazon SimpleDB (not detailed in the provided content)
Preconfigured EC2 AMIs
● EC2 AMIs (Amazon Machine Images) are preconfigured templates with a
pre-installed database system.
● Users can launch EC2 instances from these AMIs and attach an EBS
volume for persistent storage.
● Available database systems include:
○ IBM DB2, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, Sybase,
and Vertica.
● Pricing is based on EC2’s hourly cost model.
● Limitations:
○ Users are responsible for configuring, maintaining, and managing the
database.
○ Offers more flexibility, but requires technical expertise.
Amazon Relational Database Service (RDS)
● RDS is a managed relational database service that runs on EC2
infrastructure.
● AWS handles:
○ High availability & failover strategies.
○ Automatic backups, snapshots, and point-in-time recovery.
○ Database software updates and maintenance.
● Available relational database engines:
○ MySQL
○ Oracle
Key Features of RDS

1. Multi-AZ Deployment (High Availability)

○ Keeps standby synchronized copies in different availability zones.

○ If the primary database fails, the standby copy takes over.
2. Read Replicas (Performance Optimization)

○ Used for read-heavy applications.

○ Creates read-only copies of the database to reduce response times.
Amazon SimpleDB

Amazon SimpleDB is a lightweight, scalable, and flexible data storage solution for
applications that do not require a fully relational database.

Key Features

1. Supports Semi-Structured Data

○ Uses a model based on domains, items, and attributes instead of traditional

tables.
○ Offers fewer constraints on data structure, improving query performance for large
datasets.
2. Ease of Use

○ Unlike Amazon RDS, SimpleDB does not require configuration or management

by the user.
○ AWS handles availability and scaling automatically.
Domain-Based Storage

● Domains in SimpleDB are like tables in relational databases.

● However, each item can have a different structure (flexible schema).
● Data is stored in key-value pairs.

Scalability Limits

● Each domain can store up to 10 GB of data.

● By default, users can create up to 250 domains.

Operations Supported

● Users can create, delete, modify, and query domains.

● Supports batch operations (bulk insertion and deletion).
● Querying supports operators like:
○ <, >, <>, LIKE, NOT LIKE, BETWEEN, IS NULL, IS NOT NULL.
● Example Query:
sql
SELECT * FROM domain_name WHERE every(attribute_name) = 'value'
Amazon CloudFront

Amazon CloudFront is a Content Delivery Network (CDN) that helps speed up

the delivery of web content using a globally distributed network of edge
servers.

How It Works

1. Uses Edge Servers

○ CloudFront caches and serves content from servers closest to the
user.
○ This reduces transfer time and improves website performance.
2. Setting Up CloudFront
○ Users create a distribution that references an origin server (where the
content is stored).
○ The content is made available via a CloudFront URL (e.g., my-
distribution.Cloudfront.net).
 ○ Custom domain names can also be mapped to a distribution.
1. Content Delivery

○ When a request is made, CloudFront checks its cache:

■ If content is available, it serves it from the nearest edge server.
■ If content is missing or expired, CloudFront fetches it from the origin server.
○ Supports static content (HTTP/HTTPS) and streaming media (RTMP - Real-Time
Messaging Protocol).

Compatible Origin Servers

● CloudFront can fetch content from:
○ Amazon S3 buckets
○ EC2 instances
○ External servers (outside AWS).

Access Control & Updates

● Users can restrict access to specific protocols.
● Content can be invalidated to force an update before expiration
Compatible Origin Servers

● CloudFront can fetch content from:

○ Amazon S3 buckets
○ EC2 instances
○ External servers (outside AWS).

Access Control & Updates

● Users can restrict access to specific protocols.

● Content can be invalidated to force an update before expiration
Communication Services in AWS

Amazon provides communication services to help applications and services

interact efficiently within the AWS infrastructure. These services are categorized
into Virtual Networking and Messaging.

1. Virtual Networking

Virtual networking allows users to manage connectivity between compute and

storage services in AWS. The key services include:
● Amazon Virtual Private Cloud (VPC)
○ Helps create private networks within AWS.
○ Provides pre-built templates for different network setups, such as:
■ Public subnets (accessible via the internet).
■ Private networks (secured using Network Address Translation -
NAT).
 ■ Hybrid networks (combining AWS and private resources).
○ Allows connectivity control between services (like EC2 and S3) using
IAM.
○ Cost (in 2011): $0.50 per connection hour.
● Amazon Direct Connect
○ Provides dedicated network connections between AWS and on-premises
infrastructure.
● Amazon Route 53
○ AWS’s Domain Name System (DNS) service.
○ Helps connect AWS resources by managing domain names.

2. Messaging Services

Messaging services help applications communicate asynchronously by sending

messages. There are three main messaging services in AWS:
● Amazon Simple Queue Service (SQS)
○ A message queue system that enables applications to send and receive
messages.
○ Messages are securely stored and can be accessed by authorized
applications.
○ Messages are locked while being read to prevent duplicate processing.

Amazon Simple Notification Service (SNS)

● A publish-subscribe system that notifies applications when new content is
available.
● Unlike SQS (which requires constant message polling), SNS notifies
subscribers automatically.
● Supports multiple notification methods like HTTP/HTTPS, email, and SQS.
Amazon Simple Email Service (SES)
● A scalable email-sending service.
● Users verify their email addresses before sending emails.
● Emails can be sent using SMTP or custom formats (MIME).
● Provides statistics on email delivery performance.

Additional AWS Services

Apart from compute, storage, and communication services, AWS offers

additional services to help users manage applications and transactions. Two
key services are Amazon CloudWatch and Amazon Flexible Payment
Service (FPS).

1. Amazon CloudWatch
● Monitors and provides detailed statistics about AWS services.
● Collects data from EC2, S3, SimpleDB, CloudFront, and other AWS
services.
● Helps developers analyze usage patterns, performance, and costs to
optimize applications.
● Initially required a subscription but is now free for all AWS users.

2. Amazon Flexible Payment Service (FPS)

● Allows AWS users to sell goods and services using Amazon’s billing
system.
● Eliminates the need for separate payment processing systems.
● Supports various payment models:
○ One-time payments
○ Delayed payments
○ Recurring payments (for subscriptions or usage-based services)
○ Aggregated payments (combining multiple small payments into one)
Google AppEngine
● It is a Platform as a Service (PaaS) that helps developers build and host
scalable web applications.
● It automatically allocates computing resources when applications face high
traffic.

How does it work?

AppEngine is built on Google's distributed infrastructure, which ensures
reliability and scalability.
● It balances the load by distributing requests across multiple servers.
Supported Programming Languages
● Developers can create applications using Java, Python, and Go (a language
developed by Google).

Billing and Resource Usage

● AppEngine tracks how much of Google's resources (like computing power
and storage) an application uses.
● Users are billed only if they exceed the free quota provided by Google.
AppEngine Architecture & Core Components

AppEngine is logically divided into four key parts:

1. Infrastructure – The base system that ensures the platform runs efficiently.
2. Runtime Environment – Provides the necessary tools to execute
applications in supported languages.
3. Storage – The underlying database and file storage system for storing
application data.
4. Scalable Services – A set of tools that help developers design applications
that can scale easily.

This structure allows developers to build applications that can automatically

handle large numbers of users without worrying about managing servers
Microsoft Azure
SQL Azure
Cloud Applications
Business and consumer applications