UNIT V

CLOUD TECHNOLOGIES AND ADVANCEMENTS

Hadoop - MapReduce - Virtual Box -- Google App Engine - Programming

Environment for Google App Engine -- Open Stack - Federation in the Cloud
- Four Levels of Federation - Federated Services and Applications - Future of
Federation.

5.1 HADOOP
• Hadoop is an Apache open source framework written in java that allows
distributed processing of large datasets across clusters of computers using
simple programming models.
• The Hadoop framework application works in an environment that provides
distributed storage and computation across clusters of computers.
• Hadoop is designed to scale up from single server to thousands of machines,
each offering local computation and storage.
5.1.1 Hadoop Architecture
At its core, Hadoop has two major layers namely “
• Processing/Computation layer (MapReduce), and
• Storage layer (Hadoop Distributed File System).
 MapReduce
(Distributed Computation)

(Distributed Storage)

Common

Figure 5.1 Hadoop Architecture

MapReduce

• MapReduce is a parallel programming model for writing distributed

applications devised at Google for efficient processing of large amounts of data
(multi-terabyte data-sets), on large clusters (thousands of nodes) of commodity
hardware in a reliable, fault-tolerant manner. The MapReduce program runs
on Hadoop which is an Apache open-source framework.
Hadoop Distributed File System
• The Hadoop distributed file system (HDFS) is a distributed, scalable, and
portable file system written in Java for the Hadoop framework.
• A Hadoop is divided into HDFS and MapReduce. HDFS is used for storing
the data and MapReduce is used for the Processing the Data.
HDFS Architecture
HDFS has five services as follows:
1. Name Node
2. Secondary Name Node
3. Job tracker
4. Data Node
5. Task Tracker
 HDFS Architecture

Name Node
Read

Data Nodes Data Nodes

Rack 1 Rack 2
Figure 5.2 HDFS Architecture
• Top three are Master Services/Daemons/Nodes and bottom two are Slave
Services. Master Services can communicate with each other and in the same
way Slave services can communicate with each other. Name Node is a master
node and Data node is its corresponding Slave node and can talk with each
other.
• Name Node: HDFS consists of only one Name Node we call it as Master Node
which can track the files, manage the file system and has the metadata
and the whole data in it. To be particular Name node contains the details of the
No. of blocks, Locations at what data node the data is stored and where the
replications are stored and other details. As we have only one NameNode we
call it as Single Point Failure. It has Direct connect with the client.
• Data Node: A Data Node stores data in it as the blocks. This is also known as
the slave node and it stores the actual data into HDFS which is responsible for
the client to read and write. These are slave daemons. Every Data node sends
a Heartbeat message to the Name node every 3 seconds and conveys that it is
alive. In this way when Name Node does not receive a heartbeat from a data
node for 2 minutes, it will take that data node as dead and starts the process of
block replications on some other Data node.
• Secondary Name Node: This is only to take care of the checkpoints of the file
system metadata which is in the Name Node. This is also known as the
checkpoint Node. It is helper Node for the Name Node.
• Job Tracker: Basically Job Tracker will be useful in the Processing the data.
Job Tracker receives the requests for Map Reduce execution from the client.
Job tracker talks to the Name node to know about the location of the data like
Job Tracker will request the Name Node for the processing the data. Name
node in response gives the Meta data to job tracker.
• Task Tracker: It is the Slave Node for the Job Tracker and it will take the task
from the Job Tracker. And also it receives code from the Job Tracker. Task
Tracker will take the code and apply on the file. The process of applyingthat
code on the file is known as Mapper.
• Hadoop cluster has nominally a single namenode plus a cluster of datanodes,
although redundancy options are available for the namenode due to its
criticality. Each datanode serves up blocks of data over the network using a
block protocol specific to HDFS.
• The file system uses TCP/IP sockets for communication. Clients use remote
 procedure calls (RPC) to communicate with each other.
• HDFS stores large files (typically in the range of gigabytes to terabytes) across
multiple machines. It achieves reliability by replicating the data across multiple
hosts, and hence theoretically does not require redundant array of independent
disks (RAID) storage on hosts (but to increase input-output (I/ O) performance
some RAID configurations are still useful).
• With the default replication value, 3, data is stored on three nodes: two on the
same rack, and one on a different rack.
• Data nodes can talk to each other to rebalance data, to move copies around,
and to keep the replication of data high. HDFS is not fully POSIX-compliant,
because the requirements for a POSIX file-system differ from the target goals
of a Hadoop application. The trade-off of not having a fully POSIX- compliant
file-system is increased performance for data throughput and support for non-
POSIX operations such as Append.
• The HDFS file system includes a so-called secondary namenode, a
misleading term that some might incorrectly interpret as a backup namenode
when the primary namenode goes offline. In fact, the secondary namenode
regularly connects with the primary namenode and builds snapshots of the
primary namenode’s directory information, which the system then saves to
local or remote directories.
• These checkpointed images can be used to restart a failed primary namenode
without having to replay the entire journal of file-system actions, then to edit
the log to create an up-to-date directory structure. Because the namenode is the
single point for storage and management of metadata, it can become a
bottleneck for supporting a huge number of files, especially a large number of
small files.
• HDFS Federation, a new addition, aims to tackle this problem to a certain
extent by allowing multiple namespaces served by separate namenodes.
Moreover, there are some issues in HDFS such as small file issues, scalability
problems, Single Point of Failure (SPoF), and bottlenecks in huge metadata
requests.
• One advantage of using HDFS is data awareness between the job tracker and
task tracker. The job tracker schedules map or reduce jobs to task trackerswith
an awareness of the data location. For example: if node A contains data (a,
b, c) and node X contains data (x, y, z), the job tracker schedules node A to
 perform map or reduce tasks on (a, b, c) and node X would be scheduled to
perform map or reduce tasks on (x, y, z). This reduces the amount of traffic
that goes over the network and prevents unnecessary data transfer.
• When Hadoop is used with other file systems, this advantage is not always
available. This can have a significant impact on job-completion times as
demonstrated with data-intensive jobs.
• HDFS was designed for mostly immutable files and may not be suitable for
systems requiring concurrent write operations.
• HDFS can be mounted directly with a Filesystem in Userspace (FUSE) virtual
file system on Linux and some other Unix systems.
• File access can be achieved through the native Java API, the Thrift API
(generates a client in a number of languages e.g. C++, Java, Python, PHP,
Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml),
the command-line interface, the HDFS-UI web application over HTTP, or via
3rd-party network client libraries.
• HDFS is designed for portability across various hardware platforms and for
compatibility with a variety of underlying operating systems. The HDFS
design introduces portability limitations that result in some performance
bottlenecks, since the Java implementation cannot use features that are
exclusive to the platform on which HDFS is running.
• Due to its widespread integration into enterprise-level infrastructure,
monitoring HDFS performance at scale has become an increasingly important
issue.
• Monitoring end-to-end performance requires tracking metrics from
datanodes, namenodes, and the underlying operating system. There are
currently several monitoring platforms to track HDFS performance,
including Hortonworks, Cloudera, and Datadog.
Features of HDFS
• It is suitable for the distributed storage and processing.
• Hadoop provides a command interface to interact with HDFS.
• The built-in servers of namenode and datanode help users to easily check the
status of cluster.
• Streaming access to file system data.
 • HDFS provides file permissions and authentication.
Goals of HDFS
• Fault detection and recovery ” Since HDFS includes a large number of
commodity hardware, failure of components is frequent. Therefore HDFS
should have mechanisms for quick and automatic fault detection and recovery.
• Huge datasets ” HDFS should have hundreds of nodes per cluster to manage
the applications having huge datasets.
• Hardware at data ” A requested task can be done efficiently, when the
computation takes place near the data. Especially where huge datasets are
involved, it reduces the network traffic and increases the throughput.
• Hadoop framework also includes the following two modules

• Hadoop Common ” These are Java libraries and utilities required by other
Hadoop modules.
• Hadoop YARN ” This is a framework for job scheduling and cluster resource
management.
Hadoop Common
• Hadoop Common refers to the collection of common utilities and libraries that
support other Hadoop modules. It is an essential part or module of the Apache
Hadoop Framework, along with the Hadoop Distributed File System (HDFS),
Hadoop YARN and Hadoop MapReduce.
• Like all other modules, Hadoop Common assumes that hardware failures are
common and that these should be automatically handled in software by the
Hadoop Framework.
Hadoop Common is also known as Hadoop Core.
Hadoop YARN
• The fundamental idea of YARN is to split up the functionalities of resource
management and job scheduling/monitoring into separate daemons. The idea
is to have a global ResourceManager ( RM ) and per- application
ApplicationMaster (AM). An application is either a single job or a DAG of
jobs.
• The ResourceManager and the NodeManager form the data-computation
framework.
 • The ResourceManager is the ultimate authority that arbitrates resources among
all the applications in the system.
• The NodeManager is the per-machine framework agent who is responsible for
containers, monitoring their resource usage (cpu, memory, disk, network) and
reporting the same to the ResourceManager/Scheduler.
• The per-application ApplicationMaster is, in effect, a framework specific
library and is tasked with negotiating resources from the ResourceManager
and working with the NodeManager(s) to execute and monitor the tasks.

Node
Manager

Container App Mstr

Client

Resource Node
Manager Manager
Client

App Mstr Container

Node
MapReduce Status
Manager
Job Submission
Node Status
Resource Request Container Container

Figure 5.3 YARN Architecture

The ResourceManager has two main components: Scheduler and
ApplicationsManager.
i. Scheduler
Scheduler is responsible for allocating resources to various applications. This is
a pure scheduler as it does not perform tracking of status for the application. It
also does not reschedule the tasks which fail due to software or hardware errors.
The scheduler allocates the resources based on the requirements of the
applications.
ii. Application Manager
Accepts job submission.
Negotiates the first container for executing ApplicationMaster. A container
incorporates elements such as CPU, memory, disk, and network.
Restarts the ApplicationMaster container on failure.
Functions of ApplicationMaster:-
Negotiates resource container from Scheduler.
Tracks the resource container status.
Monitors progress of the application.
We can scale the YARN beyond a few thousand nodes through YARN Federation
feature. This feature enables us to tie multiple YARN clusters into a single massive
cluster. This allows for using independent clusters, clubbed together for a very large job.
Features of Yarn
a. Multi-tenancy
YARN allows a variety of access engines (open-source or propriety) on the
same Hadoop data set. These access engines can be of batch processing, real-
time processing, iterative processing and so on.
b. Cluster Utilization
With the dynamic allocation of resources, YARN allows for good use of the
cluster. As compared to static map-reduce rules in previous versions of
Hadoop which provides lesser utilization of the cluster.
c. Scalability
Any data center processing power keeps on expanding. YARN’s
ResourceManager focuses on scheduling and copes with the ever-expanding
cluster, processing petabytes of data.
d. Compatibility
MapReduce program developed for Hadoop 1.x can still on this YARN. And
this is without any disruption to processes that already work.
How Does Hadoop Work?
It is quite expensive to build bigger servers with heavy configurations that
handle large scale processing, but as an alternative, you can tie together many
 commodity computers with single-CPU, as a single functional distributed
system and practically, the clustered machines can read the datasetin parallel
and provide a much higher throughput. Moreover, it is cheaper
than one high-end server. So this is the first motivational factor behind using
Hadoop that it runs across clustered and low-cost machines.
Hadoop runs code across a cluster of computers. This process includes the
following core tasks that Hadoop performs–
Data is initially divided into directories and files. Files are divided into
uniform sized blocks of 128M and 64M (preferably 128M).
These files are then distributed across various cluster nodes for further
processing.
HDFS, being on top of the local file system, supervises the processing.
Blocks are replicated for handling hardware failure.
Checking that the code was executed successfully.
Performing the sort that takes place between the map and reduce stages.
Sending the sorted data to a certain computer.
Writing the debugging logs for each job.
Advantages of Hadoop
Hadoop framework allows the user to quickly write and test distributed
systems. It is efficient, and it automatic distributes the data and work across
the machines and in turn, utilizes the underlying parallelism of the CPU cores.
Hadoop does not rely on hardware to provide fault-tolerance and high
availability (FTHA), rather Hadoop library itself has been designed to detect
and handle failures at the application layer.
Servers can be added or removed from the cluster dynamically and Hadoop
continues to operate without interruption.
Another big advantage of Hadoop is that apart from being open source, it is
compatible on all the platforms since it is Java based.
5.2 MAP REDUCE
• MapReduce is the data processing layer of Hadoop. It is a software
framework that allows you to write applications for processing a large amount
 of data. MapReduce runs these applications in parallel on a cluster of low- end
machines. It does so in a reliable and fault-tolerant manner.
• MapReduce job comprises a number of map tasks and reduces tasks. Each task
works on a part of data. This distributes the load across the cluster. The
function of Map tasks is to load, parse, transform and filter data.
• Each reduce task works on the sub-set of output from the map tasks. Reduce
task applies grouping and aggregation to this intermediate data from the map
tasks.
• The input file for the MapReduce job exists on HDFS. The inputformat decides
how to split the input file into input splits. Input split is nothing buta byte-
oriented view of the chunk of the input file. This input split gets loaded by the
map task.
• The map task runs on the node where the relevant data is present. The data
need not move over the network and get processed locally.

Copy

Part 0 HDFS
Replication

Figure 5.4 MapReduce Data Flow

i. Map Task
The Map task run in the following phases:-
a. RecordReader
• The recordreader transforms the input split into records. It parses the
data into records but does not parse records itself. It provides the data to
the mapper function in key-value pairs. Usually, the key is the positional
information and value is the data that comprises the record.
b. Map
• In this phase, the mapper which is the user-defined function processes
the key-value pair from the recordreader. It produces zero or multiple
intermediate key-value pairs.
• The decision of what will be the key-value pair lies on the mapper
function. The key is usually the data on which the reducer function does
the grouping operation. And value is the data which gets aggregatedto get
the final result in the reducer function.
c. Combiner
• The combiner is actually a localized reducer which groups the datain
the map phase. It is optional. Combiner takes the intermediate data from
the mapper and aggregates them. It does so within the small scope of one
mapper. In many situations, this decreases the amount of data needed to
move over the network.
• For example, moving (Hello World, 1) three times consumes more
network bandwidth than moving (Hello World, 3). Combiner provides
extreme performance gain with no drawbacks. The combiner is not
guaranteed to execute. Hence it is not of overall algorithm.
d. Partitioner
• Partitioner pulls the intermediate key-value pairs from the mapper.
It splits them into shards, one shard per reducer. By default, partitioner
fetches the hashcode of the key. The partitioner performs modulus
operation by a number of reducers: key.hashcode()%(number of reducers).
• This distributes the keyspace evenly over the reducers. It also ensures that
key with the same value but from different mappers end up into the same
reducer. The partitioned data gets written on the local file system from
each map task. It waits there so that reducer can pull it.
 b. Reduce Task
The various phases in reduce task are as follows:
i. Shuffle and Sort
• The reducer starts with shuffle and sort step. This step downloads the data
written by partitioner to the machine where reducer is running. This step sorts
the individual data pieces into a large data list.
• The purpose of this sort is to collect the equivalent keys together. The
framework does this so that we could iterate over it easily in the reduce task.
This phase is not customizable.
• The framework handles everything automatically. However, the developer has
control over how the keys get sorted and grouped through a comparator object.
ii. Reduce
• The reducer performs the reduce function once per key grouping. The
framework passes the function key and an iterator object containing all the
values pertaining to the key.
• We can write reducer to filter, aggregate and combine data in a number of
different ways. Once the reduce function gets finished it gives zero or more
key-value pairs to the outputformat. Like map function, reduce function
changes from job to job. As it is the core logic of the solution.
iii. OutputFormat
• This is the final step. It takes the key-value pair from the reducer and writes
it to the file by recordwriter. By default, it separates the key and value by a tab
and each record by a newline character.
• We can customize it to provide richer output format. But none the less final
data gets written to HDFS.

HDFS Split HDFS Split HDFS Split

Input (key, value) pairs Input (key, value) pairs Input (key, value) pairs

Map Map Map

(Key1, value1) (Keyk, valuek) (Key1, value1) (Keyk,

 valuek) (Key1, value1) (Keyk, valuek)

(Key1, intermediate values) (Keyk, intermediate values)

(Final Key1, final value) (Final Keyk, final value)

Figure 5.5 How MapReduce work together

5.3 VIRTUAL BOX
• VirtualBox is open-source software for virtualizing the x86 computing
architecture. It acts as a hypervisor, creating a VM (virtual machine) in which
the user can run another OS (operating system).
• The operating system in which VirtualBox runs is called the “host” OS. The
operating system running in the VM is called the “guest” OS. VirtualBox
supports Windows, Linux, or macOS as its host OS.
• When configuring a virtual machine, the user can specify how many
CPU cores, and how much RAM and disk space should be devoted to the
VM. When the VM is running, it can be “paused.” System execution is frozen
at that moment in time, and the user can resume using it later.
History
• VirtualBox was originally developed by Innotek GmbH, and released on
January 17, 2007 as an open-source software package. The company was later
purchased by Sun Microsystems.
• On January 27, 2010, Oracle Corporation purchased Sun, and took over
development of VirtualBox.
Supported guest operating systems
Guest operating systems supported by VirtualBox include:
• Windows 10, 8, 7, XP, Vista, 2000, NT, and 98.
• Linux distributions based on Linux kernel 2.4 and newer, including Ubuntu,
Debian, OpenSUSE, Mandriva/Mandrake, Fedora, RHEL, and Arch Linux.
• Solaris and OpenSolaris.
 • macOS X Server Leopard and Snow Leopard.
• OpenBSD and FreeBSD.
• MS-DOS.
• OS/2.
• QNX.
• BeOS R5.
• Haiku.
• ReactOS.
5.4 GOOGLE APP ENGINE (GAE)
• Google has the world’s largest search engine facilities. The company has
extensive experience in massive data processing that has led to new insights
into data-center design and novel programming models that scale to
incredible sizes.
• The Google platform is based on it s search engine expertise, this
infrastructure is applicable to many other areas. Google has hundreds of data
centers and has installed more than 460,000 servers worldwide.
• For example, 200 Google data centers are used at one time for a number of
cloud applications. Data items are stored in text, images, and video and are
replicated to tolerate faults or failures. Here we discuss Google’s App Engine
(GAE) which offers a PaaS platform supporting various cloud and web
applications.
5.4.1 Google Cloud Infrastructure
• Google has pioneered cloud development by leveraging the large number of
data centers it operates.
• For example, Google pioneered cloud services in Gmail, Google Docs, and
Google Earth, among other applications. These applications can support a
large number of users simultaneously with HA.
• Notable technology achievements include the Google File System (GFS),
MapReduce, BigTable, and Chubby.
• In 2008, Google announced the GAE web application platform which is
becoming a common platform for many small cloud service providers. This
platform specializes in supporting scalable (elastic) web applications. GAE
 enables users to run their applications on a large number of data centers
associated with Google’s search engine operations.
5.4.2 GAE Architecture
• Figure 5.6 shows the major building blocks of the Google cloud platform
which has been used to deliver the cloud services highlighted earlier. GFSis
used for storing large amounts of data.
• MapReduce is for use in application program development. Chubby is used for
distributed application lock services. BigTable offers a storage service for
accessing structured data.
• Users can interact with Google applications via the web interface provided
by each application. Third-party application providers can use GAE to build
cloud applications for providing services.
• The applications all run in data centers under tight management by Google
engineers. Inside each data center, there are thousands of servers forming
different clusters.

Googld cloud
infrastructure

FIGURE 5.6 Google cloud platform and major building blocks, the blocks shown are large
clusters of low-cost servers.
• Google is one of the larger cloud application providers, although its
fundamental service program is private and outside people cannot use the
Google infrastructure to build their own service.
• The building blocks of Google’s cloud computing application include the
 Google File System for storing large amounts of data, the MapReduce
programming framework for application developers, Chubby for distributed
application lock services, and BigTable as a storage service for accessing
structural or semistructural data. With these building blocks, Google has built
many cloud applications.
• Figure 5.6 shows the overall architecture of the Google cloud infrastructure. A
typical cluster configuration can run the Google File System, MapReduce
jobs, and BigTable servers for structure data. Extra services such as Chubby
for distributed locks can also run in the clusters. GAE runs the user program
on Google’s infrastructure.
• As it is a platform running third-party programs, application developers now
do not need to worry about the maintenance of servers. GAE can be thought of
as the combination of several software components. The frontend is an
application framework which is similar to other web application frameworks
such as ASP, J2EE, and JSP. The applications can run similar to web
application containers. The frontend can be used as the dynamic web serving
infrastructure which can provide the full support of common technologies.
5.4.3 Functional Modules of GAE
• The GAE platform comprises the following five major components. The GAE
is not an infrastructure platform, but rather an application development
platform for users. We describe the component functionalities separately.
a. The datastore offers object-oriented, distributed, structured data storage
services based on BigTable techniques. The datastore secures data
management operations.
b. The application runtime environment offers a platform for scalable web
programming and execution. It supports two development languages:
Python and Java.
c. The software development kit (SDK) is used for local application
development. The SDK allows users to execute test runs of local
applications and upload application code.
d. The administration console is used for easy management of user application
development cycles, instead of for physical resource management.
e. The GAE web service infrastructure provides special interfaces to guarantee
flexible use and management of storage and network resources by GAE.
 • Google offers essentially free GAE services to all Gmail account owners. You
can register for a GAE account or use your Gmail account name to sign up for
the service. The service is free within a quota. If you exceed the quota, the
page instructs you on how to pay for the service. Then you download the SDK
and read the Python or Java guide to get started.
• Note that GAE only accepts Python, Ruby, and Java programming languages.
The platform does not provide any IaaS services, unlike Amazon, which offers
Iaas and PaaS. This model allows the user to deploy user-built applications on
top of the cloud infrastructure that are built using the programming languages
and software tools supported by the provider (e.g., Java, Python). Azure does
this similarly for .NET. The user does not manage the underlying cloud
infrastructure. The cloud provider facilitates support of application
development, testing, and operation support on a well-defined service
platform.
5.4.4 GAE Applications
• Well-known GAE applications include the Google Search Engine, Google
Docs, Google Earth, and Gmail. These applications can support large numbers
of users simultaneously. Users can interact with Google applications via the
web interface provided by each application. Third-party application providers
can use GAE to build cloud applications for providing services. The
applications are all run in the Google data centers. Inside each data center,
there might be thousands of server nodes to form different clusters. (See the
previous section.) Each cluster can run multipurpose servers.
• GAE supports many web applications. One is a storage service to store
application-specific data in the Google infrastructure. The data can be
persistently stored in the backend storage server while still providing the
facility for queries, sorting, and even transactions similar to traditional
database systems.
• GAE also provides Google-specific services, such as the Gmail account
service (which is the login service, that is, applications can use the Gmail
account directly). This can eliminate the tedious work of building customized
user management components in web applications. Thus, web applications
built on top of GAE can use the APIs authenticating users and sending e- mail
using Google accounts.
5.5 PROGRAMMING SUPPORT OF GOOGLE APP ENGINE
5.5.1 Programming the Google App Engine
 • Figure 5.7 summarizes some key features of GAE programming model for two
supported languages: Java and Python. A client environment that includes an
Eclipse plug-in for Java allows you to debug your GAE on your local
machine. Also, the GWT Google Web Toolkit is available for Java web
application developers. Developers can use this, or any other language using
a JVM based interpreter or compiler, such as JavaScript or Ruby. Python is
often used with frameworks such as Django and CherryPy, but Google also
supplies a built in webapp Python environment.
• There are several powerful constructs for storing and accessing data. The data
store is a NOSQL data management system for entities that can be, at most, 1
MB in size and are labeled by a set of schema-less properties. Queries can
retrieve entities of a given kind filtered and sorted by the values of the
properties.
• Java offers Java Data Object (JDO) and Java Persistence API (JPA) interfaces
implemented by the open source Data Nucleus Access platform, while Python
has a SQL-like query language called GQL. The data store is strongly
consistent and uses optimistic concurrency control.
Table 5.1 Comparison of MapReduce++ Subcategories along with the Loosely
Synchronous Category Used in MPI
• An update of an entity occurs in a transaction that is retried a fixed number
of times if other processes are trying to update the same entity simultaneously.
Your application can execute multiple data store operations in a single
transaction which either all succeed or all fail together. The data store
implements transactions across its distributed network using “entity groups.”
• A transaction manipulates entities within a single group. Entities of the same
group are stored together for efficient execution of transactions. Your GAE
application can assign entities to groups when the entities are created. The
performance of the data store can be enhanced by in-memory caching using
the memcache, which can also be used independently of the data store.
• Recently, Google added the blobstore which is suitable for large files as its size
limit is 2 GB. There are several mechanisms for incorporating external
resources. The Google SDC Secure Data Connection can tunnel through the
Internet and link your intranet to an external GAE application.
• The URL Fetch operation provides the ability for applications to fetch
resources and communicate with other hosts over the Internet using HTTP
 and HTTPS requests. There is a specialized mail mechanism to send e-mail
from your GAE application.
• Applications can access resources on the Internet, such as web services or other
data, using GAE’s URL fetch service. The URL fetch service retrieves web
resources using the same highspeed Google infrastructure that retrieves web
pages for many other Google products. There are dozens of Google
“corporate” facilities including maps, sites, groups, calendar, docs, and
YouTube, among others. These support the Google Data API which can be
used inside GAE.
• An application can use Google Accounts for user authentication. Google
Accounts handles user account creation and sign-in, and a user that already has
a Google account (such as a Gmail account) can use that account with your
app. GAE provides the ability to manipulate image data using a dedicated
Images service which can resize, rotate, flip, crop, and enhance images. An
application can perform tasks outside of responding to web requests. Your
application can perform these tasks on a schedule that you configure, such as
on a daily or hourly basis using “cron jobs,” handled by the Cron service.
• Alternatively, the application can perform tasks added to a queue by the
application itself, such as a background task created while handling a request.
A GAE application is configured to consume resources up to certain limits or
quotas. With quotas, GAE ensures that your application won’t exceed your
budget, and that other applications running on GAE won’t impact the
performance of your app. In particular, GAE use is free up to certain quotas.
5.5.2 Google File System (GFS)
• GFS was built primarily as the fundamental storage service for Google’s
search engine. As the size of the web data that was crawled and saved was
quite substantial, Google needed a distributed file system to redundantly store
massive amounts of data on cheap and unreliable computers.
• None of the traditional distributed file systems can provide such functions and
hold such large amounts of data. In addition, GFS was designed for Google
applications, and Google applications were built for GFS. In traditional file
system design, such a philosophy is not attractive, as there
 should be a clear interface between applications and the file system, such as
a POSIX interface.
• There are several assumptions concerning GFS. One is related to the
characteristic of the cloud computing hardware infrastructure (i.e., the high
component failure rate). As servers are composed of inexpensive commodity
components, it is the norm rather than the exception that concurrent failures
will occur all the time. Another concerns the file size in GFS. GFS typically
will hold a large number of huge files, each 100MB or larger, with files that
are multiple GB in size quite common.
• Thus, Google has chosen its file data block size to be 64MB instead of the
4 KB in typical traditional file systems. The I/O pattern in the Google
application is also special. Files are typically written once, and the write
operations are often the appending data blocks to the end of files. Multiple
appending operations might be concurrent. There will be a lot of large
streaming reads and only a little random access. As for large streaming reads,
highly sustained throughput is much more important than low latency.
• Thus, Google made some special decisions regarding the design of GFS. As
noted earlier, a 64 MB block size was chosen. Reliability is achieved by using
replications (i.e., each chunk or data block of a file is replicated across more
than three chunk servers). A single master coordinates access as wellas keeps
the metadata. This decision simplified the design and managementof the
whole cluster.
• Developers do not need to consider many difficult issues in distributed
systems, such as distributed consensus. There is no data cache in GFS as large
streaming reads and writes represent neither time nor space locality. GFS
provides a similar, but not identical, POSIX file system accessing interface.
The distinct difference is that the application can even see the physical location
of file blocks. Such a scheme can improve the upper-layer applications. The
customized API can simplify the problem and focus on Google applications.
 /foo/bar
GFS client GFS master

Chunk data Linux file system

FIGURE 5.8 Architecture of Google File System (GFS).

• The customized API adds snapshot and record append operations to facilitate
the building of Google applications. Figure 5.8 shows the GFS architecture. It
is quite obvious that there is a single master in the whole cluster. Other nodes
act as the chunk servers for storing data, while the single master stores the
metadata. The file system namespace and locking facilities are managed by the
master.
• The master periodically communicates with the chunk servers to collect
management information as well as give instructions to the chunk servers to
do work such as load balancing or fail recovery.
• The master has enough information to keep the whole cluster in a healthy state.
With a single master, many complicated distributed algorithms can be avoided
and the design of the system can be simplified.
• However, this design does have a potential weakness, as the single GFS master
could be the performance bottleneck and the single point of failure. To
mitigate this, Google uses a shadow master to replicate all the data on the
master, and the design guarantees that all the data operations are performed
directly between the client and the chunk server.
• The control messages are transferred between the master and the clients and
they can be cached for future use. With the current quality of commodity
servers, the single master can handle a cluster of more than 1,000 nodes. Figure
5.9 shows the data mutation (write, append operations) in GFS. Data blocks
must be created for all replicas.
 4 Step 1

Secondary
replica A

Primary
5
Legend:
Control
Secondary Data
replica B

FIGURE 5.9 Data mutation sequence in GFS.

The goal is to minimize involvement of the master. The mutation takes the
following steps:
1. The client asks the master which chunk server holds the current lease for the
chunk and the locations of the other replicas. If no one has a lease, the master
grants one to a replica it chooses (not shown).
2. The master replies with the identity of the primary and the locations of the
other (secondary) replicas. The client caches this data for future mutations.It
needs to contact the master again only when the primary becomes unreachable
or replies that it no longer holds a lease.
3. The client pushes the data to all the replicas. A client can do so in any order.
Each chunk server will store the data in an internal LRU buffer cache until the
data is used or aged out. By decoupling the data flow from the control flow,
we can improve performance by scheduling the expensive data flow based on
the network topology regardless of which chunk server is the primary.
4. Once all the replicas have acknowledged receiving the data, the client sends
a write request to the primary. The request identifies the data pushed earlier to
all the replicas. The primary assigns consecutive serial numbers to all the
mutations it receives, possibly from multiple clients, which provides the
necessary serialization. It applies the mutation to its own local state in serial
order.
5. The primary forwards the write request to all secondary replicas. Each
secondary replica applies mutations in the same serial number order assigned
by the primary.
6. The secondaries all reply to the primary indicating that they have completed
the operation.
7. The primary replies to the client. Any errors encountered at any replicas are
reported to the client.
• In case of errors, the write corrects at the primary and an arbitrary subset of the
secondary replicas. The client request is considered to have failed, and the
modified region is left in an inconsistent state. Our client code handles such
errors by retrying the failed mutation. It will make a few attempts at steps 3
through 7 before falling back to a retry from the beginning of the write.
• Thus, besides the writing operation provided by GFS, special appending
operations can be used to append the data blocks to the end of the files. The
reason for providing such operations is that some of the Google applications
need a lot of append operations. For example, while crawlers are gathering data
from the web, the contents of web pages will be appended to page files. Thus,
the appending operation is provided and optimized. The client specifies data to
be appended and GFS appends it to the file atomically at least once. GFS picks
the offset and the clients cannot decide the offset of the data position. The
appending operation works for concurrent writers.
• GFS was designed for high fault tolerance and adopted some methods to
achieve this goal. Master and chunk servers can be restarted in a few seconds,
and with such a fast recovery capability, the window of time in which the data
is unavailable can be greatly reduced. As we mentioned before, each chunk is
replicated in at least three places and can tolerate at least two data crashes for
a single chunk of data.
• The shadow master handles the failure of the GFS master. For data integrity,
GFS makes checksums on every 64 KB block in each chunk. With the
previously discussed design and implementation, GFS can achieve the goals
of high availability (HA), high performance, and large scale. GFS
demonstrates how to support large-scale processing workloads on commodity
hardware designed to tolerate frequent component failures optimized for huge
files that are mostly appended and read.
5.5.3 BigTable, Google’s NOSQL System
• BigTable was designed to provide a service for storing and retrieving
structured and semistructured data. BigTable applications include storageof
web pages, per-user data, and geographic locations. Here we use web pages
to represent URLs and their associated data, such as contents, crawled
metadata, links, anchors, and page rank values.
• Per-user data has information for a specific user and includes such data as user
preference settings, recent queries/search results, and the user’s e-mails.
Geographic locations are used in Google’s well-known Google Earth software.
Geographic locations include physical entities (shops, restaurants, etc.), roads,
satellite image data, and user annotations.
• The scale of such data is incredibly large. There will be billions of URLs, and
each URL can have many versions, with an average page size of about 20 KB
per version. The user scale is also huge.
• There are hundreds of millions of users and there will be thousands of queries
per second. The same scale occurs in the geographic data, which might
consume more than 100 TB of disk space.
• It is not possible to solve such a large scale of structured or semistructured data
using a commercial database system. This is one reason to rebuild the data
management system; the resultant system can be applied across many projects
for a low incremental cost. The other motivation for rebuilding the data
management system is performance. Low-level storage optimizations help
increase performance significantly, which is much harder to do when running
on top of a traditional database layer.
• The design and implementation of the BigTable system has the following
goals. The applications want asynchronous processes to be continuously
updating different pieces of data and want access to the most current data at all
times.
• The database needs to support very high read/write rates and the scale might
be millions of operations per second. Also, the database needs to support
efficient scans over all or interesting subsets of data, as well as efficient joins
of large one-to-one and one-to-many data sets. The application may need to
examine data changes over time (e.g., contents of a web page over multiple
crawls).
 • Thus, BigTable can be viewed as a distributed multilevel map. It provides a
fault-tolerant and persistent database as in a storage service. The BigTable
system is scalable, which means the system has thousands of servers, terabytes
of in-memory data, petabytes of disk-based data, millions of reads/ writes per
second, and efficient scans.
• Also, BigTable is a self-managing system (i.e., servers can be added/removed
dynamically and it features automatic load balancing). Design/initial
implementation of BigTable began at the beginning of 2004. BigTable is used
in many projects, including Google Search, Orkut, and Google Maps/ Google
Earth, among others. One of the largest BigTable cell manages ~200 TB of
data spread over several thousand machines.
The BigTable system is built on top of an existing Google cloud infrastructure.
BigTable uses the following building blocks:
1. GFS: stores persistent state
2. Scheduler: schedules jobs involved in BigTable serving
3. Lock service: master election, location bootstrapping
4. MapReduce: often used to read/write BigTable data
Example 5.1 BigTable Data Model Used in Mass Media
• BigTable provides a simplified data model compared to traditional database
systems. Figure 5.10(a) shows the data model of a sample table, Web Table.
Web Table stores the data about a web page.
• Each web page can be accessed by the URL. The URL is considered the row
index. The column provides different data related to the corresponding URL—
for example, different versions of the contents, and the anchors appearing in
the web page. In this sense, BigTable is a distributed multidimensional
stored sparse map.
• The map is indexed by row key, column key, and timestamp—that is, (row:
string, column: string, time: int64) maps to string (cell contents). Rows are
ordered in lexicographic order by row key. The row range for a table is
dynamically partitioned and each row range is called “Tablet.” Syntax for
columns is shown as a (family:qualifier) pair. Cells can store multiple versions
of data with timestamps.
 • Such a data model is a good match for most of Google’s (and other
organizations’) applications. For rows, Name is an arbitrary string and access
to data in a row is atomic. This is different from the traditional relational
database which provides abundant atomic operations (transactions). Row
creation is implicit upon storing data. Rows are ordered lexicographically, that
is, close together lexicographically, usually on one or a small numberof
machines.
“Contents” “anchor:cnnsi.com” “anchor:my.look.ca”

<html
“com.cnn.www”

(a) BigTable data model

BigTable cell

Handles failover monitoring Holds tablet data, logs

Holds metadata
handles master election
(b) BigTable structure

FIGURE 5.10 BigTable data model and system structure.

• Large tables are broken into tablets at row boundaries. A tablet holds a
contiguous range of rows. Clients can often choose row keys to achieve
locality. The system aims for about 100MB to 200MB of data per tablet.
• Each serving machine is responsible for about 100 tablets. This can achieve
faster recovery times as 100 machines each pick up one tablet from the failed
machine. This also results in fine-grained load balancing, that is, migrating
 tablets away from the overloaded machine. Similar to the design in GFS, a
master machine in BigTable makes load-balancing decisions.
• Figure 5.10(b) shows the BigTable system structure. A BigTable master
manages and stores the metadata of the BigTable system. BigTable clients use
the BigTable client programming library to communicate with the BigTable
master and tablet servers. BigTable relies on a highly available and persistent
distributed lock service called Chubby.
5.5.3.1 Tablet Location Hierarchy
• Figure 5.11 shows how to locate the BigTable data starting from the file stored
in Chubby. The first level is a file stored in Chubby that contains the location
of the root tablet. The root tablet contains the location of all tablets in a special
METADATA table. Each METADATA tablet contains the location of a set of
user tablets. The root tablet is just the first tablet in the METADATA table, but
is treated specially; it is never split to ensure that the tablet location hierarchy
has no more than three levels.
• The METADATA table stores the location of a tablet under a row key that is
an encoding of the tablet’s table identifier and its end row. BigTable includes
many optimizations and fault-tolerant features. Chubby can guarantee the
availability of the file for finding the root tablet. The BigTable master can
quickly scan the tablet servers to determine the status of all nodes. Tablet
servers use compaction to store data efficiently. Shared logs are used for
logging the operations of multiple tablets so as to reduce the log space as well
as keep the system consistent.
Other
UserTable 1
METAD
ATA
tablet
s ...
...
Chubby file
(1st METADATA tablet) ... ...
...
. .
.
UserTable N
...

...
.

...
FIGURE 5.11 Tablet location hierarchy in using the BigTable.
5.5.4 Chubby, Google’s Distributed Lock Service
• Chubby is intended to provide a coarse-grained locking service. It can store
small files inside Chubby storage which provides a simple namespace as a
file system tree.
• The files stored in Chubby are quite small compared to the huge files in GFS.
Based on the Paxos agreement protocol, the Chubby system can be quite
reliable despite the failure of any member node. Figure 5.12 showsthe
overall architecture of the Chubby system.

5 servers of a Chubby cell

Chubby
application

Master
Chubby
application

Client processes

FIGURE 5.12 Structure of Google Chubby for distributed lock service.

• Each Chubby cell has five servers inside. Each server in the cell has the same
file system namespace. Clients use the Chubby library to talk to the servers in
the cell.
• Client applications can perform various file operations on any server in the
Chubby cell. Servers run the Paxos protocol to make the whole file system
reliable and consistent.
• Chubby has become Google’s primary internal name service. GFS and
BigTable use Chubby to elect a primary from redundant replicas.
5.6 OPENSTACK
• OpenStack was been introduced by Rackspace and NASA in July 2010. The
project is building an open source community spanning technologists,
developers, researchers, and industry to share resources and technologies with
the goal of creating a massively scalable and secure cloud infrastructure.
• In the tradition of other open source projects, the software is open source and
limited to just open source APIs such as Amazon.
 • Currently, OpenStack focuses on the development of two aspects of cloud
computing to address compute and storage aspects with the OpenStack
Compute and OpenStack Storage solutions. “OpenStack Compute is the
internal fabric of the cloud creating and managing large groups of virtual
private servers” and “OpenStack Object Storage is software for creating
redundant, scalable object storage using clusters of commodity servers to store
terabytes or even petabytes of data.”
• Recently, an image repository was prototyped. The image repository contains an
image registration and discovery service and an image delivery service.
Together they deliver images to the compute service while obtaining them
from the storage service. This development gives an indication that the project
is striving to integrate more services into its portfolio.
OpenStack Compute
• As part of its computing support efforts, OpenStack is developing a cloud
computing fabric controller, a component of an IaaS system, known as Nova.
• The architecture for Nova is built on the concepts of shared-nothing and
messaging-based information exchange. Hence, most communication in Nova
is facilitated by message queues. To prevent blocking components while
waiting for a response from others, deferred objects are introduced. Such
objects include callbacks that get triggered when a response is received.
• This is very similar to established concepts from parallel computing, such as
“futures,” which have been used in the grid community by projects suchas
the CoG Kit.
• To achieve the shared-nothing paradigm, the overall system state is kept in
a distributed data system. State updates are made consistent through atomic
transactions. Nova it implemented in Python while utilizing a number of
externally supported libraries and components. This includes boto, an
Amazon API provided in Python, and Tornado, a fast HTTP server used to
implement the S3 capabilities in OpenStack.
• Figure 5.13 shows the main architecture of Open Stack Compute. In this
architecture, the API Server receives HTTP requests from boto, converts the
commands to and from the API format, and forwards the requests to the cloud
controller.
 • The cloud controller maintains the global state of the system, ensures
authorization while interacting with the User Manager via Lightweight
Directory Access Protocol (LDAP), interacts with the S3 service, and
manages nodes, as well as storage workers through a queue.
• Additionally, Nova integrates networking components to manage private
networks, public IP addressing, virtual private network (VPN) connectivity,
and firewall rules.

FIGURE 5.13 OpenStack Nova system architecture.

It includes the following types:
• NetworkController manages address and virtual LAN (VLAN) allocations
• RoutingNode governs the NAT (network address translation) conversion of
public IPs to private IPs, and enforces firewall rules
• AddressingNode runs Dynamic Host Configuration Protocol (DHCP)
services for private Networks
• TunnelingNode provides VPN connectivity
The network state (managed in the distributed object store) consists of the
following:
• VLAN assignment to a project
• Private subnet assignment to a security group in a VLAN
• Private IP assignments to running instances
• Public IP allocations to a project
• Public IP associations to a private IP/running instance
OpenStack Storage
• The OpenStack storage solution is built around a number of interacting
components and concepts, including a proxy server, a ring, an object server,
a container server, an account server, replication, updaters, and auditors. The
role of the proxy server is to enable lookups to the accounts, containers, or
objects in OpenStack storage rings and route the requests. Thus, any object is
streamed to or from an object server directly through the proxy server to or
from the user. A ring represents a mapping between the names of entities stored
on disk and their physical locations.
• Separate rings for accounts, containers, and objects exist. A ring includes the
concept of using zones, devices, partitions, and replicas. Hence, it allows the
system to deal with failures, and isolation of zones representing a drive,a
server, a cabinet, a switch, or even a data center. Weights can be used to
balance the distribution of partitions on drives across the cluster, allowing
users to support heterogeneous storage resources. According to the
documentation, “the Object Server is a very simple blob storage server that can
store, retrieve and delete objects stored on local devices.”
• Objects are stored as binary files with metadata stored in the file’s extended
attributes. This requires that the underlying file system is built around object
servers, which is often not the case for standard Linux installations. To list
objects, a container server can be utilized. Listing of containers is handled
by the account server. The first release of OpenStack “Austin” Compute and
Object Storage was October 22, 2010. This system has a strong developer
community.
5.7 FEDERATION IN THE CLOUD
• One challenge in creating and managing a globally decentralized cloud
computing environment is maintaining consistent connectivity between
untrusted components while remaining fault-tolerant. A key opportunity for
the emerging cloud industry will be in defining a federated cloud ecosystem
by connecting multiple cloud computing providers using a common standard.
• A notable research project being conducted by Microsoft, called the Geneva
Framework, focuses on issues involved in cloud federation. Geneva has been
described as a claims-based access platform and is said to help simplify access
to applications and other systems.
• The concept allows for multiple providers to interact seamlessly with others,
and it enables developers to incorporate various authentication models that will
work with any corporate identity system, including Active Directory,
LDAPv3-based directories, application-specific databases, and new user-
centric identity models such as LiveID, OpenID, and InfoCard systems. It also
supports Microsoft’s Card-Space and Novell’s Digital Me.
• The remainder of this section focuses on federation in the cloud through use
of the Internet Engineering Task Force (IETF) standard Extensible Messaging
and Presence Protocol (XMPP) and interdomain federation using the Jabber
Extensible Communications Platform (Jabber XCP), because this protocol is
currently used by a wide range of existing services offered by providers as
diverse as Google Talk, Live Journal, Earthlink, Facebook, ooVoo, Meebo,
Twitter, the U.S. Marines Corps, the Defense Information Systems Agency
(DISA), the U.S. Joint Forces Command (USJFCOM), and the National
Weather Service. We also look at federation with non-XMPP technologies
such as the Session Initiation Protocol (SIP), which is the foundation of
popular enterprise messaging systems such as IBM’s Lotus Sametime and
Microsoft’s Live Communications Server (LCS) and Office Communications
Server (OCS).
• Jabber XCP is a highly scalable, extensible, available, and device-agnostic
presence solution built on XMPP and supports multiple protocols such as
Session Initiation Protocol for Instant Messaging and Presence Leveraging
Extensions (SIMPLE) and Instant Messaging and Presence Service (IMPS).
Jabber XCP is a highly programmable platform, which makes it ideal for
adding presence and messaging to existing applications or services and for
building next-generation, presence-based solutions.
• Over the last few years there has been a controversy brewing in web services
architectures. Cloud services are being talked up as a fundamental shift in web
architecture that promises to move us from interconnected silos to a
collaborative network of services whose sum is greater than its parts. The
problem is that the protocols powering current cloud services, SOAP (Simple
• Object Access Protocol) and a few other assorted HTTP-based protocols, are
all one-way information exchanges. Therefore cloud services aren’t real- time,
won’t scale, and often can’t clear the firewall. Many believe that those barriers
can be overcome by XMPP (also called Jabber) as the protocol that
 will fuel the Software-as-a-Service (SaaS) models of tomorrow. Google,
Apple, AOL, IBM, Livejournal, and Jive have all incorporated this protocol
into their cloud-based solutions in the last few years.
• Since the beginning of the Internet era, if you wanted to synchronize services
between two servers, the most common solution was to have the client “ping”
the host at regular intervals, which is known as polling. Polling is how most of
us check our email. We ping our email server every few minutes to see if we
have new mail. It’s also how nearly all web services application programming
interfaces (APIs) work.
• XMPP’s profile has been steadily gaining since its inception as the protocol
behind the open source instant messenger (IM) server jabberd in 1998.
XMPP’s advantages include:
❖ It is decentralized, meaning anyone may set up an XMPP server.
❖ It is based on open standards.
❖ It is mature—multiple implementations of clients and servers exist.
❖ Robust security is supported via Simple Authentication and Security
Layer (SASL) and Transport Layer Security (TLS).
• It is flexible and designed to be extended.
• XMPP is a good fit for cloud computing because it allows for easy two way
communication; it eliminates the need for polling; it has rich publish subscribe
(pub-sub) functionality built in; it is XML-based and easily extensible, perfect
for both new IM features and custom cloud services; itis efficient and has
been proven to scale to millions of concurrent users ona single service (such
as Google’s GTalk); and it also has a built-in worldwidefederation model.
• Of course, XMPP is not the only pub-sub enabler getting a lot of interest from
web application developers. An Amazon EC2-backed server can run Jetty and
Cometd from Dojo. Unlike XMPP, Comet is based on HTTP and in
conjunction with the Bayeux Protocol, uses JSON to exchange data.
• Given the current market penetration and extensive use of XMPP and XCP for
federation in the cloud and that it is the dominant open protocol in that space,
we will focus on its use in our discussion of federation.
 • The ability to exchange data used for presence, messages, voice, video, files,
notifications, etc., with people, devices, and applications gain more power
when they can be shared across organizations and with other service providers.
• Federation differs from peering, which requires a prior agreement between
parties before a server-to-server (S2S) link can be established. In the past,
peering was more common among traditional telecommunications providers
(because of the high cost of transferring voice traffic). In the brave new Internet
world, federation has become a de facto standard for most email systems
because they are federated dynamically through Domain Name System (DNS)
settings and server configurations.
5.8 FOUR LEVELS OF FEDERATION
Technically speaking, federation is the ability for two XMPP servers in different
domains to exchange XML stanzas. According to the XEP-0238: XMPP Protocol Flows
for Inter-Domain Federation, there are at least four basic types of federation :
1. Permissive federation - Permissive federation occurs when a server accepts
a connection from a peer network server without verifying its identity using
DNS lookups or certificate checking. The lack of verification or
authentication may lead to domain spoofing (the unauthorized use of a third-
party domain name in an email message in order to pretend to be someone
else), which opens the door to widespread spam and other abuses. With the
release of the open source jabberd 1.2 server in October 2000, which included
support for the Server Dialback protocol (fully supported in Jabber XCP),
permissive federation met its demise on the XMPP network.
2. Verified federation - This type of federation occurs when a server accepts
a connection from a peer after the identity of the peer has been verified. It uses
information obtained via DNS and by means of domain-specific keys
exchanged beforehand. The connection is not encrypted, and the use of identity
verification effectively prevents domain spoofing. To make this work,
federation requires proper DNS setup, and that is still subject to DNS
poisoning attacks. Verified federation has been the default service policy on
the open XMPP since the release of the open-source jabberd 1.2 server.
3. Encrypted federation - In this mode, a server accepts a connection from a
peer if and only if the peer supports Transport Layer Security (TLS) as defined
 for XMPP in Request for Comments (RFC) 3920. The peer must present a
digital certificate. The certificate may be self-signed, but this prevents using
mutual authentication. If this is the case, both parties proceed to weakly verify
identity using Server Dialback. XEP-0220 defines the Server Dialback
protocol, which is used between XMPP servers to provide identity
verification. Server Dialback uses the DNS as the basis for verifying identity;
the basic approach is that when a receiving server receives a server-to-server
connection request from an originating server, it does not accept the request
until it has verified a key with an authoritative server for the domain asserted
by the originating server. Although Server Dialback does not provide strong
authentication or trusted federation, and although it is subject to DNS
poisoning attacks, it has effectively prevented most instances of address
spoofing on the XMPP network since its release in 2000. This results in an
encrypted connection with weak identity verification.
4. Trusted federation - Here, a server accepts a connection from a peer only
under the stipulation that the peer supports TLS and the peer can present a
digital certificate issued by a root certification authority (CA) that is trusted by
the authenticating server. The list of trusted root CAs may be determined by
one or more factors, such as the operating system, XMPP server software, or
local service policy. In trusted federation, the use of digital certificates results
not only in a channel encryption but also in strong authentication. The use of
trusted domain certificates effectively prevents DNS poisoning attacks but
makes federation more difficult, since such certificates have traditionally not
been easy to obtain.
5.9 FEDERATED SERVICES AND APPLICATIONS
• S2S federation is a good start toward building a real-time communications
cloud. Clouds typically consist of all the users, devices, services, and
applications connected to the network. In order to fully leverage the
capabilities of this cloud structure, a participant needs the ability to find other
entities of interest. Such entities might be end users, multiuser chat rooms, real-
time content feeds, user directories, data relays, messaging gateways, etc.
Finding these entities is a process called discovery.
• XMPP uses service discovery (as defined in XEP-0030) to find the
aforementioned entities. The discovery protocol enables any network
participant to query another entity regarding its identity, capabilities, and
 associated entities. When a participant connects to the network, it queries the
authoritative server for its particular domain about the entities associated with
that authoritative server.
• In response to a service discovery query, the authoritative server informsthe
inquirer about services hosted there and may also detail services that are
available but hosted elsewhere.
• XMPP includes a method for maintaining personal lists of other entities,
known as roster technology, which enables end users to keep track of various
types of entities. Usually, these lists are comprised of other entities the users
are interested in or interact with regularly.
• Most XMPP deployments include custom directories so that internal users
of those services can easily find what they are looking for.
5.10 The Future of Federation
• The implementation of federated communications is a precursor to buildinga
seamless cloud that can interact with people, devices, information feeds,
documents, application interfaces, and other entities.
• The power of a federated, presence-enabled communications infrastructure
is that it enables software developers and service providers to build and deploy
such applications without asking permission from a large, centralized
communications operator.
• The process of server-to-server federation for the purpose of interdomain
communication has played a large role in the success of XMPP, which relies
on a small set of simple but powerful mechanisms for domain checking and
security to generate verified, encrypted, and trusted connections between any
two deployed servers.
• These mechanisms have provided a stable, secure foundation for growth of the
XMPP network and similar real time technologies.
 UNIT 5
PART A
1. What is Haddoop?.
• Hadoop is an Apache open source framework written in java that allows
distributed processing of large datasets across clusters of computers using
simple programming models.
• The Hadoop framework application works in an environment that provides
distributed storage and computation across clusters of computers.
• Hadoop is designed to scale up from single server to thousands of machines,
each offering local computation and storage.

2. Listout two major layers of Hadoop framework.

1. Processing/Computation layer (MapReduce), and
2. Storage layer (Hadoop Distributed File System).

3. Discuss about MapReduce model.

• MapReduce is a parallel programming model for writing distributed
applications devised at Google for efficient processing of large amounts of data
(multi-terabyte data-sets), on large clusters (thousands of nodes) of commodity
hardware in a reliable, fault-tolerant manner.
• The MapReduce program runs on Hadoop which is an Apache open-source
framework.

4. What is Hadoop Distributed File System (HDFS)?.

i. The Hadoop distributed file system (HDFS) is a distributed, scalable, and
portable file system written in Java for the Hadoop framework.
ii. A Hadoop is divided into HDFS and MapReduce. HDFS is used for storing
the data and MapReduce is used for the Processing the Data.

5. Listout the HDFS services.

HDFS has five services as follows:
 1. Name Node
2. Secondary Name Node
3. Job tracker
4. Data Node
5. Task Tracker

6. What is the function of Name Node in HDFS.

HDFS consists of only one Name Node we call it as Master Node which can track the
files, manage the file system and has the meta data and the whole data in it. To be
particular Name node contains the details of the No. of blocks, Locations at what data
node the data is stored and where the replications are stored and other details. As we
have only one Name Node we call it as Single Point Failure. It has Direct connect with the
client.
7. What is the use of Name Node in HDFS.
Data Node: A Data Node stores data in it as the blocks. This is also known as the slave
node and it stores the actual data into HDFS which is responsible for the client to read
and write.
These are slave daemons. Every Data node sends a Heartbeat message to the Name
node every 3 seconds and conveys that it is alive. In this way when Name Node does
not receive a heartbeat from a data node for 2 minutes, it will take that data node as
dead and starts the process of block replications on some other Data node.
8. How job tracker is useful for processing the data?
Job Tracker: Basically Job Tracker will be useful in the Processing the data. Job
Tracker receives the requests for Map Reduce execution from the client.
Job tracker talks to the Name node to know about the location of the data like Job
Tracker will request the Name Node for the processing the data. Name node in
response gives the Meta data to job tracker.
9. What is the use of Task Tracker?.
It is the Slave Node for the Job Tracker and it will take the task from the Job Tracker.
And also it receives code from the Job Tracker. Task Tracker will take the code and
apply on the file. The process of applying that code on the file is known as Mapper.
10. List out the Features of HDFS.
1 It is suitable for the distributed storage and processing.
2 Hadoop provides a command interface to interact with HDFS.
 3 The built-in servers of namenode and datanode help users to easily check the
status of cluster.
4 Streaming access to file system data.
5 HDFS provides file permissions and authentication.

11. What are the goals of HDFS

1. Fault detection and recovery ” Since HDFS includes a large number of
commodity hardware, failure of components is frequent. Therefore HDFS
should have mechanisms for quick and automatic fault detection and
recovery.
2. Huge datasets ” HDFS should have hundreds of nodes per cluster to manage
the applications having huge datasets.
3. Hardware at data ” A requested task can be done efficiently, when the
computation takes place near the data. Especially where huge datasets are
involved, it reduces the network traffic and increases the throughput.
12. Listout the two modules of Hadoop framework.
1 Hadoop Common: These are Java libraries and utilities required by other
Hadoop modules.
2 Hadoop YARN: This is a framework for job scheduling and cluster resource
management.

13. What is Hadoop Common?.

Hadoop Common refers to the collection of common utilities and libraries that
support other Hadoop modules. It is an essential part or module of the Apache
Hadoop Framework, along with the Hadoop Distributed File System (HDFS),
Hadoop YARN and Hadoop MapReduce.
Hadoop Common is also known as Hadoop Core.

14. What are the features of Yarn?

• Multi-tenancy
• Cluster Utilization
• Scalability
• Compatibility

15. List out the Advantages of Hadoop

• Hadoop framework allows the user to quickly write and test distributed systems.
It is efficient, and it automatic distributes the data and work across the machines
and in turn, utilizes the underlying parallelism of the CPU cores.

• Hadoop does not rely on hardware to provide fault-tolerance and high

availability (FTHA), rather Hadoop library itself has been designed to detect and
handle failures at the application layer.

16. What is VirtualBox

• VirtualBox is open-source software for virtualizing the x86 computing
architecture. It acts as a hypervisor, creating a VM (virtual machine) in which
the user can run another OS (operating system).
• The operating system in which VirtualBox runs is called the “host” OS. The
operating system running in the VM is called the “guest” OS. VirtualBox
supports Windows, Linux, or macOS as its host OS.

17. How Google File System is used for google search engine?
• GFS was built primarily as the fundamental storage service for Google’s search
engine.

• As the size of the web data that was crawled and saved was quite substantial,
Google needed a distributed file system to redundantly store massive amounts of
data on cheap and unreliable computers.

18. List out the building blocks of BigTable.

1. GFS: stores persistent state
2. Scheduler: schedules jobs involved in BigTable serving
3. Lock service: master election, location bootstrapping
4. MapReduce: often used to read/write BigTable data
19. What are the four basic types of federation?
1. Permissive federation
2. Verified federation
3. Encrypted federation
4. Trusted federation

20. What is XMPP?

• XMPP uses service discovery (as defined in XEP-0030) to find the
aforementioned entities.
• The discovery protocol enables any network participant to query another entity
regarding its identity, capabilities, and associated entities.
• When a participant connects to the network, it queries the authoritative server for
its particular domain about the entities associated with that authoritative server.

PART B
1. Explain Hadoop architecture with neat diagram.
2. Discuss Hadoop YARN Architecture with its diagram.
3. How MapReduce implemented in Hadoop?.
4. What is GAE?.Explain GAE architecture and its functional modules.
5. Explain Google File System (GFS) architecture.
6. Discuss BigTable data model and its system structure in detail.
7. Explain OpenStack Nova system architecture.
8. Discuss the federated services and its applications.