Parallel and Distributed

Computing
Week 5
 Computer
Clusters
• A computer cluster is a set of computers that work
together so that they can be viewed as a single system.
• Unlike grid computers (where each node set to perform a
different task/ application), computer clusters have each
node set to perform the same task, controlled and
scheduled by software.
Computer Clusters cont…

Sun Microsystems Solaris computer

cluster
Computer Clusters cont…

• The components of a cluster are usually connected

to each other through fast local area networks, with each
node (computer used as a server) running its own
instance of an operating system.
Computer Clusters cont…
• In most circumstances, all of the nodes use the same hardware
and the same operating system, although in some setups (e.g.
using Open Source Cluster Application Resources (OSCAR)),
different operating systems can be used on each computer, or
different hardware.

• OSCAR is a Linux-based software installation for high-performance

cluster computing.
Computer Clusters cont…

• Prior to the advent of clusters, single unit fault tolerant

mainframes with modular redundancy were employed; but
the lower upfront cost of clusters, and increased speed of
network fabric has favoured the adoption of clusters.

• In contrast to high-reliability mainframes clusters are

cheaper to scale out, but also have increased complexity in
error handling.
 Why is Cluster Computing
important?
• Cluster computing gives a relatively inexpensive, unconventional to the
large server or mainframe computer solutions.
• It resolves the demand for content criticality and process services in a
faster way.
• Many organizations and IT companies are implementing cluster
computing to augment their scalability, availability, processing speed
and resource management at economic prices.
• It ensures that computational power is always available.
• It provides a single general strategy for the implementation and
application of parallel high-performance systems independent of
certain hardware vendors and their product decisions.
 Types of Cluster computing
1. High performance (HP) clusters: HP clusters use computer clusters and
supercomputers to solve advance computational problems. They are used to
performing functions that need nodes to communicate as they perform their jobs.
They are designed to take benefit of the parallel processing power of several
nodes.
2. Load-balancing clusters: Incoming requests are distributed for resources among
several nodes running similar programs or having similar content. This prevents
any single node from receiving a disproportionate amount of task. This type of
distribution is generally used in a web-hosting environment.
3. High Availability (HA) Clusters: HA clusters are designed to maintain redundant
nodes that can act as backup systems in case any failure occurs. Consistent
computing services like business activities, complicated databases, customer
services like e-websites and network file distribution are provided. They are
designed to give uninterrupted data availability to the customers.
Types of Cluster computing

4. Grid Computing Clusters(GC): Connect distributed computing resources across

multiple networks, allowing organizations to collaborate on large-scale projects
requiring extensive data processing and storage. These clusters comprise
heterogeneous computers, which can be geographically dispersed and are connected
via the internet or intranet. Grid computing utilizes middleware to manage resource
allocation, job scheduling, and data transfer, making it an ideal solution for research
institutions and collaborative computing efforts.
5. Virtual Clusters (VC): Maximize resource utilization by running multiple virtual
machines (VMs) on a single physical host. Managed by a hypervisor, these clusters
allow for dynamic scaling and resource allocation based on workload demands.
Virtual clusters provide isolation for applications, improving security and reducing
the risk of interference between workloads. They are commonly used in
development and testing environments, as well as in cloud computing platforms.
Scientific Research

 High-Performance Computing (HPC): Clusters are extensively

used in scientific fields that require large computational power. For
example, CERN uses clusters for particle physics research,
processing data from the Large Hadron Collider to identify
subatomic particles and test theories like the Standard Model of
particle physics.
 Weather Prediction: Meteorological organizations use cluster
computing to model and predict weather patterns. The sheer amount
of data from satellites, radars, and weather stations requires clusters
to simulate complex atmospheric conditions.

Example: The European Centre for Medium-Range Weather

Forecasts (ECMWF) uses supercomputing clusters to improve global
weather predictions.
Artificial Intelligence (AI) and
Machine Learning
 Deep Learning: Training deep neural networks requires significant
computational resources, which are often supplied by clusters of
GPUs (Graphics Processing Units). These clusters help in training
models faster by distributing the workload across multiple machines.

 AI Research: Large-scale AI models like OpenAI's GPT are trained

using massive clusters to process large datasets efficiently and
reduce training time.

Example: Google Brain and OpenAI both rely on clusters for training
AI models.
Big Data and E-commerce

 Data Analytics: Large e-commerce platforms like Amazon and

Alibaba use cluster computing to analyze customer behavior,
manage inventories, and recommend products. By distributing data
processing tasks across clusters, these companies can process vast
amounts of transactional data in real-time.

 Search Engines: Google's search engine infrastructure is based

on a massive cluster that indexes and ranks web pages. These
clusters allow efficient processing and retrieval of web content across
the globe.

Example: Hadoop Clusters are widely used for big data analytics,
allowing companies to handle and process massive data volumes in
parallel.
Cluster Computing
Architecture :
• It is designed with an array of interconnected individual
computers and the computer systems operating collectively as a
single standalone system.
• It is a group of workstations or computers working together as a
single, integrated computing resource connected via high speed
interconnects.
• A node – Either a single or a multiprocessor network having
memory, input and output functions and an operating system.
• Two or more nodes are connected on a single line or every node
might be connected individually through a LAN connection.
Components of a Cluster
Computer :
• Cluster Nodes
• Cluster Operating System
• The switch or node interconnect
• Network switching hardware
 Advantages of Cluster Computing :
• High Performance The systems offer better and enhanced performance
than that of mainframe computer networks.
• Easy to manage: Cluster Computing is manageable and easy to implement.
• Scalable : Resources can be added to the clusters accordingly.
• Expandability : Computer clusters can be expanded easily by adding
additional computers to the network. Cluster computing is capable of
combining several additional resources or the networks to the existing
computer system.
• Availability : The other nodes will be active when one node gets failed and
will function as a proxy for the failed node. This makes sure for enhanced
availability.
• Flexibility :It can be upgraded to the superior specification or additional
nodes can be added.
Computer Clusters cont… Challenges

• One of the challenges in the use of a computer cluster is

the cost of administrating it which can at times be
as high as the cost of administrating N independent
machines, if the cluster has N nodes.
• Some other challenges are discussed here:
 Computer Clusters
cont… Challenges (Task
scheduling)
• When a large multi-user cluster needs to access very large
amounts of data, task scheduling becomes a challenge.
• In a heterogeneous CPU-GPU cluster with a complex
application environment, the performance of each job
depends on the characteristics of the underlying cluster.
• Therefore, mapping tasks onto CPU cores and GPU devices
provides significant challenges.
 Computer Clusters cont…
Challenges (Node Failure
Management)
• When a node in a cluster fails, strategies such as fencing may be
employed to keep the rest of the system operational.

• Fencing is the process of isolating a node or protecting shared

resources when a node appears to be malfunctioning.

• There are two classes of fencing methods; one disables a node

itself, and the other disallows access to resources such as shared
disks.
Future Trends in Cluster Computing

 Exascale Computing: Computing systems that perform at least 10^18

operations per second.
 Quantum Clusters: Expected to revolutionize large-scale
computational tasks.
 Energy Efficiency: Developing energy-efficient clusters is critical to
reducing power consumption.
Thank you