Unit 2

Parallel and distributed computing

Eras of computing
• The two fundamental and dominant models of
computing are sequential and parallel. The sequential
computing era began in the 1940s; the parallel (and
distributed) computing era followed it within a decade
• The four key elements of computing developed during
these eras are architectures, compilers, applications,
and problem-solving environments.
Parallel vs. distributed computing

• The terms parallel computing and distributed computing are

often used interchangeably, even though they mean slightly
different things.
• The term parallel implies a tightly coupled system, whereas
distributed refers to a wider class of system, including those
that are tightly coupled.
• the term parallel computing refers to a model in which the
computation is divided among several processors sharing the
same memory.
• The architecture of a parallel computing system is often
characterized by the homogeneity of components: each
processor is of the same type and it has the same capability as
the others.
• The shared memory has a single address space, which is
accessible to all the processors.
• Parallel programs are then broken down into several units of
execution that can be allocated to different processors and
can communicate with each other by means of the shared
memory
• The term distributed computing encompasses any
architecture or system that allows the computation to be
broken down into units and executed concurrently on
different computing elements, whether these are processors
on different nodes, processors on the same computer, or
cores within the same processor.
• Therefore, distributed computing includes a wider range of
systems and applications than parallel computing and is often
considered a more general term.
• The term distributed often implies that the locations of the
computing elements are not the same and such elements
might be heterogeneous in terms of hardware and software
features.
 What is parallel processing?
• Processing of multiple tasks simultaneously on multiple
processors is called parallel processing.
• The parallel program consists of multiple active processes
(tasks) simultaneously solving a given problem.
• A given task is divided into multiple subtasks using a divide-
and-conquer technique, and each subtask is processed on a
different central processing unit (CPU).
• Programming on a multiprocessor system using the divide-
and-conquer technique is called parallel programming.
• Currently, many applications today require more computing
power than a traditional sequential computer can offer.
• Parallel processing provides a cost-effective solution to this
problem by increasing the number of CPUs in a computer and
by adding an efficient communication system between them.
• The workload can then be shared between different
processors.
• This setup results in higher computing power and
performance than a single-processor system offers
Need for parallel processing
• Performance Boost
• Large Data Handling
• Resource Utilization
• Real-Time Processing
• Complex Problem Solving
• Cost Efficiency
• Increases application responsiveness
Hardware architectures for parallel processing
The core elements of parallel processing are CPUs.
Based on the number of instruction and data streams that
can be processed simultaneously, computing systems
are classified into the following four categories:
• Single-instruction, single-data (SISD) systems
• Single-instruction, multiple-data (SIMD) systems
• Multiple-instruction, single-data (MISD) systems
• Multiple-instruction, multiple-data (MIMD) systems
Single-instruction, single-data (SISD) systems

• An SISD computing system is a uniprocessor machine capable of

executing a single instruction, which operates on a single data stream
(see Figure 2.2).
• In SISD, machine instructions are processed sequentially; hence
computers adopting this model are popularly called sequential
computers.
• Most conventional computers are built using the SISD model.
• All the instructions and data to be processed have to be stored in
primary memory.
• The speed of the processing element in the SISD model is limited by
the rate at which the computer can transfer information internally.
Single-instruction, multiple-data (SIMD) systems

• An SIMD computing system is a multiprocessor machine capable of

executing the same instruction on all the CPUs but operating on
different data streams
• Machines based on an SIMD model are well suited to scientific
computing
Multiple-instruction, single-data (MISD) systems

• An MISD computing system is a multiprocessor machine capable of

executing different instructions on different PEs but all of them
operating on the same data set
Multiple-instruction, multiple-data (MIMD)
systems
• An MIMD computing system is a multiprocessor machine capable of
executing multiple instructions on multiple data sets.
• Each PE in the MIMD model has separate instruction and data
streams; hence machines built using this model are well suited to any
kind of application.
• PEs in MIMD machines work asynchronously
MIMD machines are broadly categorized into
• Shared memory MIMD machines
• Distributed memory MIMD machines
Shared memory MIMD machines

• In the shared memory MIMD model, all the PEs are connected to a
single global memory and they all have access to it.
• The communication between PEs in this model takes place through
the shared memory
• Modification of the data stored in the global memory by one PE is
visible to all other PEs.
• Systems based on this model are also called tightly coupled
multiprocessor systems.
Distributed memory MIMD machines

• In the distributed memory MIMD model, all PEs have a local memory.
• The communication between PEs in this model takes place through
the interconnection network.
• Systems based on this model are also called loosely coupled
multiprocessor systems
• The network connecting PEs can be configured to tree, mesh, cube,
and so on. Each PE operates asynchronously
• The shared-memory MIMD architecture is easier to program but is
less tolerant to failures and harder to extend with respect to the
distributed memory MIMD model.
• Failures in a shared-memory MIMD affect the entire system, whereas
this is not the case of the distributed model, in which each of the PEs
can be easily isolated.
Approaches to parallel programming

Need :
• A sequential program is one that runs on a single processor
and has a single line of control.
• To make many processors collectively work on a single
program, the program must be divided into smaller
independent chunks so that each processor can work on
separate chunks of the problem.
A wide variety of parallel programming approaches are
available. The most prominent among them are the following:
• Data parallelism
• Process parallelism
• Farmer-and-worker model
Data parallelism
• The divide-and-conquer technique is used to split data into
multiple sets, and each data set is processed on different PEs
using the same instruction.
• This approach is highly suitable to processing on machines
based on the SIMD model.
Process parallelism
• In the case of process parallelism, a given operation has
multiple activities that can be processed on multiple
processors.
Farmer and worker model
• In the case of the farmer- and-worker model, a job
distribution approach is used.
• one processor is configured as master and all other
remaining PEs are designated as slaves
• the master assigns jobs to slave PEs and, on completion,
they inform the master, which in turn collects results.
Levels of parallelism

• Levels of parallelism are decided based on the collection of

code (grain size) that can be a possible candidate for
parallelism.
• Table 2.1 lists categories of code granularity for parallelism.
• To conceal latency, there must be another thread ready to
run whenever a lengthy operation occurs. The idea is to
execute concurrently two or more single-threaded
applications, such as compiling, text formatting, database
searching, and device simulation.
Elements of distributed computing

• Definitions : A distributed system is a collection of independent

computers that appears to its users as a single coherent system.
• Or
• A distributed system is one in which components located at
networked computers communicate and coordinate their actions only
by passing messages.
Components of a distributed system

• A distributed system is the result of the interaction of several

components that traverse the entire computing stack from hardware
to software.
• It emerges from the collaboration of several elements that—by
working together—give users the illusion of a single coherent system.
• At the very bottom layer, computer and network hardware constitute
the physical infrastructure;
• These components are directly managed by the operating system,
which provides the basic services for interprocess communication
(IPC), process scheduling and management, and resource
management in terms of file system and local devices.
• Taken together these two layers become the platform on top of which
specialized software is deployed to turn a set of networked
computers into a distributed system.
• At the operating system level, IPC services are implemented on top of
standardized communication protocols such TCP/IP, UDP or others.
• The middleware layer leverages such services to build a uniform
environment for the development and deployment of distributed
applications.
• The top of the distributed system stack is represented by the
applications and services designed and developed to use the
middleware
A cloud computing distributed system.
• Figure 2.11 shows an example of how the general reference architecture of
a distributed system is contextualized in the case of a cloud computing
system.
• hardware and operating system layers make up the bare-bone
infrastructure, where racks of servers are deployed and connected
together through high-speed connectivity.
• The core logic is then implemented in the middle- ware that manages the
virtualization layer, which is deployed on the physical infrastructure in
order to maximize its utilization and provide a customizable runtime
environment for applications.
• The middleware provides different facilities to application developers
according to the type of services sold to customers. These facilities, offered
through Web 2.0-compliant interfaces
Architectural styles for distributed computing

• We organize the architectural styles into two major classes:

• Software architectural styles
• System architectural styles

• The first class relates to the logical organization of the software;

• the second class includes all those styles that describe the physical
organization of distributed software systems in terms of their major
components.
Component and connectors

• A component represents a unit of software that encapsulates a

function or a feature of the system. Examples of components can be
programs, objects, processes, pipes, and filters.
• A connector is a communication mechanism that allows cooperation
and coordination among components. Differently from components,
connectors are not encapsulated in a single entity, but they are
implemented in a distributed manner over many system components.
Software architectural styles

• Software architectural styles are based on the logical arrangement of

software components.
• They are helpful because they provide an intuitive view of the whole
system, despite its physical deployment.
Types
Data centered architectures

• These architectures identify the data as the fundamental

element of the software system, and access to shared data is
the core characteristic of the data-centered architectures.
• repository architectural style: characterized by two main
components: the central data structure, which represents
the current state of the system, and a collection of
independent components, which operate on the central data
• In blackboard systems, the central data structure is the main
trigger for selecting the processes to execute. It is
characterized by three main components
• Knowledge sources. These are the entities that update the
knowledge base that is maintained in the blackboard.
• Blackboard. This represents the data structure that is shared
among the knowledge sources and stores the knowledge base of
the application.
• Control. The control is the collection of triggers and procedures
that govern the interaction with the blackboard and update the
status of the knowledge base.
Data-flow architectures
• In the case of data-flow architectures, it is the availability of
data that controls the computation.
• their design is determined by an orderly motion of data from
component to component, which is the form of
communication between them.
• Batch Sequential Style. The batch sequential style is
characterized by an ordered sequence of separate programs
executing one after the other.
• These programs are chained together by providing as input
for the next program the output generated by the last
program after its completion, which is most likely in the form
of a file.
• This design was very popular in the mainframe
• Pipe-and-Filter Style : The pipe-and-filter style is a variation of
the previous style for expressing the activity of a software
system as sequence of data transformations.
• Each component of the processing chain is called a filter, and
the connection between one filter and the next is represented
by a data stream.
• As soon as one filter produces a consumable amount of data,
the next filter can start its processing and they are connected
with in-memory data structures such as FIFO. This particular
sequencing is called pipelining.
• Example: Unix shell pipes, microprocessor pipeline
Virtual machine architectures

• The virtual machine class of architectural styles is characterized by

the presence of an abstract execution environment that simulates
features that are not available in the hardware or software.
• Applications and systems are implemented on top of this layer
• Rule-Based Style. This architecture is characterized by representing
the abstract execution environment as an inference engine.
• Programs are expressed in the form of rules
• popular in the field of artificial intelligence.

• Interpreter Style. The core feature of the interpreter style is the

presence of an engine that is used to interpret a pseudo-program
expressed in a format acceptable for the interpreter
Call & return architectures
• This category identifies all systems that are organised into components
mostly connected together by method calls.

• Top-Down Style. :
• Systems developed according to this style are composed of one large main
program that accom- plishes its tasks by invoking subprograms or
procedures
• works on divide-and-conquer approach

• Object-Oriented Style. systems that have been designed and implemented

by leveraging the abstractions of OOP.
• Systems are specified in terms of classes and implemented in terms of
objects.
•
• Layered Style. The layered system style allows the design and
implementation of software systems in terms of layers, which provide
a different level of abstraction of the system.
Architectural styles based on independent
components
• This class of architectural style models systems in terms of
independent components that have their own life cycles, which
interact with each other to perform their activities.
• Communicating Processes :Components are represented by
independent processes that leverage IPC facilities for coordination
management.
• Each of the processes provides other processes with services and can
leverage the services exposed by the other processes
• Event Systems : the components of the system are loosely coupled
and connected
• other components provide a callback that will be executed when the
event is activated. During the activity of a component, a specific
runtime condition can activate
System architectural styles

• System architectural styles cover the physical organization of

components and processes over a distributed infrastructure.
• They provide a set of reference models for the deployment of such
systems and help engineers not only have a common vocabulary in
describing the physical layout of systems but also quickly identify the
major advantages and drawbacks of a given deployment
Client/server

• Figure 2.12, the client/server model features two major components:

a server and a client.
• These two components interact with each other through a network
connection using a given protocol. The communication is
unidirectional
• The client issues a request to the server, and after processing the
request the server returns a response. There could be multiple client
components issuing requests to a server.
• the important operations in the client-server paradigm are request,
accept (client side), and listen and response (server side).
• The client/server model is suitable in many-to-one scenarios, where
the information and the services of interest can be centralized and
accessed through a single access point: the server
the client design, we identify two major models:
• Thin-client model. : In this model, the load of data processing and
transformation is put on the server side, and the client has a light
implementation
• Fat-client model. In this model, the client component is also
responsible for processing and transforming the data before returning
it to the user, whereas the server features a relatively light
implementation that is mostly concerned with the management of
access to the data
• types of architectures, which go under the name of multitiered
architectures
• Two-tier architecture : his architecture partitions the systems into two
tiers, which are located one in the client component and the other on
the server.
• Three-tier architecture/N-tier architecture :separates the
presentation of data, the application logic, and the data storage into
three tiers
Peer-to-peer
• The peer-to-peer model, depicted in Figure 2.13, introduces a
symmetric architecture in which all the components, called peers,
play the same role and incorporate both client and server capabilities
of the client/server model.
• each peer acts as a server when it processes requests from other
peers and as a client when it issues requests to other peers.
• Used for for highly decentralized architecture
Models for interprocess communication

• Distributed systems are composed of a collection of concurrent

processes interacting with each other by means of a network
connection
• IPC is used to either exchange data and information or coordinate the
activity of processes.
• IPC is what ties together the different components of a distributed
system, thus making them act as a single system.
Reference model for architecting the communication among processes.
• Message-based communication : The term message, in this case,
identifies any discrete amount of information that is passed from one
entity to another.
It encompasses any form of data representation that is limited in
size and time
Components :
Message passing. : data to be exchanged; Message-Passing Interface
(MPI) and OpenMP.
• Remote procedure call (RPC) : triggering the execution of code
• Distributed objects: implementation of the RPC model for the object-
oriented paradigm
• Distributed agents and active objects: models often make explicit use
of messages to trigger the execution of methods
• Web services. provides an implementation of the RPC concept over
HTTP
Models for message-based communication

Point-to-point message model

Publish-and-subscribe message model

Request-reply message model