1.2 Underlying Principles of Parallel and Distributed Computing

SUBJECT CODE
TYPE THE SUBJECT NAME HERE
UNIT NO 1
INTRODUCTION
1.2 underlying principles of parallel and

III VI Distributed computing
20CSPC603
CLOUD COMPUTING
20CSPC603
CLOUD COMPUTING (Common to CSE & IT)
Underlying principles of Parallel and Distributed Computing
⮚ Elements of Parallel Computing
⮚ Hardware Architectural Styles for Processing
⮚ Shared Vs Distributed MIMD model
⮚ Approaches to Parallel Computing Model
⮚ Levels of Parallelism
⮚ Components of Distributed System
⮚ Architectural Styles for Distributed Computing
⮚ Models for Inter-Process Communication
⮚ Technologies for Distributed Computing
⮚ Service Oriented Computing
2
20CSPC603
PRINCIPLES OF PARALLEL AND DISTRIBUTED COMPUTING
⮚The architecture of parallel computing system is often characterized by the homogeneity of

components (i.e.) each processor is of the same type and it has the same capability as the
others.
⮚The shared memory - a single address space, which is accessible to all the processors.
⮚Parallel programs :
⮚Broken down into several units of execution that can be allocated to different processors and
can communicate with each other by means of shared memory.
⮚Parallel systems - include all architectures that are based on the concept of shared memory,
whether this is physically present or created with the support of libraries, specific hardware
3
20CSPC603
PRINCIPLES OF PARALLEL AND DISTRIBUTED COMPUTING
⮚Distributed Computing:
• Encompasses any architecture or system that allows the computation to be broken down into units
and
executed concurrently on different computing elements,
⮚Distributed computing includes a wider range of systems and applications than parallel computing
Classic examples of distributed computing systems are
•Computing Grids
•Internet Computing Systems
4
20CSPC603
WHAT IS PARALLEL PROCESSING?
⮚ Processing of multiple tasks simultaneously on multiple processors
⮚ The parallel program consists of multiple active processes ( tasks) simultaneously solving a given
problem.
⮚ A given task is divided into multiple subtasks using a divide-and-conquer technique, and each subtask is
processed on a different central processing unit (CPU).
⮚ Programming on multi processor system using the divide-and-conquer technique is called parallel
programming.
5
20CSPC603
HARDWARE ARCHITECTURES FOR PARALLEL PROCESSING
⮚ The core elements of parallel processing are CPUs.
⮚ Based on the number of instructions and data streams, that can be processed simultaneously,
computing systems are classified into the following four categories:
• Single-instruction, Single-data (SISD) systems
• Single-instruction, Multiple-data (SIMD) systems
• Multiple-instruction, Single-data (MISD) systems
• Multiple-instruction, Multiple-data (MIMD) systems
6
20CSPC603
Single – Instruction , Single Data (SISD) systems
⮚SISD computing system is a uni-processor machine

capable of executing a single instruction, which
operates on a single data stream.
Instruction
⮚Machine instructions are processed sequentially,
hence computers adopting this model are popularly Stream
called sequential computers.
⮚Most conventional computers are built using SISD

model. Data Data
⮚The speed of processing element in the SISD Input Output
model is limited by the rate at which the computer
can transfer information internally.
⮚Dominant representative SISD systems are IBM

PC, Macintosh, and workstations.
Processor
7
20CSPC603
Single – Instruction , Multiple Data (SIMD) systems
Single Instruction Stream
Data Input 1 Data Output 1
Processor 1
Data Input 2 Data Output 2
Processor 2
Data Input N Data Output N
Processor N
8
20CSPC603
Single – Instruction , Multiple Data (SIMD) systems
⮚SIMD computing system is a multiprocessor machine capable of executing the same instruction on all the
CPUs but operating on different data streams.
⮚Machines based on this model are well suited for scientific computing since they involve lots of vector and
matrix operations.
⮚For instance statement Ci = Ai * Bi, can be passed to all the processing elements (PEs), organized data
elements of vectors A and B can be divided into multiple sets ( N- sets for N PE systems), and each PE can
process one data set.
⮚Dominant representative SIMD systems are Cray’s Vector processing machine and Thinking Machines Cm*,
and GPGPU accelerators
9
CS8791
Multiple – Instruction , Single Data (MISD) systems
⮚MISD computing system is a multi processor machine capable of executing different instructions
on different PEs all of them operating on the same data set.
For example
y = sin(x) + cos(x) + tan(x)
⮚Machines built using MISD model are not useful in most of the applications.
⮚Few machines are built but none of them available commercially.
⮚This type of systems are more of an intellectual exercise than a practical configuration.
10
CS8791
Multiple – Instruction , Multiple Data (MIMD) systems
Instruction Instruction Instruction

Stream 1 Stream 2 Stream N
Data Input
Data Output 1
1
Processor 1
Data Input
Data Output 2
2
Processor 2
Data Input
Data Output 3
N
Processor N
11
CS8791
Multiple – Instruction , Multiple Data (MIMD) systems
⮚MIMD computing system is a multi processor machine capable of executing multiple instructions on
multiple data sets.
⮚Each PE in the MIMD model has separate instruction and data streams, hence machines built using
this model are well suited to any kind of application.
⮚Unlike SIMD, MISD machine, PEs in MIMD machines work asynchronously
⮚MIMD machines are broadly categorized into shared-memory MIMD and distributed memory MIMD
based on the way PEs are coupled to the main memory.
12
CS8791
Shared Memory MIMD machines
⮚All the PEs are connected to a single global memory and they all have access to it.
⮚Systems based on this model are also called tightly coupled multi processor systems.
⮚The communication between PEs in this model takes place through the shared memory.
⮚Modification of the data stored in the global memory by one PE is visible to all other PEs.
⮚Dominant representative shared memory MIMD systems are silicon graphics machines and Sun/IBM SMP
( Symmetric Multi-Processing)
13
CS8791
Shared Memory MIMD machines
Processor 1 Processor 2 Processor N
Memory
Bus
Global System Memory
14
CS8791
Shared Vs Distributed MIMD model
⮚The shared memory MIMD architecture is easier to program but is less tolerant to failures and harder to
extend with respect to the distributed memory MIMD model.
⮚Failures, in a shared memory MIMD affect the entire system, whereas this is not the case of the distributed
model, in which each of the PEs can be easily isolated.
⮚Moreover, shared memory MIMD architectures are less likely to scale because the addition of more PEs leads
to memory contention.
⮚This is a situation that does not happen in the case of distributed memory, in which each PE has its own
memory.
⮚As a result, distributed memory MIMD architectures are most popular today.
15
CS8791
Approaches to Parallel Programming
⮚A sequential program is one that runs on a single processor and has a single line of control.
⮚To make many processors collectively work on a single program, the program must be divided
into smaller independent chunks so that each processor can work on separate chunks of the
problem.
⮚The program decomposed in this way is a parallel program.A wide variety of parallel
programming approaches are available.
16
CS8791
Approaches to Parallel Programming
⮚In the case of Process Parallelism, a given operation has multiple (but distinct) activities that
can be processed on multiple processors.
⮚In the case of Farmer-and-Worker model, a job distribution approach is used, one processor is
configured as master and all other remaining PEs are designated as slaves, the master assigns
the jobs to slave PEs and, on completion, they inform the master, which in turn collects results.
⮚These approaches can be utilized in different levels of parallelism.
17
CS8791
Levels of Parallelism
⮚Levels of Parallelism are decided on the lumps of code ( grain size) that can be a potential
candidate of parallelism.
•The table shows the levels of parallelism.
GRAIN SIZE CODE ITEM PARALLELIED BY
Large Separate and heavy Programmer

weight process
Medium Function or procedure Programmer
Fine Loop or instruction Parallelizing compiler

block
Very fine Instructor Processor
18
CS8791
All these approaches have a common goal:
•To boost processor efficiency by hiding latency.
•To conceal latency, there must be another thread ready to run whenever a lengthy operation
occurs.
•The idea is to execute concurrently two or more single-threaded applications. Such as

compiling, text formatting, database searching, and device simulation.
19
CS8791
Messages Messages
IPC IPC Large Level
(Processes, Tasks)
Task 1 Task 2 Task N
Shared Shared
function f1() Memory function f2() Memory function fJ() Medium Level
{…} 1
Function {…} 2
Function {…} J
Function (Threads, Functions)
a[1] = … Fine Level

a[0] = … a[k] = …
b[1] = … (Processor,
b[0] = …
Statements Statements b[k] = …
Statements Instructions)
Very Fine Level

+ x load (Cores, Pipeline,
Instructions)
20
CS8791
Laws of Caution
⮚Studying how much an application or a software system can gain from parallelism.
⮚In particular what need to keep in mind is that parallelism is used to perform multiple
activities together so that the system can increase its throughput or its speed.
⮚But the relations that control the increment of speed are not linear.
⮚For example: for a given n processors, the user expects speed to be increase by in times.
This is an ideal situation, but it rarely happens because of the communication overhead.
21
CS8791
LAWS OF CAUTION
Here two important guidelines to take into account.
⮚Speed of computation is proportional to the square root of the system cost; they never increase
linearly. Therefore, the faster a system becomes, the more expensive it is to increase its speed
⮚Speed by a parallel computer increases as the logarithm of the number of processors (i.e.
y=k*log(N)).
22
CS8791
General concepts and definitions
⮚ A distributed system is a collection of independent computers that appears to its users as a

single coherent system.
⮚ A distributed system is one in which components located at networked computers communicate

and coordinate their action only by passing messages
⮚ The components of a distributed system communicate with some sort of message passing. This
is a term the encompasses several communication models.
23
CS8791
COMPONENTS OF DISTRIBUTED SYSTEM
Frameworks for
distributed Applications
programming
IPC primitives Middleware

for control and
data.
Operating System
Networking and
Parallel
Hardware
Hardware
24
CS8791
Architectural styles for distributed computing
⮚ Software Architectural styles :

Relates to the logical organization of the software.
⮚ System Architectural styles:

Describe the physical organization of distributed software systems in terms of their major
components.
25
CS8791
Software Architectural Styles
⮚Software architectural styles are based on the logical arrangement of software components.
⮚They are helpful because they provide an intuitive view of the whole system, despite its physical
deployment.
⮚They also identify the main abstractions that are used to shape the components of the system and
the expected interaction patterns between them
26
CS8791
SYSTEM ARCHITECTURAL STYLES
Category Most common architectural styles
Data centered Repository
Data Flow Pipe and filter

Batch Sequential
Virtual Machine Rule based Interpreter
Call and return Main Program and subroutine

call/top –down systems
Layered systems
Independent Communication processes
Components Event Systems
27
CS8791
DATA CENTERED ARCHITECTURES

⮚Identify the data as the fundamental element of the software system
⮚Access to shared data is the core characteristics of the data-centered architectures.
⮚The repository architectural style is the most relevant reference model in this category.
⮚ It is characterized by two main components – the central data structure, which represents the current
state of the system, and a collection of independent component, which operate on the central data
⮚The ways in which the independent components interact with the central data structure can be very
heterogeneous.
⮚In particular repository based architectures differentiate and specialize further into subcategories
according to the choice of control discipline to apply for the shared data structure. Of particular
interest are databases and blackboard systems.
⮚In the repository systems, the dynamics of the system is controlled by independent components,
which by issuing an operation on the central repository, trigger the selection of specific
processes that operate on data. 28
CS8791
DATA FLOW ARCHITECTURES
⮚Access to data is the core feature, data-flow styles explicitly incorporate the pattern of data-flow
⮚Styles within this category differ in one of the following ways:

How the control is exerted, the degree of concurrency among components,
The topology that describes the flow of data.
⮚Batch Sequential:
The batch sequential style is characterized by an ordered sequence of separate programs executing
one after the other.
These programs are chained together by providing as input for the next program the output
generated by the last program after its completion, which is most likely in the form of a file
Pipe-and-Filter Style:
Each component of the processing chain is called a filter, and the connection between one filter and
the next is represented by a data stream.
29
CS8791
VIRTUAL MACHINE ARCHITECTURES
⮚The virtual machine class of architectural styles is characterized by the presence of an abstract execution
environment (virtual machine) that simulates features that are not available in the hardware or software.
⮚Applications and systems are implemented on top of this layer and become portable over different
hardware and software environments.
Examples : Rule based systems, Interpreters, and Command language processors
⮚Rule-Based Style:
This architecture is characterized by representing the abstract execution environment as an inference

engine.
Programs are expressed in the form of rules or predicates that hold true. The input data for applications is
generally represented by a set of assertions or facts that the inference engine uses to activate rules or to
apply predicates, thus transforming data.
⮚Interpreter Style: The presence of engine to interpret the style.

30
CS8791
System Architectural Styles
•System architectural styles cover the physical organization of components and processes over a
distributed infrastructure.
Two fundamental reference style
Client / Server
The information and the services of interest can be centralized and accessed through a single access
point : the server.
Multiple clients are interested in such services and the server must be appropriately designed to efficiently
serve requests coming from different clients.
Peer- to – Peer
Symmetric architectures in which all the components, called peers, play the same role and incorporate both
client and server capabilities of the client/server model.
More precisely, each peer acts as a server when it processes requests from other peers and as a client
when it issues requests to other peers.
31
CS8791
Client / Server architectural Styles
request
Two Tier
(Classic Model)
client response server
Three Tier
client Server/client server
N Tier
server
client Server/client Server/client

server
32
CS8791
Peer-to-Peer architectural Style
peer
peer
peer peer
peer
peer
peer
33
CS8791
Models for Inter process Communication
⮚Distributed systems are composed of a collection of concurrent processes interacting with each other
by means of a network connection.
⮚IPC is a fundamental aspect of distributed systems design and implementation.
⮚IPC is used to either exchange data and information or coordinate the activity of processes.
⮚IPC is what ties together the different components of a distributed system, thus making them act as a
single system.
⮚There are several different models in which processes can interact with each other – these maps to
different abstractions for IPC.
⮚Among the most relevant that we can mention are shared memory, remote procedure call (RPC), and
message passing.
⮚At lower level, IPC is realized through the fundamental tools of network programming.
⮚Sockets are the most popular IPC primitive for implementing communication channels between
distributed processes.
34
CS8791
Message-based communication
⮚The term message-based communication model can be used to refer to any model for IPC.
⮚Message Passing :
The entities exchanging information explicitly encode in the form of a message the data to be
exchanged. The structure and the content of a message vary according to the model.
Examples :
Message-Passing-Interface (MPI)
OpenMP.
35
CS8791
⮚ Remote Procedure Call (RPC) : This paradigm extends the concept of procedure call beyond the
boundaries of a single process, thus triggering the execution of code in remote processes.
⮚ Distributed Objects : This is an implementation of the RPC model for the object-oriented paradigm and
contextualizes this feature for the remote invocation of methods exposed by objects.
⮚ Examples of distributed object infrastructures are
Common Object Request Broker Architecture (CORBA)

Component Object Model (COM, DCOM, and COM+)
Java Remote Method Invocation (RMI)
.NET Remoting.
36
CS8791
⮚ Distributed agents and active Objects
⮚ Web Service
⮚Point-to-Point message model
⮚Publish – and – Subscribe message model
Push Strategy
Pull Strategy
⮚Request- reply message model
37
.
CS8791
Technologies for distributed computing
⮚Remote Procedure Call (RPC)
RPC is the fundamental abstraction enabling the execution procedures on clients’ request.
RPC allows extending the concept of a procedure call beyond the boundaries of a process and a
single memory address space.
The called procedure and calling procedure may be on the same system or they may be on
different systems.
The important aspect of RPC is marshalling and unmarshalling.
.
38
CS8791
Remote Procedure Call (RPC)
Node A Node B
Main Procedure Procedure

Registry
Procedure Procedure C:Node

A B Procedure C
Procedure
B RPC Library RPC Service
Program A (RPC Program C (RPC
Client) Server)
Parameters Return Value Parameters Return Value
Marshaling and Unmarshaling Unmarshaling and Marshaling
Procedure Name Procedure Name
Network
39
CS8791
Distributed Object Programming model
Node A Node B
Application B
21
10 16
Instance Remote
15
Instance
5: Object
1: Ask for Object Proxy Object Skeleton Activation
Reference
9 11 20 17 14 6 4
Remote Reference Module Remote Reference Module
Application A
8 12 7
13
2 19 18 3
Network
40
CS8791
Examples of distributed Object frameworks

⮚Common Object Request Broker Architecture (CORBA):
cross platform and cross language interoperability among distributed components.
⮚Distributed Component Object Model (DCOM/COM+) :
Microsoft technology for distributed object programming before the introduction of .NET technology.
⮚Java Remote Method Invocation (RMI):

technology provided by Java for enabling RPC among distributed Java objects.
⮚.NET Remoting: IPC among .NET applications, a uniform platform for accessing remote objects from within
any application developed in any of the languages supported by .NET.
41
CS8791
Service Oriented Computing
⮚Service – oriented computing organizes distributed systems in terms of services, which represent the
major abstraction for building systems.
⮚Service orientation expresses applications and software systems as an aggregation of services that are
coordinated within a service oriented architecture (SOA).
⮚Even though there is no designed technology for the development of service-oriented software systems,
web services are the de facto approach for developing SOA.
⮚Web services, the fundamental component enabling Cloud computing systems, leverage the Internet as
the main interaction channel between users and the system
42

1.2 Underlying Principles of Parallel and Distributed Computing

Uploaded by

Copyright:

Available Formats

1.2 Underlying Principles of Parallel and Distributed Computing

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1.2 Underlying Principles of Parallel and Distributed Computing

Uploaded by

Copyright:

Available Formats

SUBJECT CODE

TYPE THE SUBJECT NAME HERE

1.2 underlying principles of parallel and

Underlying principles of Parallel and Distributed Computing

⮚ Elements of Parallel Computing

⮚ Hardware Architectural Styles for Processing

⮚ Shared Vs Distributed MIMD model

⮚ Approaches to Parallel Computing Model

⮚ Components of Distributed System

⮚ Architectural Styles for Distributed Computing

⮚ Models for Inter-Process Communication

⮚ Technologies for Distributed Computing

⮚ Service Oriented Computing

PRINCIPLES OF PARALLEL AND DISTRIBUTED COMPUTING

⮚The architecture of parallel computing system is often characterized by the homogeneity of

PRINCIPLES OF PARALLEL AND DISTRIBUTED COMPUTING

Classic examples of distributed computing systems are

•Internet Computing Systems

WHAT IS PARALLEL PROCESSING?

⮚ Processing of multiple tasks simultaneously on multiple processors

HARDWARE ARCHITECTURES FOR PARALLEL PROCESSING

⮚ The core elements of parallel processing are CPUs.

• Single-instruction, Single-data (SISD) systems

• Single-instruction, Multiple-data (SIMD) systems

• Multiple-instruction, Single-data (MISD) systems

• Multiple-instruction, Multiple-data (MIMD) systems

⮚SISD computing system is a uni-processor machine

⮚Most conventional computers are built using SISD

⮚Dominant representative SISD systems are IBM

Single – Instruction , Multiple Data (SIMD) systems

Single Instruction Stream

Data Input 1 Data Output 1

Data Input 2 Data Output 2

Data Input N Data Output N

Single – Instruction , Multiple Data (SIMD) systems

Multiple – Instruction , Single Data (MISD) systems

y = sin(x) + cos(x) + tan(x)

⮚Few machines are built but none of them available commercially.

Instruction Instruction Instruction

⮚Unlike SIMD, MISD machine, PEs in MIMD machines work asynchronously

Shared Memory MIMD machines

Shared Memory MIMD machines

Processor 1 Processor 2 Processor N

Global System Memory

Shared Vs Distributed MIMD model

Approaches to Parallel Programming

Approaches to Parallel Programming

⮚These approaches can be utilized in different levels of parallelism.

•The table shows the levels of parallelism.

GRAIN SIZE CODE ITEM PARALLELIED BY

Large Separate and heavy Programmer

Fine Loop or instruction Parallelizing compiler

All these approaches have a common goal:

•To boost processor efficiency by hiding latency.

•The idea is to execute concurrently two or more single-threaded applications. Such as

a[1] = … Fine Level

Very Fine Level

Here two important guidelines to take into account.

General concepts and definitions

⮚ A distributed system is a collection of independent computers that appears to its users as a