+ Content Parallel

Processing
Chapter 17

n Clusters
n Multiple processor organizations
n Types of parallel processor systems n Cluster configurations
n Parallel organizations n Operating system design issues
n Symmetric multiprocessors n Cluster computer architecture
n Organization n Blade servers
n Multiprocessor operating system
design considerations n Clusters compared to SMP

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

+

n Single instruction, single data n Multiple instruction, single data

(SISD) stream (MISD) stream
n Single processor executes a n A sequence of data is transmitted
single instruction stream to to a set of processors, each of
operate on data stored in a single which executes a different
memory instruction sequence
n Uniprocessors fall into this n Not commercially implemented
category

n Single instruction, multiple data n Multiple instruction, multiple

(SIMD) stream data (MIMD) stream
n A single machine instruction n A set of processors simultaneously
controls the simultaneous
execute different instruction
execution of a number of
processing elements on a sequences on different data sets
lockstep basis n SMPs, clusters and NUMA systems
n Vector and array processors fall fit this category
into this category

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

 Processor Organizations

Single Instruction, Single Instruction, Multiple Instruction, Multiple Instruction,

Single Data Stream Multiple Data Str eam Single Data Stream Multiple Data Str eam
(SISD) (SIMD) (MISD) (MIMD)

Uniprocessor

Vector Array Shared Memory Distributed Memory

Processor Processor (tightly coupled) (loosely coupled)

Clusters

Symmetric Nonumiform
Multiprocessor Memory
(SMP) Access
(NUMA)

Figure 17.1 A Taxonomy of Parallel Pr ocessor Architectures

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
 IS DS DS
CU PU MU PU1 L M1

(a) SISD DS
PU2 L M2
IS
CU

IS DS DS
CU1 PU1 PUn L Mn

IS DS (b) SIMD (with distributed memory)

CU2 PU2

Memory
Shared
IS DS
CU1 PU1 L M1

Interconnection
IS DS IS DS
CUn PUn CU2 PU2 L M2

Network
(c) MIMD (with shared memory)

CU = control unit SISD = single instruction,

IS = instruction stream single data stream IS DS
PU = processing unit SIMD = single instruction, CUn PUn L Mn
DS = data stream multiple data stream
MU = memory unit MIMD = multiple instruction, (d) MIMD (with distributed memory)
LM = local memory multiple data stream

Figure 17.2 Alternative Computer Organizations

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
 Time

Process 1

Process 2

Process 3

(a) Interleaving (multiprogramming, one processor)

Process 1

Process 2

Process 3

(b) Interleaving and overlapping (multiprocessing; two processors)

Blocked Running

Figure 17.3 Multiprogramming and Multiprocessing

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
+
SMP Advantages
n Performance
n If some work can be done in parallel

n Availability
n Since all processors can perform the same functions, failure of a
single processor does not halt the system

n Incremental growth
n User can enhance performance by adding additional processors

n Scaling
n Vendors can offer range of products based on number of
processors
 Processor Processor Processor

I/O

I/O
Interconnection
Network

I/O

Main Memory

Figure 17.4 Generic Block Diagram of a Tightly Coupled Multipr ocessor

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
+
Time Shared Bus
n Simplest form

n Structure and interface similar to single processor system

n Following features provided

n Addressing - distinguish modules on bus
n Arbitration - any module can be temporary master
n Time sharing - if one module has the bus, others must wait and
may have to suspend

n Now have multiple processors as well as multiple I/O

modules
 Processor Processor Processor
L 1 Cache L 1 Cache L 1 Cache

L 2 Cache L 2 Cache L 2 Cache

shared bus

Main I/O
Memory I/O Adapter
Subsytem

I/O
Adapter

I/O
Adapter

Figure 17.5 Symmetric Multipr ocessor Organization

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
+
The bus organization has several
attractive features:

n Simplicity
n Simplest approach to multiprocessor organization

n Flexibility
n Generally easy to expand the system by attaching more
processors to the bus

n Reliability
n The bus is essentially a passive medium and the failure of any
attached device should not cause failure of the whole system

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

+
Disadvantages of the bus organization:

n Main drawback is performance

n All memory references pass through the common bus
n Performance is limited by bus cycle time

n Each processor should have cache memory

n Reduces the number of bus accesses

n Leads to problems with cache coherence

n If a word is altered in one cache it could conceivably invalidate a
word in another cache
n To prevent this the other processors must be alerted that an
update has taken place
n Typically addressed in hardware rather than the operating system

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

+

n Simultaneous concurrent processes

n OS routines need to be reentrant to allow several processors to execute the same IS code simultaneously
n OS tables and management structures must be managed properly to avoid deadlock or invalid operations

n Scheduling
n Any processor may perform scheduling so conflicts must be avoided
n Scheduler must assign ready processes to available processors

n Synchronization
n With multiple active processes having potential access to shared address spaces or I/O resources, care must be
taken to provide effective synchronization
n Synchronization is a facility that enforces mutual exclusion and event ordering

n Memory management
n In addition to dealing with all of the issues found on uniprocessor machines, the OS needs to exploit the available
hardware parallelism to achieve the best performance
n Paging mechanisms on different processors must be coordinated to enforce consistency when several processors
share a page or segment and to decide on page replacement

n Reliability and fault tolerance

n OS should provide graceful degradation in the face of processor failure
n Scheduler and other portions of the operating system must recognize the loss of a processor and restructure
accordingly

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

 Clusters
n Alternative to SMP as an approach to providing
high performance and high availability

n Particularly attractive for server applications

n Defined as:
n A group of interconnected whole computers working
together as a unified computing resource that can
create the illusion of being one machine
n (The term whole computer means a system that can run
on its own, apart from the cluster)

n Each computer in a cluster is called a node

+ n Benefits:
n Absolute scalability
n Incremental scalability
n High availability
n Superior price/performance
© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
 P P P P

High-speed message link

M I/O I/O I/O I/O M

(a) Standby server with no shared disk

High-speed message link

P P I/O I/O P P

M I/O I/O I/O I/O M

RAID

(b) Shared disk

Figure 17.8 Cluster Configurations

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
 Table 17.2
Clustering Methods: Benefits and Limitations
Clustering Method Description Benefits Limitations
Passive Standby A secondary server Easy to implement. High cost because the
takes over in case of secondary server is
primary server failure. unavailable for other
processing tasks.
Active Secondary: The secondary server is Reduced cost because Increased complexity.
also used for processing secondary servers can
tasks. be used for processing.
Separate Servers Separate servers have High availability. High network and
their own disks. Data is server overhead due to
continuously copied copying operations.
from primary to
secondary server.
Servers Connected Servers are cabled to the Reduced network and Usually requires disk
to Disks same disks, but each server overhead due to mirroring or RAID
server owns its disks. If elimination of copying technology to
one server fails, its disks operations. compensate for risk of
are taken over by the disk failure.
other server.
Servers Share Disks Multiple servers Low network and server Requires lock manager
simultaneously share overhead. Reduced risk software. Usually used
access to disks. of downtime caused by with disk mirroring or
disk failure. RAID technology.
© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
+

n How failures are managed depends on the clustering method used

n Two approaches:
n Highly available clusters
n Fault tolerant clusters

n Failover
n The function of switching applications and data resources over from a failed system
to an alternative system in the cluster

n Failback
n Restoration of applications and data resources to the original system once it
has been fixed

n Load balancing
n Incremental scalability
n Automatically include new computers in scheduling
n Middleware needs to recognize that processes may switch between machines

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

 Effective use of a cluster requires executing
software from a single application in parallel

Three approaches are:

Parallelizing complier Parallelized application Parametric computing

•Determines at compile time which •Application written from the outset •Can be used if the essence of the
parts of an application can be to run on a cluster and uses application is an algorithm or
executed in parallel message passing to move data program that must be executed a
•These are then split off to be between cluster nodes large number of times, each time
assigned to different computers in with a different set of starting
the cluster conditions or parameters

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

 Parallel Applications

Sequential Applications Parallel Programming E nvir onment

Cluster Middleware
(Single System Image and Availability Infrastructur e)

PC/Workstation PC/Workstation PC/Workstation PC/Workstation PC/Workstation

Comm SW Comm SW Comm SW Comm SW Comm SW

Net. Interface HW Net. Interface HW Net. Interface HW Net. Interface HW Net. Interface HW

High Speed Network/Switch

Figure 17.9 Cluster Computer Architecture

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

+
n Both provide a configuration with multiple processors to
support high demand applications
n Both solutions are available commercially

SMP Clustering
n Easier to manage and n Far superior in terms of
configure incremental and absolute
scalability
n Much closer to the original
single processor model for n Superior in terms of
which nearly all applications availability
are written
n All components of the system
n Less physical space and lower can readily be made highly
power consumption redundant

n Well established and stable

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.

+

Thank You....!!!
All The Best

© 2016 Pearson Education, Inc., Hoboken, NJ. All rights reserved.