DISTRIBUTED

COMPUTING
Sunita Mahajan,
Mahajan Principal, Institute of
Computer Science, MET League of
Colleges, Mumbai

Seema Shah,
Shah Principal, Vidyalankar
Institute of Technology, Mumbai
University

© Oxford University Press 2011

 Chapter - 7
Distributed Shared Memory

© Oxford University Press 2011

 Topics
• Introduction
• Basic concepts of DSM
• Hardware DSM
• Design issues in DSM
• Issues in implementing DSM systems
• Heterogeneous and other DSM systems
• Case Study

© Oxford University Press 2011

 Introduction

© Oxford University Press 2011

 IPC paradigms
• Message passing
• Shared memory

• Multi computer systems are easier to build but

harder to program while multiprocessor systems are
complex to build but easier to program
• Distributed Shared Memory systems (DSM) are both
easy to program and easy to build

© Oxford University Press 2011

Basic Concepts Of DSM

© Oxford University Press 2011

 DSM
• A DSM system provides a logical abstraction of
shared memory which is built using a set of
interconnected nodes having physically
distributed memories.

© Oxford University Press 2011

 DSM architecture-1
• DSM:
– Ease of programming and portability
– Scalable with very high computing power

© Oxford University Press 2011

 DSM architecture-2
• Cluster based architecture

© Oxford University Press 2011

Comparison of IPC paradigms

© Oxford University Press 2011

 Types of DSMs
• Hardware level DSM
• Software level DSM
• Hybrid level DSM

© Oxford University Press 2011

 Hardware level DSM

• Smaller unit of sharing

• HDSM is used where performance is more
important than cost
• Not scalable to more than 10 systems

© Oxford University Press 2011

 Software level DSM
• SDSM can be implemented where compiler
detects shared accesses and inserts call to
synchronization routines into executable
code.

© Oxford University Press 2011

 Hybrid level DSM

• Lies between HDSM and SDSM

• Eg NUMA(Non uniform memory access)

© Oxford University Press 2011

 Advantages of DSM
• Simple abstraction
• Improved portability of distributed application
programs
• Provides better performance in some
applications
• Large memory space at no extra cost
• Better than message passing systems

© Oxford University Press 2011

Hardware DSM

© Oxford University Press 2011

 Hardware architectures
• On chip memory
• Bus based multiprocessor
• Ring based multiprocessor
• Switched multiprocessor

© Oxford University Press 2011

 On chip memory
- The address and data lines are directly connected from all
CPU to single shared memory
- Very Expensive and practically impossible

© Oxford University Press 2011

 Bus based multiprocessor
• Use bus arbitration mechanism

© Oxford University Press 2011

Consistency protocols

© Oxford University Press 2011

 Cache consistency protocol
• Properties:
– Consistency is achieved since all caches do us
snooping
– Protocol is built into MMU
– The algorithm is performed in one memory cycle

© Oxford University Press 2011

 Ring based Multiprocessor are
implemented in Memnet DSM
• Single shared
Memory is divided
– Private areas
– Shared areas

© Oxford University Press 2011

Memnet: Node memory

© Oxford University Press 2011

• Valid – bit tells Whether block is
Present in cache

• Exclusive bit tells whether it is local copy

Or only copy

• Home bit is set if it is data blocks home

Machine

• Interrupt is used to force interrupts

• Location field tells whether block is locate d in cache

And whether it is present and valid

© Oxford University Press 2011

 Comparison
• The major difference between bus based and
ring based multiprocessors is that the former
are tightly coupled while the latter are loosely
coupled.

• Ring based multiprocessors are almost

hardware implementation of DSM.

© Oxford University Press 2011

 Switched multiprocessor
Multiple clusters interconnected by a bus offer better scalability

• Example : Dash system

© Oxford University Press 2011

Design Issues In DSM

© Oxford University Press 2011

 DSM design issues
• Granularity of sharing
• Structure of data
• Consistency models
• Coherence protocols

© Oxford University Press 2011

 Granularity
• False sharing
• Thrashing

© Oxford University Press 2011

 DSM structure
• Organization of data items in the shared
memory

© Oxford University Press 2011

 Consistency models
• Refers to how recent the shared memory updates
are visible to all the other processes running on
different machines

© Oxford University Press 2011

 Strict consistency
• Strongest form of consistency

© Oxford University Press 2011

 Sequential consistency
• All processors in the system observe the same
ordering of reads and writes which are issued in
sequence by the individual processors

© Oxford University Press 2011

 Causal consistency
• Weakening of sequential consistency for better
concurrency
• Causally related operation is the one which has
influenced the other operation

© Oxford University Press 2011

 PRAM consistency
• Pipelined Random Access Memory consistency
• Write operations performed by a single process are seen by
all other processes in the order in which they were performed
just as if these write operations were performed by a single
process in a pipeline.
• Write operations performed by different processes may be
seen by different processes in different orders.

© Oxford University Press 2011

 Processor consistency
• Adheres to the PRAM consistency
• Constraint on memory coherence
• Order in which the memory operations are
seen by two processors need not be identical,
but the order of writes issued by each
processor must be preserved

© Oxford University Press 2011

 Weak consistency
• Use a special variable called the
synchronization variable

© Oxford University Press 2011

 Properties of the weak consistency
model
• Access to synchronization variables is
sequentially consistent
• Only when all previous writes are completed
everywhere, access to synchronizations
variable is allowed
• Until all previous accesses to synchronization
variables are performed, no read write data
access operations will be allowed.

© Oxford University Press 2011

 Release consistency
• Synchronization variables: acquire and release
• Use barrier mechanism

© Oxford University Press 2011

Eager Release Consistency

© Oxford University Press 2011

Lazy Release Consistency

© Oxford University Press 2011

 Entry consistency
• Use acquire and release at the start and end of
each critical section, respectively.
• Each ordinary shared variable is associated with
some synchronization variable such as a lock or
barrier.
• Entry consistency (EC) is similar to LRC but more
relaxed; shared data is explicitly associated with
synchronization primitives and is made
consistent when such an operation is performed

© Oxford University Press 2011

 Scope consistency
• A scope is a limited view of memory with
respect to which memory references are
performed

© Oxford University Press 2011

Comparison of consistency models-1
• Most common: sequential consistency model

© Oxford University Press 2011

Comparison of Consistency models-2
• Based on efficiency and programmability

© Oxford University Press 2011

 Coherence protocols
• Specifies how the rules set by the memory
consistency model are to be implemented

© Oxford University Press 2011

 Coherence algorithms
• Maintain consistency among replicas

© Oxford University Press 2011

 Multiple Reader/ Multiple Writer
algorithm
• Uses twin and diff creation technique

© Oxford University Press 2011

 Write Protocols for consistency
• Write Update (WU)
• Write Invalidate (WI) protocols

© Oxford University Press 2011

Issues In Implementing DSM
Systems

© Oxford University Press 2011

 Issues
• Thrashing
• Responsibility of DMS management
• Replication v/s migration
• Replacement strategy

© Oxford University Press 2011

 Thrashing
• False sharing
• Techniques to reduce
thrashing:
– Application controlled lock
– Pin the block to a node for
specific time
– Customize algorithm to
shared data usage pattern

© Oxford University Press 2011

 Responsibility for DSM management
• Algorithms for data location and consistency
management:
– Centralized manager algorithm
– Broadcast algorithm
– Fixed Distributed manager algorithm
– Dynamic distributed manager algorithm

© Oxford University Press 2011

Centralized Manager algorithm

© Oxford University Press 2011

Broadcast algorithm

Replicates

© Oxford University Press 2011

Fixed Distributed manager algorithm

© Oxford University Press 2011

Dynamic distributed manager algorithm

© Oxford University Press 2011

Replication versus migration strategies
• Replication strategy: • Non Replicated and Non
– No replication Migrating Block-
– Replication NRNMB
• Migration strategy • Non Replicated,
– No migration Migrating Block- NRMB
– Migration • Replicated, Migrating
Block- RMB
• Replicated Non
Migrating Block-RNMB

© Oxford University Press 2011

Replacement strategy

© Oxford University Press 2011

Heterogeneous And Other DSM
Systems

© Oxford University Press 2011

 Issues in building Heterogeneous
DSM systems
• Data compatibility and conversion
– DSM as a collection of source language objects
• Block size selection
– Largest page size
– Smallest page size
– Intermediate page size

© Oxford University Press 2011

 Approaches to DSM design
• Based on data caching management DSM is
managed by:
1. Operating system
2. MMU Hardware
3. Language runtime system

© Oxford University Press 2011

 Case Study

© Oxford University Press 2011

 Case studies
• Munin
• Linda
• Teamster
• JUMP

© Oxford University Press 2011

 Munin

© Oxford University Press 2011

 Linda-1

© Oxford University Press 2011

 Linda-2

© Oxford University Press 2011

 Linda-3

© Oxford University Press 2011

 Teamster-1

© Oxford University Press 2011

 Teamster-2

© Oxford University Press 2011

 JUMP-1

© Oxford University Press 2011

 JUMP-2

Migrating Home Protocol

© Oxford University Press 2011
 Summary
• Introduction
• Basic concepts of DSM
• Hardware DSM
• Design issues in DSM
• Issues in implementing DSM systems
• Heterogeneous and other DSM systems
• Case Study

© Oxford University Press 2011