Chapter 12: Distributed Shared Memory

Ajay Kshemkalyani and Mukesh Singhal

Distributed Computing: Principles, Algorithms, and Systems

Cambridge University Press

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

1 / 48

Distributed Computing: Principles, Algorithms, and Systems

Distributed Shared Memory Abstractions

communicate with Read/Write ops in shared virtual space No Send and Receive primitives to be used by application
Under covers, Send and Receive used by DSM manager

Locking is too restrictive; need concurrent access With replica management, problem of consistency arises! = weaker consistency models (weaker than von Neumann) reqd
process process process
response invocation

CPU Memory Memory manager

CPU Memory Memory manager

CPU Memory Memory manager

response response invocation invocation

Memory manager

Memory manager

Memory manager

Shared Virtual Memory

Shared Virtual Memory

A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory

Distributed shared memory

CUP 2008

2 / 48

Distributed Computing: Principles, Algorithms, and Systems

Distributed Shared Memory Abstractions

communicate with Read/Write ops in shared virtual space No Send and Receive primitives to be used by application
Under covers, Send and Receive used by DSM manager

Locking is too restrictive; need concurrent access With replica management, problem of consistency arises! = weaker consistency models (weaker than von Neumann) reqd
process process process
response invocation

CPU Memory Memory manager

CPU Memory Memory manager

CPU Memory Memory manager

response response invocation invocation

Memory manager

Memory manager

Memory manager

Shared Virtual Memory

Shared Virtual Memory

A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory

Distributed shared memory

CUP 2008

2 / 48

Distributed Computing: Principles, Algorithms, and Systems

Advantages/Disadvantages of DSM
Advantages: Shields programmer from Send/Receive primitives Single address space; simplies passing-by-reference and passing complex data structures Exploit locality-of-reference when a block is moved DSM uses simpler software interfaces, and cheaper o-the-shelf hardware. Hence cheaper than dedicated multiprocessor systems No memory access bottleneck, as no single bus Large virtual memory space DSM programs portable as they use common DSM programming interface Disadvantages: Programmers need to understand consistency models, to write correct programs DSM implementations use async message-passing, and hence cannot be more ecient than msg-passing implementations By yielding control to DSM manager software, programmers cannot use their own msg-passing solutions.
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 3 / 48

Distributed Computing: Principles, Algorithms, and Systems

Advantages/Disadvantages of DSM
Advantages: Shields programmer from Send/Receive primitives Single address space; simplies passing-by-reference and passing complex data structures Exploit locality-of-reference when a block is moved DSM uses simpler software interfaces, and cheaper o-the-shelf hardware. Hence cheaper than dedicated multiprocessor systems No memory access bottleneck, as no single bus Large virtual memory space DSM programs portable as they use common DSM programming interface Disadvantages: Programmers need to understand consistency models, to write correct programs DSM implementations use async message-passing, and hence cannot be more ecient than msg-passing implementations By yielding control to DSM manager software, programmers cannot use their own msg-passing solutions.
Distributed Shared Memory CUP 2008 3 / 48

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Computing: Principles, Algorithms, and Systems

Issues in Implementing DSM Software

Semantics for concurrent access must be clearly specied Semantics replication? partial? full? read-only? write-only? Locations for replication (for optimization) If not full replication, determine location of nearest data for access Reduce delays, # msgs to implement the semantics of concurrent access Data is replicated or cached Remote access by HW or SW Caching/replication controlled by HW or SW DSM controlled by memory management SW, OS, language run-time system

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

4 / 48

Distributed Computing: Principles, Algorithms, and Systems

Issues in Implementing DSM Software

Semantics for concurrent access must be clearly specied Semantics replication? partial? full? read-only? write-only? Locations for replication (for optimization) If not full replication, determine location of nearest data for access Reduce delays, # msgs to implement the semantics of concurrent access Data is replicated or cached Remote access by HW or SW Caching/replication controlled by HW or SW DSM controlled by memory management SW, OS, language run-time system

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

4 / 48

Distributed Computing: Principles, Algorithms, and Systems

Comparison of Early DSM Systems

Type of DSM single-bus multiprocessor switched multiprocessor NUMA system Page-based DSM Shared variable DSM Shared object DSM

Examples Firey, Sequent Alewife, Dash Buttery, CM* Ivy, Mirage Midway, Munin Linda, Orca

Management by MMU by MMU by OS by OS by language runtime system by language runtime system

Caching hardware control hardware control software control software control software control software control

Remote access by hardware by hardware by hardware by software by software by software

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

5 / 48

Distributed Computing: Principles, Algorithms, and Systems

Memory Coherence

si memory operations by Pi (s1 + s2 + . . . sn )!/(s1 !s2 ! . . . sn !) possible interleavings Memory coherence model denes which interleavings are permitted Traditionally, Read returns the value written by the most recent Write Most recent Write is ambiguous with replicas and concurrent accesses DSM consistency model is a contract between DSM system and application programmer
process

op1

op2

op3

opk
invocation response

local memory manager

invocation invocation invocation response response response

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

6 / 48

Distributed Computing: Principles, Algorithms, and Systems

Strict Consistency/Linearizability/Atomic Consistency

Strict consistency
1

A Read should return the most recent value written, per a global time axis. For operations that overlap per the global time axis, the following must hold. All operations appear to be atomic and sequentially executed. All processors see the same order of events, equivalent to the global time ordering of non-overlapping events.
op1 op2 op3 opk
invocation response

2 3

process

local memory manager

invocation invocation invocation response response response

Sequential invocations and responses to each Read or Write operation.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

7 / 48

Distributed Computing: Principles, Algorithms, and Systems

Strict Consistency / Linearizability: Examples

P 1 P2 Write(x,4) Write(y,2) Read(y,2) Read(x,0)

(a)Sequentially consistent but not linearizable Write(x,4) Write(y,2) Read(y,2) Read(x,4)

P 1 P2

(b) Sequentially consistent and linearizable Write(x,4) Write(y,2) Read(y,0) Read(x,0)

P 1 P 2

(c) Not sequentially consistent (and hence not linearizable)

Initial values are zero. (a),(c) not linearizable. (b) is linearizable

A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 8 / 48

Distributed Computing: Principles, Algorithms, and Systems

Linearlzability: Implementation
Simulating global time axis is expensive. Assume full replication, and total order broadcast support.
(shared var) int: x ; (1) When the Memory Manager receives a Read or Write from application: (1a) total order broadcast the Read or Write request to all processors; (1b) await own request that was broadcast; (1c) perform pending response to the application as follows (1d) case Read: return value from local replica; (1e) case Write: write to local replica and return ack to application. (2) When the Memory Manager receives a total order broadcast(Write, x, val) from network: (2a) write val to local replica of x . (3) When the Memory Manager receives a total order broadcast(Read, x) from network: (3a) no operation.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

9 / 48

Distributed Computing: Principles, Algorithms, and Systems

Linearizability: Implementation (2)

When a Read in simulated at other processes, there is a no-op. Why do Reads participate in total order broadcasts? Reads need to be serialized w.r.t. other Reads and all Write operations. See counter-example where Reads do not participate in total order broadcast.

Write(x,4) P_i P_j P_k Read(x,4)

total order broadcast

Read(x,0)

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

10 / 48

Distributed Computing: Principles, Algorithms, and Systems

Linearizability: Implementation (2)

When a Read in simulated at other processes, there is a no-op. Why do Reads participate in total order broadcasts? Reads need to be serialized w.r.t. other Reads and all Write operations. See counter-example where Reads do not participate in total order broadcast.

Write(x,4) P_i P_j P_k Read(x,4)

total order broadcast

Read(x,0)

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

10 / 48

Distributed Computing: Principles, Algorithms, and Systems

Sequential Consistency

Sequential Consistency.
The result of any execution is the same as if all operations of the processors were executed in some sequential order. The operations of each individual processor appear in this sequence in the local program order. Any interleaving of the operations from the dierent processors is possible. But all processors must see the same interleaving. Even if two operations from dierent processors (on the same or dierent variables) do not overlap in a global time scale, they may appear in reverse order in the common sequential order seen by all. See examples used for linearizability.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

11 / 48

Distributed Computing: Principles, Algorithms, and Systems

Sequential Consistency

Only Writes participate in total order BCs. Reads do not because: all consecutive operations by the same processor are ordered in that same order (no pipelining), and Read operations by dierent processors are independent of each other; to be ordered only with respect to the Write operations. Direct simplication of the LIN algorithm. Reads executed atomically. Not so for Writes. Suitable for Read-intensive programs.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

12 / 48

Distributed Computing: Principles, Algorithms, and Systems

Sequential Consistency using Local Reads

(shared var) int: x ;

(1) When the Memory Manager at Pi receives a Read or Write from application: (1a) case Read: return value from local replica; (1b) case Write(x,val): total order broadcasti (Write(x,val)) to all processors including itself.

(2) When the Memory Manager at Pi receives a total order broadcastj (Write, x, val) from network (2a) write val to local replica of x ; (2b) if i = j then return ack to application.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

13 / 48

Distributed Computing: Principles, Algorithms, and Systems

Sequential Consistency using Local Writes

(shared var) int: x ; (1) When the Memory Manager at Pi receives a Read(x ) from application: (1a) if counter = 0 then (1b) return x (1c) else Keep the Read pending. (2) When the Memory Manager at Pi receives a Write(x,val) from application: (2a) counter counter + 1; (2b) total order broadcasti the Write(x , val ); (2c) return ack to the application.

(3) When the Memory Manager at Pi receives a total order broadcastj (Write, x, val) from network (3a) write val to local replica of x . (3b) if i = j then (3c) counter counter 1; (3d) if (counter = 0 and any Reads are pending) then (3e) perform pending responses for the Reads to the application. Locally issued Writes get acked immediately. Local Reads are delayed until the locally preceding Writes have been acked. All locally issued Writes are pipelined.
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 14 / 48

Distributed Computing: Principles, Algorithms, and Systems

Causal Consistency
In SC, all Write ops should be seen in common order. For causal consistency, only causally related Writes should be seen in common order.
W(x,2) W(x,4) R(x,4) W(x,7) P2 P3 P4 R(x,2) R(x,7) R(x,4) R(x,7)

P1

Causal relation for shared memory systems

At a processor, local order of events is the causal order A Write causally precedes Read issued by another processor if the Read returns the value written by the Write. The transitive closure of the above two orders is the causal order

(a)Sequentially consistent and causally consistent W(x,2) W(x,4) W(x,7) R(x,7) R(x,2)

P1 P2 P3 P4

R(x,4)

R(x,7)

(b) Causally consistent but not sequentially consistent W(x,2) W(x,4) R(x,4) W(x,7) R(x,2) R(x,7)

P1 P2

P3 Total order broadcasts (for SC) also R(x,7) R(x,4) P4 provide causal order in shared memory (c) Not causally consistent but PRAM consistent A. systems. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 15 / 48

Distributed Computing: Principles, Algorithms, and Systems

Pipelined RAM or Processor Consistency

PRAM memory
Only Write ops issued by the same processor are seen by others in the order they were issued, but Writes from dierent processors may be seen by other processors in dierent orders. PRAM can be implemented by FIFO broadcast? PRAM memory can exhibit counter-intuitive behavior, see below.
(shared variables) int: x , y ; Process 1 ... (1a) x 4; (1b) if y = 0 then kill(P2 ). Process 2 ... (2a) y 6; (2b) if x = 0 then kill(P1 ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

16 / 48

Distributed Computing: Principles, Algorithms, and Systems

Slow Memory

Slow Memory
Only Write operations issued by the same processor and to the same memory location must be seen by others in that order.
P1 P2 W(x,2) W(y,4) R(y,4) W(x,7) R(x,0) R(x,0) R(x,7)

(a) Slow memory but not PRAM consistent W(x,2) W(y,4) R(y,4) W(x,7) R(x,7) R(x,0) R(x,2)

P1 P2

(b) Violation of slow memory consistency

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

17 / 48

Distributed Computing: Principles, Algorithms, and Systems

Hierarchy of Consistency Models

no consistency model pipelined RAM (PRAM) Sequential consistency Linearizability/ Atomic consistency/ Strict consistency Causal consistency Slow memory

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

18 / 48

Distributed Computing: Principles, Algorithms, and Systems

Synchronization-based Consistency Models: Weak Consistency

Consistency conditions apply only to special synchronization instructions, e.g., barrier synchronization Non-sync statements may be executed in any order by various processors. E.g.,weak consistency, release consistency, entry consistency

Weak consistency:
All Writes are propagated to other processes, and all Writes done elsewhere are brought locally, at a sync instruction. Accesses to sync variables are sequentially consistent Access to sync variable is not permitted unless all Writes elsewhere have completed No data access is allowed until all previous synchronization variable accesses have been performed Drawback: cannot tell whether beginning access to shared variables (enter CS), or nished access to shared variables (exit CS).
Distributed Shared Memory CUP 2008 19 / 48

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Computing: Principles, Algorithms, and Systems

Synchronization based Consistency Models: Release Consistency and Entry Consistency

Two types of synchronization Variables: Acquire and Release

Release Consistency
Acquire indicates CS is to be entered. Hence all Writes from other processors should be locally reected at this instruction Release indicates access to CS is being completed. Hence, all Updates made locally should be propagated to the replicas at other processors. Acquire and Release can be dened on a subset of the variables. If no CS semantics are used, then Acquire and Release act as barrier synchronization variables. Lazy release consistency: propagate updates on-demand, not the PRAM way.

Entry Consistency
Each ordinary shared variable is associated with a synchronization variable (e.g., lock, barrier) For Acquire /Release on a synchronization variable, access to only those ordinary variables guarded by the synchronization variables is performed.
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 20 / 48

Distributed Computing: Principles, Algorithms, and Systems

Shared Memory Mutual Exclusion: Bakery Algorithm

(shared vars) array of boolean: choosing [1 . . . n]; array of integer: timestamp [1 . . . n]; repeat (1) Pi executes the following for the entry section: (1a) choosing [i ] 1; (1b) timestamp [i ] maxk [1...n] (timestamp [k ]) + 1; (1c) choosing [i ] 0; (1d) for count = 1 to n do (1e) while choosing [count ] do no-op; (1f) while timestamp [count ] = 0 and (timestamp [count ], count ) < (timestamp [i ], i ) do (1g) no-op. (2) Pi executes the critical section (CS) after the entry section (3) Pi executes the following exit section after the CS: (3a) timestamp [i ] 0. (4) Pi executes the remainder section after the exit section until false;

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

21 / 48

Distributed Computing: Principles, Algorithms, and Systems

Shared Memory Mutual Exclusion

Mutual exclusion
Role of line (1e)? Wait for others timestamp choice to stabilize ... Role of line (1f)? Wait for higher priority (lex. lower timestamp) process to enter CS

Bounded waiting: Pi can be overtaken by other processes at most once (each) Progress: lexicographic order is a total order; process with lowest timestamp in lines (1d)-(1g) enters CS Space complexity: lower bound of n registers Time complexity: (n) time for Bakery algorithm Lamports fast mutex algorithm takes O (1) time in the absence of contention. However it compromises on bounded waiting. Uses W (x ) R (y ) W (y ) R (x ) sequence necessary and sucient to check for contention, and safely enter CS

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

22 / 48

Distributed Computing: Principles, Algorithms, and Systems

Lamports Fast Mutual Exclusion Algorithm

(shared variables among the processes) integer: x , y ; array of boolean b [1 . . . n]; repeat (1) Pi (1 i n) executes entry section: (1a) b [i ] true ; (1b) x i ; (1c) if y = 0 then (1d) b [i ] false ; (1e) await y = 0; (1f) goto (1a); (1g) y i ; (1h) if x = i then (1i) b [i ] false ; (1j) for j = 1 to N do (1k) await b [j ]; (1l) if y = i then (1m) await y = 0; (1n) goto (1a); (2) Pi (1 i n) executes critical section: (3) Pi (1 i n) executes exit section: (3a) y 0; (3b) b [i ] false ; forever. // shared register initialized // ags to indicate interest in critical section

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

23 / 48

Distributed Computing: Principles, Algorithms, and Systems

Shared Memory: Fast Mutual Exclusion Algorithm

Need for a boolean vector of size n: For Pi , there needs to be a trace of its identity and that it had written to the mutex variables. Other processes need to know who (and when) leaves the CS. Hence need for a boolean array b [1..n]. Process Pi Wi (x ) R i (y ) R j (y ) Wi (y ) Wj (y ) Ri (x ) W k (x ) Rj (x ) Process Pj W j (x ) Process Pk variables x = j, y = 0 x = i, y = 0 x = i, y = 0 x = i, y = 0 x = i, y = i x = i, y = j x = i, y = j x = k, y = j x = k, y = j

Examine all possible race conditions in algorithm code to analyze the algorithm.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

24 / 48

Distributed Computing: Principles, Algorithms, and Systems

Hardware Support for Mutual Exclusion

Test &Set and Swap are each executed atomically!!
(shared variables among the processes accessing each of the dierent object types) register: Reg initial value; // shared register initialized (local variables) integer: old initial value; // value to be returned (1) Test &Set (Reg ) returns value : (1a) old Reg ; (1b) Reg 1; (1c) return(old ). (2) Swap (Reg , new ) returns value : (2a) old Reg ; (2b) Reg new ; (2c) return(old ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

25 / 48

Distributed Computing: Principles, Algorithms, and Systems

Mutual Exclusion using Swap

(shared variables) register: Reg false ; (local variables) integer: blocked 0;

// shared register initialized // variable to be checked before entering CS

repeat (1) Pi executes the following for the entry section: (1a) blocked true ; (1b) repeat (1c) Swap (Reg , blocked ); (1d) until blocked = false ; (2) Pi executes the critical section (CS) after the entry section (3) Pi executes the following exit section after the CS: (3a) Reg false ; (4) Pi executes the remainder section after the exit section until false;

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

26 / 48

Distributed Computing: Principles, Algorithms, and Systems

Mutual Exclusion using Test &Set , with Bounded Waiting

(shared variables) register: Reg false ; array of boolean: waiting [1 . . . n]; (local variables) integer: blocked initial value; // shared register initialized

// value to be checked before entering CS

repeat (1) Pi executes the following for the entry section: (1a) waiting [i ] true ; (1b) blocked true ; (1c) while waiting [i ] and blocked do (1d) blocked Test &Set (Reg ); (1e) waiting [i ] false ; (2) Pi executes the critical section (CS) after the entry section (3) Pi executes the following exit section after the CS: (3a) next (i + 1)mod n; (3b) while next = i and waiting [next ] = false do (3c) next (next + 1)mod n; (3d) if next = i then (3e) Reg false ; (3f) else waiting [next ] false ; (4) Pi executes the remainder section after the exit section until false;
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 27 / 48

Distributed Computing: Principles, Algorithms, and Systems

Wait-freedom
Synchronizing asynchronous processes using busy-wait, locking, critical sections, semaphores, conditional waits etc. = crash/ delay of a process can prevent others from progressing. Wait-freedom: guarantees that any process can complete any synchronization operation in a nite number of low-level steps, irresp. of execution speed of others. Wait-free implementation of a concurrent object = any process can complete on operation on it in a nite number of steps, irrespective of whether others crash or are slow. Not all synchronization problems have wait-free solutions, e.g., producer-consumer problem. An n 1-resilient system is wait-free.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

28 / 48

Distributed Computing: Principles, Algorithms, and Systems

Register Hierarchy and Wait-freedom

During concurrent access, behavior of register is unpredictable For a systematic study, analyze most elementary register, and build complex ones based on the elementary register Assume a single reader and a single writer

Safe register
A Read that does not overlap with a Write returns the most recent value written to that register. A Read that overlaps with a Write returns any one of the possible values that the register could ever contain.

P1 P2 P3

Write11 (x,4)

Write21 (x,6) Read32 (x,?)

Read12 (x,?) Read22 (x,?)

Write13 (x,6)

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

29 / 48

Distributed Computing: Principles, Algorithms, and Systems

Register Hierarchy and Wait-freedom (2)

Regular register
Safe register + if a Read overlaps with a Write, value returned is the value before the Write operation, or the value written by the Write.

Atomic register
Regular register + linearizable to a sequential register

P1 P2 P3

Write11 (x,4)

Write21 (x,6) Read32 (x,?)

Read12 (x,?) Read22 (x,?)

Write13 (x,6)

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

30 / 48

Distributed Computing: Principles, Algorithms, and Systems

Classication of Registers and Register Constructions

R1 . . . Rq are weaker registers that are used to construct stronger register types R. Total of n processes assumed. Table 12.2. Classication by type, value, write-access, read-access
Type safe regular atomic Value binary integer Writing Single-Writer Multi-Writer Reading Single-Reader Multi-Reader

R
R1

Write to

Writes to individual Ri

Rq

Reads from individual Ri

Read from

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

31 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 1: SRSW Safe to MRSW Safe

Single Writer P0 , Readers P1 . . . Pn . Here, q = n. Registers could be binary or integer-valued Space complexity: n times that of a single register Time complexity: n steps
(shared variables) SRSW safe registers R1 . . . Rn 0; (1) Write(R , val ) executed by single writer P0 (1a) for all i {1 . . . n} do (1b) Ri val . (2) Readi (R , val ) executed by reader Pi , 1 i n (2a) val Ri (2b) return(val ). // Ri is readable by Pi , writable by P0

Construction 2: SRSW Regular to MRSW Regular is similar.

A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 32 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 3: Bool MRSW Safe to Integer MRSW Safe

For integer of size m, log (m) boolean registers needed. P0 writes value in binary notation; each of the n readers reads log (m) registers Space complexity log (m). Time complexity log (m).
(shared variables) boolean MRSW safe registers R1 . . . Rlog (m) 0; by P0 . (local variable) array of boolean: Val [1 . . . log (m)]; (1) Write(R , Val [1 . . . log m]) executed by single writer P0 (1a) for all i {1 . . . log (m)} do (1b) Ri Val [i ]. (2) Readi (R , Val [1 . . . log (m)]) executed by reader Pi , 1 i n (2a) for all j {1 . . . log m} do Val [j ] Rj (2b) return(Val [1 . . . log (m)]).
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 33 / 48

// Ri readable by all, writable

Distributed Computing: Principles, Algorithms, and Systems

Construction 4: Bool MRSW Safe to Bool MRSW Regular

q = 1. P0 writes register R1 . The n readers all read R1 . If value is before; Write is to write , then a concurrent Read may get either or 1 . How to convert to regular register? Writer locally tracks the previous value it wrote. Writer writes new value only if it diers from previously written value. Space and time complexity O (1). Cannot be used to construct binary SRSW atomic register.
(shared variables) boolean MRSW safe register: R 0; (local variables) boolean local to writer P0 : previous 0; (1) Write(R , val ) executed by single writer P0 (1a) if previous = val then (1b) R val ; (1c) previous val . (2) Read(R , val ) process Pi , 1 i n (2a) val R ; (2b) return(val ).
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 34 / 48

// R is readable by all, writable by P0 .

Distributed Computing: Principles, Algorithms, and Systems

Construction 5: Boolean MRSW Regular to Integer MRSW Regular

q = m, the largest integer. The integer is stored in unary notation. P0 is writer. P1 to Pn are readers, each can read all m registers. Readers scan L to R looking for rst 1; Writer writes 1 in Rval and then zeros out entries R to L. Complexity: m binary registers, O (m) time.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

35 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 5: Algorithm
(shared variables) boolean MRSW regular registers R1 . . . Rm1 0; Rm 1; // Ri readable by all, writable by P0 . (local variables) integer: count ; (1) Write(R , val ) executed by writer P0 (1a) Rval 1; (1b) for count = val 1 down to 1 do (1c) Rcount 0. (2) Readi (R , val ) executed by Pi , 1 i n (2a) count = 1; (2b) while Rcount = 0 do (2c) count count + 1; (2d) val count ; (2e) return(val ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

36 / 48

Distributed Computing: Principles, Algorithms, and Systems

Illustrating Constructions 5 and 6:

R
Zero out entries

Write val to R Write 1

R1

R2 R3

R val

Rm

Scan for "1"; return index. (bool MRSW reg to int MRSW reg) Scan for first "1"; then scan backwards and update pointer to lowestranked register containing a "1"
(bool MRSW atomic to int MRSW atomic)

Read( R )

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

37 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 6: Boolean MRSW regular to integer-valued MRSW atomic

Construction 5 cannot be used to construct a MRSW atomic register because of a possible inversion of values while reading. In example below, Read 2b returns 2 after the earlier Read 1b returned 3, and the value 3 is older than value 2. Such an inversion of read values is permitted by regular register but not by an atomic register. One solution is to require Reader to also scan R to L after it nds 1 in some location. In the backward scan, the smallest value is returned to the Read. Space complexity: m binary registers, Time complexity O (m)

Write1 a(R,2)
Write(R2 ,1)

Write2 a(R,3)
1

Write(R1 ,0) Write(R3 ,1) Write(R2 ,0) Write(R ,0)

a
Read(R 3,1) Read(R 1,0) Read(R 2,1)

Read(R 1,0) Read(R 2,0)

Pb

Read1 b(R,?) returns 3

A. Kshemkalyani and M. Singhal (Distributed Computing)

Read2 b(R,?) returns 2

CUP 2008 38 / 48

Distributed Shared Memory

Distributed Computing: Principles, Algorithms, and Systems

Construction 6: Algorithm
(shared variables) boolean MRSW regular registers R1 . . . Rm1 0; Rm 1. // Ri readable by all; writable by P0 . (local variables) integer: count , temp ; (1) Write(R , val ) executed by P0 (1a) Rval 1; (1b) for count = val 1 down to 1 do (1c) Rcount 0. (2) Readi (R , val ) executed by Pi , 1 i n (2a) count 1; (2b) while Rcount = 0 do (2c) count count + 1; (2d) val count ; (2e) for temp = count down to 1 do (2f) if Rtemp = 1 then (2g) val temp ; (2h) return(val ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

39 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 7: Integer MRSW Atomic to Integer MRMW Atomic

q = n, each MRSW register Ri is readable by all, but writable by Pi With concurrent updates to various MRSW registers, a global linearization order needs to be established, and the Read ops should recognize it. Idea: similar to the Bakery algorithm for mutex. Each register has 2 elds: R .data and R .tag , where tag = pid , seqno . The Collect is invoked by readers and the Writers The Collect reads all registers in no particular order. A Write gets a tag that is lexicographically greater then the tags read by it. The Writes (on dierent registers) get totally ordered (linearized) using the tag A Read returns data corresp. lexicographically most recent Write A Read gets ordered after the Write whose value is returned to it.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

40 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 7: Integer MRSW Atomic to Integer MRMW Atomic

(shared variables) MRSW atomic registers of type data, tag , where tag = seq no , pid : R1 . . . Rn ; (local variables) array of MRSW atomic registers of type data, tag , where tag = seq no , pid : Reg Array [1 . . . n]; integer: seq no , j , k ; (1) Writei (R , val ) executed by Pi , 1 i n (1a) Reg Array Collect (R1 , . . . , Rn ); (1b) seq no max(Reg Array [1].tag .seq no , . . . Reg Array [n].tag .seq no ) + 1; (1c) Ri (val , seq no , i ). (2) Readi (R , val ) executed by Pi , 1 i n (2a) Reg Array Collect (R1 , . . . , Rn ); (2b) identify j such that for all k = j , Reg Array [j ].tag > Reg Array [k ].tag ; (2c) val Reg Array [j ].data; (2d) return(val ). (3) Collect (R1 , . . . , Rn ) invoked by Read and Write routines (3a) for j = 1 to n do (3b) Reg Array [j ] Rj ; (3c) return(Reg Array ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

41 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 8: Integer SRSW Atomic to Integer MRSW Atomic

Naive solution: q = n. n replicas of R and the Writer writes to all replicas. Fails! Readi and Readj are serial, and both concurrent with Write. Readi could get the newer value and Readj could get the older value because this execution is non-serializable. Each reader also needs to know what value was last read by each other reader! Due to SRSW registers, construction needs n2 mailboxes for all reader process pairs Reader reads value set aside for it by other readers, as well as the value set aside for it by the writer (n such mailboxes; from Writer to each reader. Last Read [0..n] is local array. Last Read Values [1..n, 1..n] are the reader-to-reader mailboxes.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

42 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 8: Data Structure

1,1

1,2

1,n

R1

R2

Rn

SRSW atomic registers, one per process

2,1

2,2

2,n

mailboxes Last_Read_Values[1..n,1..n] (SRSW atomic registers)

n,1

n,2

n,n

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

43 / 48

Distributed Computing: Principles, Algorithms, and Systems

Construction 8: Algorithm
(shared variables) SRSW atomic register of type data, seq no , where data, seq no are integers: R1 . . . Rn 0, 0 ; SRSW atomic register array of type data, seq no , where data, seq no are integers: Last Read Values [1 . . . n, 1 . . . n] 0, 0 ; (local variables) array of data, seq no : Last Read [0 . . . n]; integer: seq , count ; (1) Write(R , val ) executed by writer P0 (1a) seq seq + 1; (1b) for count = 1 to n do (1c) Rcount val , seq . // write to each SRSW register (2) Readi (R , val ) executed by Pi , 1 i n (2a) Last Read [0].data, Last Read [0].seq no Ri ; // Last Read [0] stores value of Ri (2b) for count = 1 to n do // read into Last Read [count ], the latest values stored for Pi by Pcount (2c) Last Read [count ].data, Last Read [count ].seq no Last Read Values [count , i ].data, Last Read Values [count , i ].seq no ; (2d) identify j such that for all k = j , Last Read [j ].seq no Last Read [k ].seq no ; (2e) for count = 1 to n do (2f) Last Read Values [i , count ].data, Last Read Values [i , count ].seq no Last Read [j ].data, Last Read [j ].seq no ; (2g) val Last Read [j ].data; (2h) return(val ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

44 / 48

Distributed Computing: Principles, Algorithms, and Systems

Wait-free Atomic Snapshots of Shared Objects using Atomic MRSW objects

Given a set of SWMR atomic registers R1 . . . Rn , where Ri can be written only by Pi and can be read by all processes, and which together form a compound high-level object, devise a wait-free algorithm to observe the state of the object at some instant in time. The following actions are allowed on this high-level object. Scani : This action invoked by Pi returns the atomic snapshot which is an instantaneous view of the object (R1 , . . . , Rn ) at some instant between the invocation and termination of the Scan. Updatei (val ): This action invoked by Pi writes the data val to register Ri .
P1 UPDATE data seq_no old_snapshot R1 Scan Scan Pn UPDATE

data seq_no old_snapshot Rn

snapshot object composed of n MRSW atomic registers

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

45 / 48

Distributed Computing: Principles, Algorithms, and Systems

Wait-free Atomic Snapshot of MRSW Object

To get an instantaneous snapshot, double-collect (2 scans) may always fail because Updater may intervene. Updater is inherently more powerful than Scanner To have the same power as Scanners, Updater is required to rst do double-collect and then its update action. Additionally, the Updater also writes the snapshot it collected, in the Register. If a scanners double collect fails (because some Updater has done an Update in between), the scanner can borrow the snapshot recorded by the Updater in its register. changed [k ] tracks the number of times Pk spoils Pi s double-collect. changed [k ] = 2 implies the second time the Updater spoiled the scanners double-collect, the update was initiated after the Scanner began its task. Hence the Updaters recorded snapshot is within the time duration of the scanners trails. Scanner can borrow Updaters recorded snapshot. Updaters recorded snapshot may also be borrowed. This recursive argument holds at most n 1 times; the nth time, some double-collect must be successful. Scans and Updates get linearized. Local and shared space complexity both are O (n2 ). Time complexity O (n2 )
A. Kshemkalyani and M. Singhal (Distributed Computing) Distributed Shared Memory CUP 2008 46 / 48

Distributed Computing: Principles, Algorithms, and Systems

Wait-free Atomic Snapshot of MRSW Object: Algorithm

(shared variables) MRSW atomic register of type data, seq no , old snapshot , where data, seq no are of type integer, and old snapshot [1 . . . n] is array of integer: R1 . . . Rn ; (local variables) array of int: changed [1 . . . n]; array of type data, seq no , old snapshot : v 1[1 . . . n], v 2[1 . . . n], v [1 . . . n]; (1) Updatei (x ) (1a) v [1 . . . n] Scani ; (1b) Ri (x , Ri .seq no + 1, v [1 . . . n]). (2) Scani (2a) for count = 1 to n do (2b) changed [count ] 0; (2c) while true do (2d) v 1[1 . . . n] collect (); (2e) v 2[1 . . . n] collect (); (2f) if (k , 1 k n)(v 1[k ].seq no = v 2[k ].seq no ) then (2g) return(v 2[1].data, . . . , v 2[n].data); (2h) else (2i) for k = 1 to n do (2j) if v 1[k ].seq no = v 2[k ].seq no then (2k) changed [k ] changed [k ] + 1; (2l) if changed [k ] = 2 then (2m) return(v 2[k ].old snapshot ).

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

47 / 48

Distributed Computing: Principles, Algorithms, and Systems

Wait-free Atomic Snapshots of Shared Objects using Atomic MRSW Objects

Double collect Collect Collect

Pi (a) Double collect sees identical values in both Collects j P i j changed[j]=1

Pj writes in this period

j changed[j]=2
Pj writes in this period Pj writes

Pj

Pj writes

(b) P_js DoubleCollect nested within P_is SCAN. The DoubleCollect is successful, or P_j borrowed snapshot from P_ks DoubleCollect nested within P_js SCAN. And so on recursively, up to n times.

A. Kshemkalyani and M. Singhal (Distributed Computing)

Distributed Shared Memory

CUP 2008

48 / 48