UNIT - V

Distributed Scheduling: Issues in Load Distributing, Components of a Load Distributed Algorithm, Stability, Load Distributing Algorithms,
Requirements for Load Distributing, Task Migration, Issues in task Migration Distributed Shared Memory: Architecture and Motivation,
Algorithms for Implementing DSM, Memory Coherence, Coherence Protocols, Design Issues

1. INTRODUCTION TO DISTRIBUTED SCHEDULING

Distributed system consists of a collection of autonomous computers, connected by a
LAN. In a locally distributed system, there is a good possibility that several computers are
heavily loaded while others are ideal or lightly loaded. If we can move jobs around, the overall
performance of the system can be maximized.
A distributed scheduler is a resources management component of a distributed operating
system that focuses on transparently redistributing the load of the system among the computers
to maximize the overall performance.
In distributed system, at each independent computer, tasks to be executed arrive at
random intervals and their execution time (Burst Time) is also random. So, some of the systems
are heavily loaded while others are idle. To balance load among distributed systems, some of the
tasks from heavily loaded systems are migrated to idle systems.
Consider a system of N identical, independent M/M/1 servers. Let P be the probability that the
system is in state in which at least 1 task is waiting for service and at least 1 server is idle. Let P
be the utilization of each servers. For moderate system utilization, Value of P is high, i.e. at least
1 node is idle. Hence, performance can be improved by sharing of task.

2. ISSUES IN LOAD DISTRIBUTING

The issue in load distribution in Distributed System are:
(i). LOAD: Load on a system/node can be determined by CPU queue length. The CPU queue
contains processes (or tasks) that need to be executed. When tasks/processes are moved from
the heavily loaded systems to others idle or less loaded system, then load can be balanced.
 When idle node accepts tasks, other heavily loaded system will transfer few of their tasks
to it. But, when many systems transfer their task to it, then may cause the idle node to be
highly loaded. This affects performance. The solution for this problem is, artificially
increment the queue length when a task is accepted for transfer from remote node. When the
CPU queue length reaches the average limit, then stop accepting the tasks from remote node.
(ii). Types of Load Distributing Algorithms: The basic function of load distributing algorithms
is to transfer load (tasks) from heavily loaded systems to idle or lightly loaded systems. The
load distributing algorithms are broadly categorized into:
 Static load distribution algorithms: Decisions are hard-coded into an algorithm with a
priori knowledge of system.
 Dynamic load distribution: It uses system state information such as task queue length,
processor utilization. The collection of load information itself can add load on the system
because messages need to be exchanged among systems. These algorithms will continue
the collecting state information even at high loads also.
 Adaptive load distribution: It adapts the approach based on system state. Adaptive
distribution algorithms may stop collecting state information at high loads to avoid
imposing additional overhead on the system.

(iii). Load Balancing vs. Load sharing:

Load balancing: The concept is to equalize load on all the nodes in distributed system.
Transfer tasks even if a node is not heavily loaded so that queue length on all nodes are
approximately equal. It results in more number of tasks transfers and might degrade
performance.
Load sharing: It tries to reduce burden of an overloaded node. In this policy, tasks are
transferred only when the queue length exceeds a certain threshold. In this, less number of
task transfers takes place.
One solution is to use anticipatory task transfer. In this policy, tasks are transferred from
overloaded nodes to ones that are likely to become idle/lightly loaded.

(iv). Preemptive vs. Non-preemptive:

Preemptive transfers: In this policy, the task that is partially executed is also transferred to
balance the load. This type of transfer is expensive due to collection of task's state
information.
Non-preemptive transfers: In this policy, the tasks that have not begun execution only
transferred to balance the load.

3. COMPONENTS OF LOAD DISTRIBUTION ALGORITHMS

Load distribution algorithms have four components. They are:
Transfer policy: It determines whether a node is in a suitable state to participate in a task
transfer. In other words it identify the sender node based on threshold value.
Selection policy: It determines which task should be transferred. The best way is to select newly
arrived process than partially executed process.

Location policy: It determines the receiver node to accept a sent task from sender node. A
widely used method for finding a suitable node is through polling. In polling, a node polls
another node to find out whether it is a suitable node for load sharing. Nodes can be polled either
serially or in parallel (e.g., multicast). A node can be selected for polling either randomly, based
on the information collected during the previous polls or on a nearest-neighbor basis. An
alternative to polling is to broadcast a query to find out if any node is available for load sharing.

Information policy: It is responsible for deciding when information about the states of other
nodes in the system should be collected, where it should be collected from, and what information
should be collected. Most information policies are one of the following three types:

Demand-driven: In this class of policy, a node collects the state of other nodes only when it
becomes either a sender or a receiver (decided by the transfer and selection policies at the node),
making it a suitable candidate to initiate load sharing. Note that a demand-driven information
policy is inherently a dynamic policy, as its actions depend on the system state. Demand-driven
policies can be sender-initiated, receiver-initiated, or symmetrically initiated.

 In sender-initiated policies, senders look for receivers to transfer their load.

 In receiver-initiated policies, receivers ask for load from senders.
 A symmetrically initiated policy is a combination of both, where load sharing actions
are triggered by the demand for extra processing power or extra work.

Periodic: In this class of policy, nodes exchange load information periodically. Based on the
information collected, the transfer policy at a node may decide to transfer jobs.

State-change-driven: In this policy, nodes broadcast information when their state changes by a
certain degree.

4. STABILITY
There are two views of stability of system in distributed scheduling. They are:

The queuing-theoretic perspective: When the arrival rate of jobs to a system is greater
than the rate at which it performs work, then CPU queue length grows and leads to unstable. By
transferring the new jobs to other system, we can maintain stability.

The algorithmic perspective: If an algorithm performs indefinite fruitless actions, then it

is called unstable. For example, when tasks transferred to receiver, may increase the receiver’s
queue length to the point of overload. As a result these transferred tasks need to be transferred to
other receiver node. If the same happens, instead of executing the tasks, they are moved from
one node to other node. A distributed scheduling algorithm is effective and stable if it execute
the tasks with little movements.

5. LOAD DISTRIBUTING ALGORITHMS

Over-loaded node is a node whose queue length exceed threshold, T. Under-loaded node is a
node whose queue length less than threshold, T. Over-loaded node transfer tasks to under-loaded
nodes. The load distributing algorithms are the following types:
• Sender initiated algorithm
• Receiver initiated algorithm
• Symmetrically initiated algorithm
Sender Initiated Algorithms
An overloaded node (sender) tries to send a task to an under loaded node (receiver).
Information policy: Demand-driven: Once sender node is identified, then it collects load
information from other nodes. The CPU queue length is used as load information metric.
Transfer policy: If CPU queue exceeds threshold T for any node, then it becomes sender. It
always checks for T value to determine sender, when a new task arrives at any node.
Selection policy: Newly arrived tasks or newly started processes without requiring preemption
are selected for transfer from sender to receiver node.
Location policy: It identifies the receiver node to accept load from sender node. There are three
different approaches. They are:
1. Random: Select any node to transfer the task at random. The selected node X may be
overloaded. If transferred task is treated as new arrival then X node may transfer the task
to other node.
2. Threshold: Nodes are sequentially polled in random order. A sender node performs
polling until a receiver is found. If none is a receiver then the sender commits to the task.
3. Shortest: A random subset of nodes is polled and the one with the shortest queue length
is chosen. Main aim is to choose the best receiver node rather than the first suitable one.
Stability of the algorithm:
 In high load scenarios there may not (likely) be any receivers, but polling would still keep
on, triggered more and more frequently as task arrival rate increases.
 The algorithm overhead can become higher and higher, eventually overcoming the
service capacity of the system, causing system instability.

Receiver Initiated Algorithms

In receiver-initiated algorithms, load-distributing activity is initiated from an under-loaded node
(receiver), which tries to get a task from an overloaded node (sender).
Transfer policy: A node is identified as a sender if its queue length exceeds the threshold T. A
node becomes receiver if its queue length falls below the threshold T.

Selection policy: It prefers to transfer newly arrived process than partially executed processes.

Location policy: It selects a node at random and polls it to determine whether transferring a task
would place its queue length below the threshold level. If yes, then the polled node transfers a
task. Otherwise, other node is selected at random, and the procedure is repeated until either a
node that can transfer a task (a sender) is found or a static poll limit number of tries has failed to
find a sender.

Information policy: The information policy is demand driven, since polling starts only after a
node becomes a receiver.
Stability: Receiver-initiated algorithms do not cause system instability because, at high system
loads, a receiver is likely to find a suitable sender within a few polls. Consequently, polls are
increasingly effective with increasing system load, and little waste of CPU capacity results.
Drawback: If round-robin CPU scheduling algorithm used, a newly arrived task is quickly
provided a quantum of service. Consequently, most transfers are preemptive and therefore
expensive.

Symmetrically initiated algorithms

Under symmetrically initiated algorithms," both senders and receivers initiate load-distributing
activities for task transfers. These algorithms have the advantages of both sender and receiver-
initiated algorithms. At low system loads, the sender-initiated component is more successful at
finding under-loaded nodes. At high system loads, the receiver-initiated component is more
successful at finding overloaded nodes. However, these algorithms may also have the
disadvantages of both sender and receiver-initiated algorithms. As with sender-initiated
algorithms, polling at high system loads may result in system instability. As with receiver-
initiated algorithms, a preemptive task transfer facility is necessary. A simple symmetrically
initiated algorithm can be constructed by combining the transfer and location policies described
for sender-initiated and receiver-initiated algorithms.
Transfer policy: The transfer policy is a threshold policy that uses two adaptive thresholds.
These thresholds are equidistant from the node's estimate of the average load across all nodes.
For example, if a node's estimate of the average load is 3, then the lower threshold = 2 and the
upper threshold = 4. A node whose load is less than the lower threshold is considered a receiver,
while a node whose load is greater than the upper threshold is considered a sender. Nodes that
have loads between these thresholds lie within the acceptable range, so they are neither senders
nor receivers.

Location policy: The location policy has the following two components:
  Sender-initiated component: In this component, the sender node broadcasts a REQUEST
message to find receiver node. If receiver node found, then sender chooses best task to
transfer.
 Receiver-initiated component: In this component, the receiver node is one with load less
than the lower threshold value is chosen as receiver. Once receiver identified, it
broadcasts a ACCEPT message to find sender node. If sender node found, then sender
chooses best task to transfer to receiver node.

Selection policy: It prefers to transfer newly arrived process than partially executed processes. A
partially executed task is transferred only if its response time improves after moving.

Information policy: The information policy is demand-driven. When the communication

network is heavily/lightly loaded, then the acceptable load at each node can increase/decrease to
adopt to load.

Adaptive algorithms
Each node maintains three lists such as senders list, receivers list and OK list. Initially all the
nodes are added to receivers list and sender list is empty. OK list contain all the nodes whose
load is in between lower threshold (LT) and upper threshold (UT). A node moves from one list
to other list based on the load values. This algorithm adapts to the changes in load.

Transfer policy: The transfer policy is a threshold policy that uses two adaptive thresholds. The
transfer policy is triggered when new task originate or a task departs. A node whose load is less
than the lower threshold (LT) is considered a receiver, while a node whose load is greater than
the upper threshold (UT) is considered a sender. Nodes that have loads between these thresholds
lie within the acceptable range, so they are neither senders nor receivers.

Location policy: The location policy has the following two components:

 Sender-initiated component: In this component, the sender node send REQUEST

message to all the nodes present in the receivers list. If receiver node found, then sender
chooses best task to transfer.
 Receiver-initiated component: In this component, the receiver node send TOLOW
message to all the nodes present in the senders list. If sender node found, then sender
chooses best task to transfer to receiver node.

Selection policy: The sender-initiated component considers only newly arrived tasks for transfer.
The receiver-initiated component can make use of any of the general selection policies.

Information policy: The information policy is demand-driven, as the polling activity starts
when a node becomes a sender or a receiver.
6. REQUIREMENTS FOR LOAD DISTRIBUTING
A distributed scheduler is a resource management component of a distributed operating system
that focuses on transparently redistributing the load of the system among the individual units to
enhance overall performance. Users submit tasks at their host computers for processing. The
need for load distribution arises in such environments because, due to the random arrival of tasks
and their random CPU service time requirements, there is a good possibility that several
computers are idle or lightly loaded and some others are heavily loaded, which would degrade
the performance. In real life applications there is always a possibility that one server or system is
idle while a task is being waited upon at another server.

7. TASK MIGRATION & ITS ISSUES

Task Migration: It is the movement of an executing task from one computer to other computer
in distributed computing system.

Benefits of task Migration

 Load Balancing: -Improve performance for a distributed computing system overall or a

distributed application by spreading thee load more evenly over a set of host.
 Reduction in Communication overhead: - by locating on one host a group of tasks with
intensive communication amongst them.
 Resource access:-not all resources are available across the network, a task may need to
migrate in order to access a special device, or to satisfy a need for a large amount of
physical memory
 Fault tolerance:-allowing long running processes to survive the planned shutdown or
failure of a host.

Steps involved in task migration

 Suspending the task on the source.

 Extracting and transmitting the state of the task to destination.
 Reconstructing the state on the destination
 Deleting the task on the source and resuming the task’s execution on the destination.

Task Migration Issues: There are three issues in the task migration. They are:
i. State Transfer: There are two important issues to be considered. They are:
The cost to support remote execution may be high, which includes delays due to freezing
the task, obtaining and transferring the state and unfreezing the task. The lengthy freezing
of task during migration can cause abortion of tasks interacting with it due to timeouts.
 Residual dependencies: Refer to the amount of resources a host of a migrated task
continues to dedicate to service requests from the migrated task. They are undesirable for
three reasons:-Reliability, performance and complexity.
ii. Location Transparency: Task Migration should hide the locations of tasks. Location
transparency in principle requires that name (process name, file name) be independent of
their locations (host names). Uniform name spacer throughout the system should be used.
iii. Structure of a migration mechanism: Generally there will be interaction between
the task migration mechanism, the memory management system, the inter-process
communication mechanism and the file system.
The mechanism should be designed to be independent for one another so that if one
mechanism’s protocol changes; the other’s need not the migration mechanism can be
turned off without interfering with other mechanisms.
iv. Performance: The performance of implementing task migration in different type of
systems varies due to different hardware, operating systems, IPC mechanisms, file
systems, etc.

8. DISTRIBUTED SHARED MEMORY: Architecture and Motivation

Shared memory is the memory block that can be accessed by more than one program. A
shared memory concept reduces redundancy by avoiding duplicate data storage. Distributed
shared memory (DSM) is a form of memory architecture where physically separated memories
can be connected and treated as one logically shared address space. A distributed-memory
system also known as loosely coupled systems consists of multiple independent processing nodes
with local memory modules which are connected by a general interconnection network. The
shared memory model provides a virtual address space that is shared among all computers in a
distributed system. An example of this layout can be seen in the following diagram.
 Figure: Architecture of Distributed Shared Memory

Every node consists of one or additional CPU’s and a memory unit. High-speed communication
network is used for connecting these nodes. When a process accesses data in the shared address
space, a mapping manager maps the shared memory address space to the physical memory. To
reduce delays due to communication latency, DSM may move data at shared memory address
from a remote node to the node that is accessing data.

Advantages of Distributed Shared Memory (DSM):

 DSM hides data movement and provides a simpler abstraction for sharing data. This
makes Programmers don't need to worry about memory transfers and can design parallel
algorithms.

 DSM allows the passing of complex structures by reference between nodes. This
simplifies development of algorithms for distributed applications.

 DSM takes advantage of "locality of reference" by moving the entire page containing the
data referenced rather than just the piece of data. This reduce burden on communication
network.

 Building DSM is cheaper than multiprocessor systems. It can be implemented using

normal hardware and do not require anything complex to connect the shared memory to
the processors.
  Larger memory sizes are available to programs, by combining all physical memory of all
nodes. This large memory will not incur disk latency due to swapping like in traditional
distributed systems.

 Unlimited number of nodes can be used. Unlike multiprocessor systems where main
memory is accessed via a common bus, thus limiting the size of the multiprocessor
system.

 Programs written for shared memory multiprocessors can be run on DSM systems.

9. ALGORITHMS FOR IMPLEMENTING DSM

The common challenges to implement DSM are:

 Tracking of the memory address (location) of data stored remotely in shared memory.
 To reduce the communication delays and high overhead associated with the references to
remote data.
 Controlling the concurrent access of the data shared in DSM.

Based on these challenges there are algorithms designed to implement distributed shared
memory. There are four algorithms. They are:
(i). Central Server Algorithm:
All shared data is maintained by the central server. Other nodes of the distributed
system request the server for reading and writing data. The server serves these
requests with acknowledgment messages. These acknowledgment messages are used to
provide the status of the data request.
It is simpler to implement but the central server can become bottleneck. To overcome
this, shared data can be distributed among several servers. This distribution can be
addressed by using a mapping function to locate the appropriate server.
(ii). Migration Algorithm:
Instead of using a central server to serve each request, in migration algorithm the
requested data is migrated (shipped) to request node which allows subsequent access to
the same data can be performed locally. This algorithm is good if when a system accesses
the same block of data multiple times.
Only one node is able to access the shared data element at a time and the whole block
is migrated to that node. Also, this algorithm is more prone to thrashing due to the
migration of data items upon request by the node.

(iii). Read Replication Algorithm

In the read replication algorithm, the data block that is to be accessed is
replicated and only reading is allowed in all the copies. If a write operation is to be
done, then all read access is put on halt till all the copies are updated.
Overall system performance is improved as concurrent access is allowed. But
write operation is expensive due to the requirement of updating all blocks that are
shared to maintain concurrency. All copies of data element are to be tracked to
maintain consistency.

(iv). Full Replication Algorithm

 It is an extension of read replication algorithm which allows multiple nodes to have both
read and write access to shared data blocks.
  Since many nodes can write shared data concurrently, the access to shared data must be
controlled to maintain consistency.
 To maintain consistency, it can use a gap free sequences in which all nodes wishing to
modify shared data will send the modification to sequencer which will then assign a
sequence number and multicast the modification with sequence number to all nodes that
have a copy of shared data item.

10. MEMORY COHERENCE

Memory coherence is a desirable condition in which a data value stored at many memory
locations each corresponding to one processing element in a multi-core processor should always
contain the same cached data. In other words, if X=10 and stored this X=10 at memory locations
of all processing elements, then at any time all copies of X value should be same. That is if any
processing element modifies its copy of X value, then all other copies of X should contain the
same.
In a uniprocessor system, there is only one processing element doing all the work and
perform read or write at a given memory location. As a result, when a value is changed, all
subsequent read operations of the corresponding memory location will see the updated value.

On the other hand, in multiprocessor systems, two or more processing elements work
concurrently. Once in a while they will simultaneously access the same memory location. As
long as no processing element alters the data at the affected location, all of those elements can
share and cache the data without any problems. But if one of the processing elements changes
the data in the shared location and does not inform the others of the change, the other elements
might use the outdated version of the data that remains in their local caches.

Consequently, some scheme is required to notify all the processing elements of changes
to shared values; such a scheme is known as a memory coherence protocol, and if such a
protocol is employed the system is said to have a coherent memory.
11. COHERENCE PROTOCOLS
To provide concurrent access, DSM systems make use of data replication (duplications), where
copies of data are maintained at all the nodes accessing the data. A fundamental problem with
data replication is the difficulty in ensuring that all copies have the same information and that
nodes do not access outdated data. In other words, a protocol to keep replicas coherent is needed.
Two basic protocols to maintain coherence are the write-invalidate protocol and the write-update
protocol.

WRITE-INVALIDATE PROTOCOL:
Let us consider three processors P1, P2, and P3 having a consistent copy of data element ‘X’ in
their local cache memory and in the shared memory. That is all the caches and shared memory
have the same value. This is shown in the above diagram. When processor P1 modifies the X
value to X1, then it writes X1 in its cache memory using write-invalidate protocol. This
protocol makes all other copies are invalidated via the bus. It is denoted by ‘I’. This is shown in
the below diagram. Invalidated blocks are also known as dirty, i.e. they should not be used. This
protocol is inefficient if many nodes frequently access the data which is modified more
frequently because an updated object not copied back to many nodes immediately after every
invalidation.

WRITE-UPDATE PROTOCOL

In the write-update method, a write to a shared data causes all copies of that data to be updated.
This is shown in the below diagram. This approach is more difficult to implement than the
previous approach, as a new value has to be sent instead of sending invalidation messages. This
protocol can be expected to generate considerable network traffic. But, it is better approach, if
the frequency of modified data is high.
12. DESIGN ISSUES
The two issues to be considered in the design of a DSM system are granularity and page
replacement.

They are important because the efficiency Of DSM depends on the effectiveness Of the size
chosen for granularity and the protocol used for page replacement.

Granularity: Granularity refers to the size of the shared memory unit. When the page size
matches with the underlying hardware or the memory management system page size, it allows
for the integration Of DSM and the memory management systems. By integrating DSM with the
underlying memory management system, a DSM system can take advantage of the built-in
protection mechanism to detect incoherent memory references, and use built-in fault handlers to
prevent and recover from inappropriate references.

The design issue is to chose either large page size or small page size. A large page size for the
shared memory unit will take advantage of the locality of reference exhibited by processes. By
transferring large pages, less overhead is incurred due to paging activity and processing
communication protocols. However, the larger the page size, the greater the chance for
contention to access a page by many processes. Smaller page sizes are less apt to cause
contention as they reduce the likelihood of false sharing. False sharing of a page occurs when
two different data items, not shared but accessed by two different processes, are allocated to a
single page.

Page Replacement: A memory management system has to address the issue of page
replacement because the size of physical memory is limited. In DSM systems that support data
movement, traditional methods such as least recently used (LRU) page replacement algorithm
cannot be used directly. Other approaches are required to implement LRU.

- O0Oo -