ISSN (Print) : 0974-6846

Indian Journal of Science and Technology, Vol 10(28), DOI: 10.17485/ijst/2017/v10i28/112308, July 2017 ISSN (Online) : 0974-5645

Improving Performance of Distributed Shared

Memory (DSM) on Multiprocessor Framework with
Software Approach
Hemant D. Vasava1,2 and Jagdish M. Rathod2
School of Engineering, RK University, Rajkot – 360020, Gujarat, India; [email protected]
1

2
BVM Engineering College, Vallabh Vidyanagar, Anand – 388120, Gujarat, India; [email protected]

Abstract
Objectives: To design Distributed Shared Memory (DSM) for the multiprocessor distributed framework using a different
software parametric approach that provides significant performance improvement against convention software
based architectures. Methods/Statistical Analysis: Software distributed shared memory can be architected by using
a different concept of an operating system, by utilizing a programming library and by extending underlying virtual
address space architecture. It incorporates various design options like granularity, consistency model, implementation
level, data organization, algorithms, protocols, etc. We have proposed few software parameter choices and impact
which gives significant performance improvement compared to past designs to manage software distributed shared
memory. This paper also discusses various issues that exist while moving toward software distributed shared memory
implementation. Findings: There are two methodologies by which it is possible to achieve distributed shared memory
design are first in hardware like cache coherence circuits and network interfaces and the second is software. Here the
proposed system architecture makes major impact on programming, performance, design and cost. An algorithm is
designed such a unique manner which resides in memory controller and make efficient global virtual memories. It is using
variable as granularity which are shared that is more flexible for complex data structure and large database. It is defined
using unique identifier which makes its mapping and retrieval more manageable using proposed consistency mechanism.
Application/Improvements: Distributed shared memory optimization is a most important area of improving distributed
system performance. By taking care of good choice on underlying issues and according to system’s design requirement,
it possible to gain advantages of improved architecture which can be more used for various distributed applications where
shared data plays a major role.

Keywords: Distributed Shared Memory, Distributed Systems, Granularity, Interprocess Communication, Virtual Address
Space

1. Introduction such systems are transparent, means that the underlying

distributed address space is completely hidden to its users.
A Distributed Shared Memory (DSM) system sometimes However, software distributed shared memory system is
called a multicomputer system where multiple indepen- implemented at the programming library or language.
dent process nodes with locally allocated memory are Its language paradigm or libraries are not transparent,
connected by a general high speed network. A Software but developers need to program them differently. These
DSM systems can be implemented can be at the operating shared memory systems offer a more portable approach
system level, or by using a programming library and can for implementations. A Distributed Shared Memory
be thought of as expending the underlying virtual address (DSM) system uses a shared memory model of a physi-
space architecture that depends upon the requirements.1 cally distributed independent memory for Interprocess
When memory is implemented insight operating system, Communication (IPC).

*Author for correspondence

 Improving Performance of Distributed Shared Memory (DSM) on Multiprocessor Framework with Software Approach

A shared memory system controls separate, inde- distributed architecture systems for reasons of fault tol-
pendent memory into virtual shared segments which is erance, availability, and increasing computational power.
distributed amongst all nodes, main memory and distrib- Motivation for shared memory is the recent trend in design
utes all memory between nodes.2,3 It is required to choose distributed systems using a collection of disk less main
an appropriate coherence protocol in accordance with a computing servers like workstations and at other data
consistent model that maintains the memory coherence. servers or file server architectures.3 In this framework
Conventionally, all memory programming can create the data for executing program has been cached from
bottlenecks because of multiple accesses by process in the main server or remote node. There can be too kind
critical regions in memory address space. This is because of issues here. First is a scheduling decision of where of
of large amounts of data processing will requires integ- execution of program is best left to a higher level policy
rity, consistency and stability of the application and data making decision. And the second is the task of fetching
with system performance. The pre and post development in the required program code and data.2 In this paper we
efforts can be more significant for shared data items and have also discussed our experience of implementing a
message communication of distributed programming software DSM system using granularity based approach.
to access its operating system kernel, thus increasing This is also discussed various designing parameters and
overhead to the system at all levels.2 Figure 1 shows dis- its impact on software distributed shared memory archi-
tributed memory architecture. This implementation of tecture. Various suitable parameters or design choices
creating asynchronous application for distributed tasks taken into account while designing is considered accord-
which can reduce the performance in bottlenecks and ing to granularity. We have presented an implementation
improve the consistency and scalability in the memory of this model in the context of a Linux operating system.
system.2 The Distributed Shared Memory (DSM) can
operate on a uniform and non uniform memory archi-
tecture that have different access times to processor’s
2. DSM Requirements
memory. The core factors like consistency, synchroniza- In any distributed network system sharing of data will be
tion and coherency are required to consider when going an essential requirement. Also the recent trends of tech-
designing distributed memory architecture for large scale nology advancement of processor configuration, user will
data processing in a distributed environment. After this still demand for more performance within parallel run-
right pre and post development programming efforts and ning processes.2 So, it is required to upgrade processor
efficient tuning like memory issues, it is possible to mini- design to get good performance with the distributed
mize all the bottlenecks in the system.2 shared memory environment.4 However, the single pro-
cessor design provides a way to create environment
in that multiple processors run simultaneously to get
good performance at low cost. It is better to run 15 or
more inexpensive processors in parallel rather to design
or buy 15 times faster processor.5-8 This type of design
is required to decrease various overhead in distributed
virtual address space which can be accessed by simulta-
neous parallel running processors.9,10 Solutions to such
parallel and distributed communication is DSM which
required investigating new architecture design by opti-
mizing imbalance between processors and memory
which should be upgradable eventually.11

3. Implementation Level
Figure 1. Distributed Memory Architecture.
The level where the Distributed Shared Memory (DSM) is
The distributed shared memory programmings is implemented is one of most important decisions for design-
well understood and focus the interest in parallel and ing distributed shared memory system because it affects

2 Vol 10 (28) | July 2017 | www.indjst.org Indian Journal of Science and Technology
 Hemant D. Vasava and Jagdish M. Rathod

both overall programming and system performance with hardware components, which also utilizes software
its cost.4,12-14 Distributed shared memory shared space is involvements.4,16 Another is software methodology
distributed among local node boundaries. It is required to which also provide flexibility in organization of shared
execute lookup on each access to find whether requested data segment in different ways. Like page based method
data are in local memory or located remotely.2 If it is not organization of shared memory into fixed size pages. The
in local memory, then the node must fetch data into local object based or shared variable based approach organi-
memory. Here, the system must require executing an zation of the shared data region storing data of variable
action to preserve and manage coherence view of shared sizes. One other commonly described implementation
data items.3 So, both lookup and action can be imple- type that uses a tuple space where the unit is sharing is
ment using software, hardware, and combination of a tuple.
both. According to this implementation choice design
falls into three group software, hardware, and hybrid
methodology.
5. Achieving Methods and
The implementation choice usually depends Implementation
on requirements and price performance trade-offs. From the usual ways of architecting DSM system like
Software based distributed shared memory implemen- hardware as cache coherence configuration and network
tation originated during 1980s and is to provide the interfaces and based on application , software distributed
shared memory parameters to the developer using vari- shared memory systems can be implemented in program-
ous factors. Traditionally, it can be architects in user ming library, at the operating system level or at underlying
level, the OS, at run time library routines or using a pro- memory architect can be used to achieve the goal with
gramming language.13 Thus, if the requested data is not different software parameters.19,20 Implementation at the
in local memory, then memory controller will retrieve operating system level will remain transparent to pro-
the data or address of local memory of another node, or grammer that means distributed shared memory will
disk of remote node by mapping. Typical representatives remain completely hidden to its user but system can
of approach are IVY, Mermaid, Munin, Midway, Tread take benefits of design. Here, software distributed shared
Marks, Orca, Linda, Mirage and Clouds. IVY is few ini- memory systems implemented as language or library level
tially proposed software distributed memory solutions are not completely transparent and hidden to program-
with different design parameters.15-17 This design gener- mers usually they have to do programming separately.
ally more flexible in comparison with hardware support But advantage is that these kinds of systems are more
which enable better tailoring of consistency model to the portable and easy updatable software approach to DSM
application characteristics but cannot compete with hard- system design.
ware methodology in terms of cost and performance.18 The DSM systems implementation of virtual shared
memory model of independent physically distributed
4. Software DSM Model memory system. If we consider one of its design param-
eter granularity, then by using three ways it is possible to
There are no of reasons to program DSM for distrib- design a software DSM like page based method that use
uted environment are stated in the past. Like complex the system’s virtual address space, shared variable method
and large data structures can be easily communicated.2 which utilize programming routines to share and access
Shared data item becomes a big issue which can easily shared variables and object based approach which access
managed in software, but not in hardware independent shared data through object oriented concepts.21 It also
multiprocessor distributed system. It gives transparent involves various other implementation choices depend
process to process communication. Shared data programs upon its design requirements which includes distributed
are usually smaller and easier to understand. Program shared memory algorithms, level of implementation like
implementation of shared memory is a well understood in hardware, software or hybrid, critical region semantics,
problem. Compact program design, easy implementa- semantics of replication level, naming scheme specifica-
tion and expansion. The distributed shared data memory tion has to be used to access remotely located shared data,
can be implemented two different ways one by using replication location for optimization, system consistency

Vol 10 (28) | July 2017 | www.indjst.org Indian Journal of Science and Technology 3
 Improving Performance of Distributed Shared Memory (DSM) on Multiprocessor Framework with Software Approach

model, the granularity of shared data, remote access and 5.2 Data Organization
caching/replication control of hardware or software, dis-
To work distributed system efficiently it is required to
tributed shared memory controlled by virtual memory
organize and manage data using proper conditions. Any
management software, OS or language run time system
process can share large amount of shared variables and
etc.2 According to design choices and issues memory can
data structures which are utilize by more than one node
be implemented.22 Here, subsequently our distributed
in the environment.2 So, it is required to decide changes
shared memory system design considerations systems
from paging on the network to how to maintain a poten-
are explained in brief with its advantages over conven-
tially distributed, replicated data items consisting of the
tional designs which provide another methodology to
shared variables and data structures. Granularity is one
implement DSM that provide significant performance
of the parameters are the size of data unit that exists
improvement. However the design choice may vary
in distributed systems.2 This is a very important deci-
according to system design requirement and perfor-
sion which makes another big impact on distributed
mance, cost and scaling parameters.
system design. Based on granularity parameter data can be
organized using shared variables and data structure, pages
5.1 Algorithm Architecture or by objects as the unit for data transfer. All granular-
An algorithm for implementing Distributed Shared ity design having its own advantages and disadvantages. A
Memory (DSM) is the most significant part of the distributed shared memory using shared variable is to let
architecture. Conceptually, these algorithms make the applications or developer decide as to which variables
extension of the local virtual address spaces to extend are to be shared in an environment and memory control-
multiple host machine connected by a Local Area ler will manage a database of the shared variables and
Network (LAN), and many of them can be integrated in data structures in the distributed framework.2 By using a
virtual memory systems. The distributed shared memory memory controller the developer to explicitly declare
implementation algorithm deal, with two basic issues first data variables are to be shared globally in the applica-
is static and dynamic allocation of shared data across dis- tion2. It removes the issues of thrasing and false sharing
tributed network system to minimize its latency.2 Another issues. All process of different independent machine can
is to preservation of coherence view of shared data when access shared variables and data structures by requesting
designer minimize coherence management overhead. through memory controller.
There are two strategies can be used to manage shared
data which are migration and replication.23 In migra- 5.3 Memory Controller
tion implementation methodology only one copy of a
Memory controller mapping and manage virtual memory
data exists at time in the system while in replication mul-
used by different processes to communicate via shared
tiple copies of similar data item reside at different local
variables to each other. It will make interprocess com-
memories. So, to decrease memory coherence overhead,
munication via global virtual address space. Mapped
developer more prefer this method when sequential con-
shared data or memory tuning between the environment
sistency of writing sharing is prevalent.5,19,24
There is basically four algorithms client server; of remotely located shared data and process’s own local
migration read replication, full replication based on memory. Some programs or as divide file into variable
migration and replication.2 Each having a different size chunks and copy them into virtual address space so
methodology to manage distributed shared data. Here that can be available in a process’s address space. After
distributed shared memory algorithm will manage that process can read/write remote contents with ordinary
all memory management activity through memory memory access. This permits fast access of shared content.
controller using some protocol. If requested data not Figure 2 shows the memory mapping mechanism done by
available in local memory, then request go to memory different process.
controller that will check look up to serve requests for Here, memory controller will plays major role insight
remotely located data. Multiple processes can read and distributed system. It will dynamically manages all the
write simultaneously to share data. If it is required to activities of distributed memory system like managing
update the system to improve system then it will indexing/mapping memory segment or shared data items,
be easily manageable. handle requests from processors, converting instruc-

4 Vol 10 (28) | July 2017 | www.indjst.org Indian Journal of Science and Technology
 Hemant D. Vasava and Jagdish M. Rathod

tion format sets, locking mechanism, retrieving data and idle waiting time. This program instruction execution
location from remote memories, scheduling and creates order often categories in to different types. In this kind
virtual global address space. of shared memory system the execution is in the correct
order if the result same as one in which the instruction
were executed in program order. Such kind of system
design known as sequential.
Here, distributed shared memory model utilizing
sequential consistency order which ensures no memory
operation execution begins until all the previous opera-
tion have been completed. Figure 3(a) and (b) shows
sequential consistency implementation among differ-
ent process. In a distributed system an efficient, ordered
effective broadcast mechanism, shared variables could be
grouped together on one or more variable, and operations
of shared variable require to be broadcasted and all nodes
Figure 2. Illustration Distributed Shared Memory Mapping.
have to require seeing the similar order of memory point-
ers.
5.4 Naming Scheme Specification
In a distributed shared memory systems if the pro-
cess wishing to access remotely located data then it
required knowing on which system data resides and fetch
it from remote node. System will take the help of memory
controller to manage the shared memory system activity.
So, all shared data is visible to the memory controller, it
is also managing lookup or indexing. The memory con-
troller has required using unique naming mechanism to
avoid any conflicts while shared data items are distributed
over remote location. And possible solution is to assign
a logical global naming. The memory manager at each (a)
machine performs the conversion of the logical address
to provide the location of the data segment on a remote
node. But this approach is not being useful if the gran-
ularity units of shared data items are less than a page.2
Memory controller plays an important role between local
and remote node to access requires shared data. So, first (b)
process will use data items from its own local memory. Figure 3. (a) and (b) Sequential Operation Exexution.
If required data items are not in local memory then
requested transfer to the memory controller. Memory 5.6 Protocol Mechanism
controller satisfies the request by mapping mechanism A Distributed system needs communication between
and to respond to requests made. Shared data items hav- multiple nodes insight system. In this proposed
ing unique global identification that is registered with architecture internet domain sockets implemented to
memory controller. All process wishing to access shared make connections between processes on different nodes
data items use the same identifier. which are connected by a high speed network. Machines
are connected via local area network. In this write invalidate
5.5 Consistency Model protocol write access to the shared data item will invalidation
Today’s modern high speed processor executes of all replication excluding one before the write begins.
operations in a different order than specified in past Invalidated replicated data units are not accessible through
literature to increase execution speed and decrease any node of system. So, the major drawback of writing the

Vol 10 (28) | July 2017 | www.indjst.org Indian Journal of Science and Technology 5
 Improving Performance of Distributed Shared Memory (DSM) on Multiprocessor Framework with Software Approach

invalidate protocol implementation is that invalidation that make distributed shared memory possible. Latency
require to send each node whether they have used shared plays an important role while shopping the protocol. This
data or not. It is better suitable for programs where several implementation uses sequential consistency which is the
updates held during multiple reads, as well as when an strongest consistency model that is practicable when there
application exhibits in which high degree per node locality is delay in communication over network. The motivation
of reference. However, sometimes this write invalidate for this design was to implement using shared variable
protocol is inefficient when no of nodes frequently access and data structure, its DSM algorithm, memory controller
data items, because an updated data will require to be concept and specific issues, kernel programming, inter-
copied back to many processes on nodes immediately process communication and protocol involved. These
after every invalidation occurs. No of distributed memory goals have been largely met to get improved performance
systems have used to write invalidate protocol like IVY and reliability.
which support strict consistency model, Clouds that have Future work can be done to improve algorithm and
used release consistency model and many more in the past. memory controller activities, respectively to design issues
that will add more benefits to distribute shared memory
5.7 Advantages and Disadvantages management. It can manage more no of processors simul-
taneously, increased execution speed and also provide
This software based distributed shared memory system
security. More effort can be added to increase the con-
having various advantages as below:
sistency mechanism of system to make the design more
• Provides large virtual memory space reliable against the failure.
• Well expansion with increase in process nodes
• Underlying message communication between 7. References
nodes will remain invisible outside.
• Complex structure and large database support 1. Sinha KP. Distributed Operating Systems Concepts and
• Generally cheaper than using a multiprocessor Design; 2009. p. 1–90.
2. Vasava HD, Rathod JM. A Survey of Software based
system
Distributed Shared Memory implementation methodologies
• More portable because of common interfaces
for Multiprocessor Environments. International Journal
• Protect its developer from sending and receiving of Innovative Research in Science Engineering and
primitives Technology. 2013 Jul; 2(7):3055–60.
3. Tanenbaum SA, Steen MV. Distributed Systems Principles
It also gives few disadvantages like becomes slower and Paradigms. 2nd Edition. PHI Learning; 2011. p. 1–705.
to access in comparison with non distributed shared 4. Naeem A, Jantsch A, Lu Z. Scalability analysis of memory
memory and it is requiring additional concurrency consistency models in NoC based distributed shared
control mechanism against concurrent accesses towards memory SoCs. IEEE transaction on computer aided design
shared data. of integrated circuits and systems. 2013 May; 32(5):760–73.
5. Jin X, Guan N, Lv M, Deng Q. Improving the Performance
of Shared Memory Communication in Impulse C. IEEE
6. Conclusion and Future Work Embedded Systems LETTERS. 2010 Sep; 2(3):49–52.
Crossref
A distributed system can be implemented using vari-
6. Choi I, Zhao M, Yang X, Yeung D. Experience with
ous distributed model where shared memory plays
Improving Distributed Shared Cache Performance on
major role in architecture. In this research paper we have Tileras Tile Processor IEEE Computer Architecture. 2011
described a new way of architecture using software Dec; 10(2):45–8.
parameters of shared memory system design. So, shared 7. Ibrahim KZ, Hofmeyr S, Lancu C. The Case for Partitioning
memory implementing requires the careful evolution of Virtual Machines on Multicore Architectures. IEEE
several design issues such as interaction with shared Transactions on Parallel and Distributed Systems. 2014
data by node, data organization, consistency model and Oct; 25(10):2683–96. Crossref
algorithm, protocols, scheduling etc. It requires com- 8. Yu Z, Xiao R, You K. A 16-Core Processor with Shared
plex aspects in core architecture and close association Memory and Message-Passing Communications. 2014 Apr;
between virtual address management and mechanism 61(4):1081–94.

6 Vol 10 (28) | July 2017 | www.indjst.org Indian Journal of Science and Technology
 Hemant D. Vasava and Jagdish M. Rathod

9. Vasava HD, Rathod JM. Software based Distributed Shared 17. Marongiu A, Benini L. An Open MP Compiler for Efficient
Memory (DSM) model using shared variables between Use of Distributed Scratchpad Memory in MPSoCs. IEEE
Multiprocessors. In Proceeding of IEEE International Transactions on Computers. 2012 Feb; 61(2):222–36.
Conference on Communication and Signal Processing; Crossref
2015. p. 1431–5. Crossref 18. Lee M, Ahn M, Kim EJ. Fast Secure Communications
10. Gopal B, Beg R, Kumar P. Memory Management Techniques in Shared Memory Multiprocessor Systems. IEEE
for Paging on Distributed Shared Memory Framework. Transactions on Parallel and Distributed Systems. 2011
International Journal of Computer Science and Information Oct; 22(10):1714–21. Crossref
Technology. 2010 Apr; 2(2):141–53. Crossref 19. Krzyzanowski P. Distributed Shared Memory and
11. Dhruv S, Mehta, Iye VS. Study and Implementation of Consistency Model. Rutgear University; 2009. p. 1–10.
Distributed Shared Memory. NIT: Nagpur; 2004-2005. 20. Dash A, Demsky B. Integrating Caching and Prefetching
Mechanisms in a Distributed Transactional Memory. IEEE
12. Seidmann T. Distributed Shared Memory in Modern
Transactions on Parallel and Distributed Systems. 2011
Operating Systems. University of Technology in Bratislava;
Aug; 22(8):1284–98. Crossref
2004.
21. Shi Q, Khan O. Toward Holistic Soft-Error-Resilient
13. Kshemkalyani A, Singhal M. Distributed Computing: Shared-Memory Multicores. IEEE Computer Society Press.
Principles Algorithms and Systems. Cambridge University Los Alamitos, USA. 2013 Oct; 46(10):56–64. Crossref
Press; 2011 Mar. p. 1–756. 22. Tumeo A, Villa O, Chavarria-miranda DG. Aho-corasick
14. Penurkar MK, Sugandhi R. OPDSM: A Combinational Page stirng matching on shared and distributed memory parallel
based and Object based DSM Model. International Journal architecture. IEEE Transaction on Parallel and Distributed
of Computer Applications. 2010 Feb; 1(6):1–5. Crossref Systems. 2012 Mar; 23(3):436–43. Crossref
15. Castanie L, Mion C, Bruno L. Distributed Shared 23. Apostolakis A, Gizopoulos D, Psarakis M, Pachalis A.
Memory for Roaming Large Volumes. IEEE Transactions Software based Self Testing of Symetric Shared Memory
on Visualization and Computer Graphics. 2006 Oct; Multiprocessors. IEEE Transactions on Computers. 2009
12(5):1299–306. Crossref Dec; 58(12):1682–94. Crossref
16. Navratil PA, Childs H, Fussell SD. Exploring the Spectrum of 24. Diaz J, Caro CM, Nino A. A Survey of Parallel Programming
Dynamic Scheduling Algorithms for Scalable Distributed- Models and Tools in the Multi and Many-Core Era. IEEE
Memory Ray Tracing. Transactions on Visualization and Transactions on Parallel and Dist Systems. 2012 Aug;
Computer Graphics; 2013. p. 1–14. PMid:23831285 23(8):1369–86. Crossref

Vol 10 (28) | July 2017 | www.indjst.org Indian Journal of Science and Technology 7