Scribd
Upload
10 views

lecture 3

The document discusses parallel computer memory architectures, focusing on shared memory and distributed memory systems. Shared memory systems allow multiple processors to access a global address space, while distributed memory systems operate independently with local memory for each processor. A hybrid model combining both architectures is also mentioned, highlighting the advantages of scalability and the challenges of increased programmer complexity.

Uploaded by

Priah Rajput
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
10 views

lecture 3

The document discusses parallel computer memory architectures, focusing on shared memory and distributed memory systems. Shared memory systems allow multiple processors to access a global address space, while distributed memory systems operate independently with local memory for each processor. A hybrid model combining both architectures is also mentioned, highlighting the advantages of scalability and the challenges of increased programmer complexity.

Uploaded by

Priah Rajput
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Parallel and Distributed Computing

Lecture 3

Muhammad Shahid Khan


Department of Computer Science
COMSATS University Islamabad, Wah Campus
Parallel Computer Memory Architectures
Shared Memory
General Characteristics
✓ Shared memory parallel computers vary widely but generally
have in common the ability for all processors to access all
memory as global address space.
✓ Multiple processors can operate independently but share the
same memory resources.
✓ Changes in a memory location effected by one processor are
visible to all other processors.
✓ Historically, shared memory machines have been classified
as UMA and NUMA, based upon memory access times.

2
Uniform Memory Access (UMA)
✓ Most commonly represented today by Symmetric
Multiprocessor (SMP) machines
✓ Symmetry – Each processor has equal access to
system resources (CPU, memory, and I/O).
✓ Identical processors
✓ Equal access and access times to memory
✓ Sometimes called CC-UMA - Cache Coherent UMA.
Cache coherent means if one processor updates a
location in shared memory, all the other processors
know about the update. Cache coherency is
accomplished at the hardware level.
3
Uniform Memory Access (UMA)

4
Non-Uniform Memory Access (NUMA)
✓ Often made by physically linking two or more SMPs
✓ One SMP can directly access memory of another SMP
✓ Not all processors have equal access time to all
memories
✓ Memory access across link is slower
✓ If cache coherency is maintained, then may also be
called CC-NUMA - Cache Coherent NUMA

5
Non-Uniform Memory Access (NUMA)

6
Advantages
✓ Global address space provides a user-friendly
programming perspective to memory
✓ Data sharing between tasks is both fast and uniform
due to the proximity of memory to CPUs

7
Disadvantages
✓ Primary disadvantage is the lack of scalability
between memory and CPUs. Adding more CPUs can
geometrically increase traffic on the shared memory-
CPU path, and for cache coherent systems,
geometrically increase traffic associated with
cache/memory management.
✓ Programmer responsibility for synchronization
constructs that ensure "correct" access of global
memory.

8
Distributed Memory
General Characteristics

✓ Like shared memory systems, distributed memory


systems vary widely but share a common
characteristic. Distributed memory systems require a
communication network to connect inter-processor
memory.
✓ Processors have their own local memory. Memory
addresses in one processor do not map to another
processor, so there is no concept of global address
space across all processors.
9
Distributed Memory
General Characteristics

✓ Because each processor has its own local memory, it operates


independently. Changes it makes to its local memory have no
effect on the memory of other processors. Hence, the concept
of cache coherency does not apply.
✓ When a processor needs access to data in another processor,
it is usually the task of the programmer to explicitly define how
and when data is communicated. Synchronization between
tasks is likewise the programmer's responsibility.
✓ The network "fabric" used for data transfer varies widely,
though it can be as simple as Ethernet.

10
Distributed Memory

11
Advantages
✓ Memory is scalable with the number of processors.
Increase the number of processors and the size of
memory increases proportionately.
✓ Each processor can rapidly access its own memory
without interference and without the overhead
incurred with trying to maintain global cache
coherency.
✓ Cost effectiveness: can use commodity, off-the-shelf
processors and networking.

12
Disadvantages
✓ The programmer is responsible for many of the
details associated with data communication between
processors.
✓ It may be difficult to map existing data structures,
based on global memory, to this memory
organization.
✓ Non-uniform memory access times - data residing on
a remote node takes longer to access than node local
data.

13
Hybrid Distributed-Shared Memory
General Characteristics
✓ The largest and fastest computers in the world today
employ both shared and distributed memory
architectures.

14
Hybrid Distributed-Shared Memory
✓ The shared memory component can be a shared memory
machine and/or graphics processing units (GPU).
✓ The distributed memory component is the networking of
multiple shared memory/GPU machines, which know only
about their own memory - not the memory on another
machine. Therefore, network communications are required to
move data from one machine to another.
✓ Current trends seem to indicate that this type of memory
architecture will continue to prevail and increase at the high
end of computing for the foreseeable future.

15
Advantages and Disadvantages
✓ Whatever is common to both shared and
distributed memory architectures.
✓ Increased scalability is an important advantage
✓ Increased programmer complexity is an important
disadvantage

16

You might also like