Scribd
Upload
26 views16 pages

Lect 1

Uploaded by

Mehwish Majeed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
26 views16 pages

Lect 1

Uploaded by

Mehwish Majeed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 16

Parallel and Distributed Systems

Parallel and distributed systems


• What is a parallel computer?
– A collection of processing elements that
communicate and coorperate to solve large
problems fast.
• What is a distributed system?
– A collection of independent computers that
appear to its users as a single coherent system
What are they?
• More than one processing element
• Elements communicate and cooperate
• Appear as one system

• PDS hardware is everywhere


– Multicore workstations/laptops
– (4-node cluster, 2-CPU SMP, 8 cores per node)
– Hadoop clusters
– IBM Blue Gene/L at DOE/NNSA/LLNL (262992 processors).
– Data centers
Why parallel and distributed
computing?
• Moore’s law: the number of transistors
double every 18 months.
Why parallel and distributed
computing?
• How to make good use of the increasing number of
transistors on a chip?
– Increase the computation width (70’s and 80’s)
• 4bit->8bit->16bit->32bit->64bit->….
– Instruction level parallelism (90’s)
• Pipeline etc
– ILP to the extreme (early 00’s)
• Out of order execution

– Sequential program performance keeps increasing.


• The clock rate keeps increasing, clock cycles get smaller
and smaller.
Why parallel and distributed
computing?
• The fundament limit of the speed of a sequential
processor.
– Power wall (high frequency results in heat)
– Latency wall (speed of light does not change)
Power wall: what will happen if
we keep pushing ILP?
Latency wall
• Speed of light = 300000000m/s
• One cycle at 4Ghz = 0.00000000025s
• The distance that the light can move at one cycle:
– 0.00000000025 * 300000000 = 7.5cm

Intel chip dimension = 1.47 in x 1.47 in


= 3.73cm x 3.73cm
Why parallel and distributed
computing?
• Power wall and latency wall indicate that the
era of single thread performance improvement
through Moore’s law is ending/has ended.
• More transistors on a chip are now applied to
increase system throughput, the total number
of instructions executed, but not latency, the
time for a job to finish.
– Improving ILP improves both.
– We see a multi-core era
• The marching of multi-core (2004-now)
– Mainstream processors:
• INTEL Quad-core Xeon (4 cores),
• AMD Quad-core Opteron (4 cores),
• SUN Niagara (8 cores),
• IBM Power6 (2 cores)
– Others
• Intel Tflops (80 cores)
• CISCO CSR-1 (180 cores) (network processors)

– Increase the throughput without increasing the


clock rate.
• Why parallel and distributed computing?

– Programming multi-core systems is fundamentally


different from programming traditional computers.
• Parallelism needs to be explicitly expressed in the program.
– This is traditionally referred to as parallel computing.
• PDC is not really a choice anymore.
For (I=0; I<500; I++)
a[I] = 0;

GO PARALLEL
Performance

1970 1980 1990 2000 2006


• In the foreseeable future, parallel systems
with multi-core processors are going
mainstream.
– Lots of hardware, software, and human issues
in mainstream parallel/distributed computing
• How to make use a large number of processors
simultaneously
• How to write parallel programs easily and
efficiently?
• What kind of software supports is needed?
• Analyzing and optimizing parallel programs is still a
hard problem.
• How to debug concurrent program?
Parallel computing issues
• Data sharing – single versus multiple
address space.
• Process coordination – synchronization
using locks, messages, and other means.
• Distributed versus centralized memory.
• Connectivity – single shared bus versus
network with many different topologies.
• Fault tolerance/reliability.
Distributed systems issues
• Transparency
– Access (data representation), location,
migration, relocation, replication, concurrency,
failure, persistence
• Scalability
– Size, geographically
• Reliability/fault tolerance
What we will be doing in this
course?
• This is the first class in PDC.
– Systematically introduce most mainstream
parallel computing platforms and programming
paradigms.

• The coverage is not necessary in depth.

You might also like