CS0051 - Module 01 - Subtopic 1

Module 1
Introduction to Parallel and

Distributed Computing
Module 1
Subtopic 1
Introduction to Parallel and
Distributed Computing
• To discuss the evolution of software crisis
• To recognize different hardware architecture based on
Flynn’s taxonomy
• To determine the theoretical speedup when using
multiple processors using Amhdal’s Law
“To put it quite bluntly: as long as there were no
machines, programming was no problem at all; when we
had a few weak computers, programming became a mild
problem, and now we have gigantic computers,
programming has become an equally gigantic problem”
-- E. Dijkstra, 1972 Turing Award Lecture

• Time frame: 60’s and 70s
• Problem: Assembly Language Programming
• Computers could handle larger more complex programs
• Needed to get Abstraction and Portability without

losing performance
• High Level Languages for von-Neumann Machines
• Fortran and C
• Provide “common machine language” for
uniprocessors
• Time Frame: ’80s and ’90s
• Problem: Inability to build and maintain complex and

robust applications requiring multi-million lines of
code developed by hundreds of programmers
• Computers could handle larger more complex programs
• Needed to get Composability, Malleability and

Maintainability
• High-performance was not an issue →left for Moore’s Law
• Object Oriented Programming
• C++, C# and Java
• Better tools
• Component Libraries
• Better software engineering methodology

• Design patterns, specification, testing, code reviews
• Solid boundary between Hardware and Software
• Programmers don’t have to know anything about the processor „
• High level languages abstract away the processors
• Java bytecode is machine independent „
• Moore’s law does not require the programmers to know anything about the
processors to get good speedups
• Programs are oblivious of the processor → work on all
processors „
• A program written in ’70 using C still works and is much faster today
• This abstraction provides a lot of freedom for the
programmers
• Time Frame: 2005 to 20??
• Problem: Sequential performance is left behind by Moore’s law
• Needed continuous and reasonable performance improvements „

• to support new features „
• to support larger datasets
• While sustaining portability, malleability and maintainability without
unduly increasing complexity faced by the programmer
• →critical to keep-up with the current rate of evolution in software

https://upload.wikimedia.org/wikipedia/commons/thumb/0/00/Moore%27s_Law_Transistor_Count_1970-2020.png/1280px-Moore%27s_Law_Transistor_Count_1970-2020.png
12th Gen Intel® Core™ Processors Product Brief
AMD Ryzen™ 5000 Series Processors | Fastest in the Game | AMD
• Flynn's taxonomy is a classification of computer architectures, proposed by Michael J.
Flynn in 1966 and extended in 1972.
• The classification system has stuck, and it has been used as a tool in design of modern
processors and their functionalities.
• An SISD computing system is a uniprocessor machine which is capable
of executing a single instruction, operating on a single data stream.
• In SISD, machine instructions are processed in a sequential manner and computers adopting this
model are popularly called sequential computers. Most conventional computers have SISD
architecture. All the instructions and data to be processed have to be stored in primary memory.
• An SIMD system is a multiprocessor machine capable of
executing the same instruction on all the CPUs but operating
on different data streams.
• Machines based on an SIMD model are well suited to scientific computing since they
involve lots of vector and matrix operations.
• An MISD computing system is a multiprocessor machine
capable of executing different instructions on different PEs but
all of them operating on the same dataset .
• An MIMD system is a multiprocessor machine which is
capable of executing multiple instructions on multiple data
sets.
• Each PE in the MIMD model has separate instruction and data streams; therefore machines
built using this model are capable to any kind of application.
• PEs in MIMD machines work asynchronously.
• MIMD machines are broadly categorized into shared-
memory MIMD and distributed-memory MIMD based on
the way PEs are coupled to the main memory.
• In the shared memory MIMD model (tightly coupled multiprocessor

systems), all the PEs are connected to a single global memory and they all
have access to it.
• In Distributed memory MIMD machines (loosely coupled multiprocessor
systems) all PEs have a local memory.
The serial execution of tasks is a sort of chain, where the first task is
followed by the second one, the second is followed by the third, and so on.
The important point here is that tasks are physically executed without
overlapping time periods.
• Simplicity
• It is a straightforward approach, with a clear set of step-by-step instructions about what to do and
when to do it.
• Scalability
• A system is considered scalable if its performance improves after adding more processing
resources. In the case of sequential computing, the only way to scale the system is to increase the
performance of system resources used – CPU, memory, etc.
• Overhead
• In the sequential computing, no communication or synchronization is required between different
steps of the program execution. But there is an indirect overhead of the underutilization of
available processing resources
• Concurrency is when multiple tasks can run in overlapping
periods. It's an illusion of multiple tasks running in parallel
because of a very fast switching by the CPU. Two tasks
can't run at the same time in a single-core CPU
Image: https://golangbot.com/concurrency/
• Parallelism is when tasks actually run in parallel in multiple
CPUs
Image: https://golangbot.com/concurrency/
Image: https://devopedia.org/concurrency-vs-parallelism
• Complexity of designing parallel algorithms
• Removing task dependencies
• Can add large overheads
• Limited by memory access speeds
• Execution speed is sensitive to data
• Real world problems are most naturally described with

mathematical recurrences
• Saman Amarasinghe and Rodric Rabbah, 6.189 Multicore Programming Primer, January (IAP) 2007.
(Massachusetts Institute of Technology: MIT OpenCourseWare). http://ocw.mit.edu (accessed 07 27, 2022).
License: Creative Commons Attribution-Noncommercial-Share Alike.
• https://en.wikipedia.org/wiki/Moore%27s_law
• https://www.intel.com/content/www/us/en/products/docs/processors/core/12th-gen-core-mobile-
processors-brief.html
• https://en.wikipedia.org/wiki/Flynn%27s_taxonomy
• https://www.geeksforgeeks.org/computer-architecture-flynns-taxonomy/
• https://livebook.manning.com/book/grokking-concurrency/chapter-2/v-4/108
• https://stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism
• https://www.geeksforgeeks.org/memory-layout-of-c-program/

CS0051 - Module 01 - Subtopic 1

Uploaded by

Copyright:

Available Formats

CS0051 - Module 01 - Subtopic 1

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CS0051 - Module 01 - Subtopic 1

Uploaded by

Copyright:

Available Formats

Module 1

Introduction to Parallel and

-- E. Dijkstra, 1972 Turing Award Lecture

• Needed to get Abstraction and Portability without

• Problem: Inability to build and maintain complex and

• Needed to get Composability, Malleability and

• Better software engineering methodology

• Needed continuous and reasonable performance improvements „

• →critical to keep-up with the current rate of evolution in software

• In the shared memory MIMD model (tightly coupled multiprocessor

• Limited by memory access speeds

• Execution speed is sensitive to data

• Real world problems are most naturally described with

You might also like