Wollo University, Kombolcha Institute of

Technology
College of Informatics, Software Engineering
Department
A lecture note by Tilahun A. On courses of
“Recent topics In Software Engineering”
Chapter 3: Introduction to High Performance Computing

Kombolch, Amhara, Ethiopia

12th May 2022
By Tilahun A. 1
 Introduction to High performance Computing
List of Acronyms
SIMD: Single Instructions multiple data
GPU: Graphics Processing Unit
SISD: Single Instruction single Data
MISD: Multiple Instruction single Data
MIMD: Multiple Instruction multiple data
VPU: Vector Processing Unit
FPU: Floating Point Unit

By Tilahun A. 2
 Introduction to High performance Computing
Introduction
✓High Performance computer systems can be regarded as the most powerful
and flexible research instruments today.
✓They are employed to model phenomena in fields so various as climatology,
quantum chemistry, computational medicine, High-Energy Physics and
many, many other areas.
✓The hardware structure or architecture determines to a large extent what
the possibilities and impossibilities are in speeding up a computer system
beyond the performance of a single CPU core.
✓Another important factor that is considered in combination with the
hardware is the capability of compilers to generate efficient code to be
executed on the given hardware platform.

By Tilahun A. 3
 Introduction to High performance Computing
Rationale behind HPC

By Tilahun A. 4
Introduction to High performance Computing

By Tilahun A. 5
Introduction to High performance Computing

By Tilahun A. 6
Introduction to High performance Computing
Introduction

By Tilahun A. 7
Introduction to High performance Computing
Introduction

By Tilahun A. 8
 Introduction to High performance Computing
…Introduction (The main architectural classes)
✓This classification is based on the way of manipulating of instruction
and data streams and comprises four main architectural classes.
SISD machines:
✓These are the conventional systems that contain one
CPU and hence can accommodate one instruction stream that is
executed serially.
✓Nowadays about all large servers have more than one CPU but each of
these execute instruction streams that are unrelated.

By Tilahun A. 9
 Introduction to High performance Computing
…Introduction (The main architectural classes)
SIMD machines:
✓Such systems often have a large number of processing units, that all
may execute the same instruction on different data in lock-step. So, a
single instruction manipulates many data items in parallel.
✓Nevertheless, the concept is still interesting and it is recurring these
days as a co-processor in HPC systems, albeit in a somewhat restricted
form in some computational accelerators like GPUs

By Tilahun A. 10
 Introduction to High performance Computing
…Introduction (The main architectural classes)
MISD machines:
✓Theoretically in these types of machines multiple
instructions should act on a single stream of data. As yet no practical
machine in this class has been constructed nor are such systems easy
to conceive. We will disregard them in the following discussions.
MIMD machines:
✓These machines execute several instruction streams
in parallel on different data. The difference with the multi-processor
SIMD machines mentioned above lies in the fact that the instructions
and data are related because they represent different parts of the same
task to be executed. So, MIMD systems may run many sub-tasks in
parallel in order to shorten the time-to-solution for the main task to
be executed.
By Tilahun A. 11
 Introduction to High performance Computing
…Introduction (The main architectural classes)
Block diagram of a vector processor system is shown as below

By Tilahun A. 12
 Introduction to High performance Computing
…Introduction (Distributed-memory SIMD machines )
✓Machines of the DM-SIMD type are sometimes also known as
processor-array machines
✓Because all processors execute the same instruction at the same time
(but on different data items), no synchronization between processors is
required.
✓A control processor issues the instructions that are to be executed by
the processors in the processor array.
✓Presently, no commercially available machines of the processor-array
type are marketed.

By Tilahun A. 13
 Introduction to High performance Computing
…Introduction (Distributed-memory SIMD machines )
✓DM-SIMD machines use a front-end processor to which they are
connected by a data path to the control processor.
✓Operations that cannot be executed by the processor array or by the
control processor are offloaded to the frontend system.
✓For instance, I/O may be through the front-end system, by the
processor array machine itself, or by both.

By Tilahun A. 14
Introduction to High performance Computing
…Introduction (Distributed-memory SIMD machines )
Some examples of
interconnection structures
used in shared-memory
MIMD systems.

By Tilahun A. 15
 Introduction to High performance Computing
…Introduction (Shared-memory MIMD machines)
✓ More than one FPU and/or VPU may be possible in one system.
✓The main problem one is confronted with in shared-memory systems
is that of the connection of the CPUs to each other and to the memory.
✓More CPUs are added, the collective bandwidth to the memory
ideally should increase linearly with the number of processors, while
each processor should preferably communicate directly with all others
without the much slower alternative of having to use the memory in an
intermediate stage.
✓Unfortunately, full interconnection is quite costly, growing with O(n2)
while increasing the number of processors with O(n).
By Tilahun A. 16
 Introduction to High performance Computing
Recent trends in High performance Computing
✓The introduction of vector computers started the era of ‘Supercomputing’.
✓The initial success of vector computers in the seventies was driven by raw
performance.
✓Massive Parallel Processors (MPPs) became successful in the early nineties
due to their better price/performance ratios, which was enabled by the attack
of the ‘killer-micros’.
✓The success of microprocessor based Symmetric Multiprocessor (SMP)
concepts even for the very high-end systems, was the basis for the emerging
cluster concepts in the early 2000’s.

By Tilahun A. 17
 Introduction to High performance Computing
Recent trends in High performance Computing
✓The first half of this decade clusters of PC’s and
workstations have become the prevalent architecture for many HPC
application areas on all ranges of performance.
✓However, the Japanese Earth Simulator vector system demonstrated
that many scientific applications could benefit greatly from other
computer architectures.
✓At the same time there is renewed broad interest in the scientific HPC
community for new hardware architectures and new programming
paradigms.

By Tilahun A. 18
 Introduction to High performance Computing
Recent trends in High performance Computing
✓High-Performance Computing systems (HPC) always characterized by a rapid
change of vendors, architectures, technologies and the usage of systems.
✓Despite all these changes, the evolution of performance on a large scale seems
to be a very steady and continuous process [Moore’s Law].
✓The Introduction of this type of computer systems started the era of
‘Supercomputing’.
✓the overall HPC market is a recent demand now and in the future.

By Tilahun A. 19
Introduction to High performance Computing
Recent trends in High performance Computing

By Tilahun A. 20
 Introduction to High performance Computing
Introduction to Grid Computing
✓A group of networked computers which work together as a virtual
supercomputer to perform large tasks, such as analyzing huge sets of
data or weather modeling.
✓Through the cloud, you can assemble and use vast computer grids for
specific time periods and purposes, paying, if necessary, only for what
you use to save both the time and expense of purchasing and
deploying the necessary resources yourself.
✓Also by splitting tasks over multiple machines, processing time is
significantly reduced to increase efficiency and minimize wasted
resources.
By Tilahun A. 21
Introduction to High performance Computing
Introduction to Grid Computing
✓ Early Virtualizations primary
focus was on Internal Resources.
✓ However, the intended
requirements of increased
resources and storage
requirements leads to the external
virtualization
✓ Its all about effective computing
with efficient algorithms.

By Tilahun A. 22
 Introduction to High performance Computing
…Introduction to Grid Computing(parallel CPU
computing)
✓The potential for massive parallel CPU capacity is one of the most common
visions and attractive features of a grid.
✓In addition to pure scientific needs, such computing power is driving a new
evolution in industries such as the bio-medical field, financial modeling, oil
exploration, motion picture animation, and many others.
✓The common attribute among such uses is that the applications have been written
to use algorithms that can be partitioned into independently running parts.
✓A CPU-intensive grid application can be thought of as many smaller sub jobs,
each executing on a different machine in the grid. To the extent that these sub jobs
do not need to communicate with each other, the more scalable the application
becomes.
✓A perfectly scalable application will, for example, finish in one tenth of the time if
it uses ten times the number of processors
By Tilahun A. 23
Introduction to High performance Computing
…Introduction to Grid Computing(Virtual resources and
virtual organizations for collaboration )
✓Another capability enabled by grid computing is to provide an
environment for collaboration among a wider audience.
✓In the past, distributed computing promised this collaboration
and achieved it to some extent.
✓Grid computing can take these capabilities to an even wider
audience, while offering important standards that enable very
heterogeneous systems to work together to form the image of a
large virtual computing system offering a variety of resources.

By Tilahun A. 24
Introduction to High performance Computing
✓ A data grid can expand data capabilities in several
ways. First, files or databases can span many systems
and thus have larger capacities than on any single
system.
✓ Such spanning can improve data transfer rates
through the use of striping techniques.
✓ Data can be duplicated throughout the grid to serve
as a backup and can be hosted on or near the
machines most likely to need the data, in conjunction
with advanced scheduling techniques.
✓ Sharing is not limited to files, but also includes other
resources, such as specialized devices, software,
services, licenses, and so on.
✓ These resources are virtualized to give them a more
uniform interoperability among heterogeneous grid
participants.

By Tilahun A. 25
 Introduction to High performance Computing
Access to other resources
✓As already stated, in addition to CPU and storage resources, a grid can provide
access to other resources as well.
✓The additional resources can be provided in additional numbers and/or capacity.
✓For example, if a user needs to increase their total bandwidth to the Internet to
implement a data mining search engine, the work can be split among grid
machines that have independent connections to the Internet.
✓In this way, total searching capability is multiplied, since each machine has a
separate connection to the Internet.
✓If the machines had shared the connection to the Internet, there would not have
been an effective increase in bandwidth.
✓Some machines may have expensive licensed software installed that users
require. Users’ jobs can be sent to such machines, more fully exploiting the
software licenses.
By Tilahun A. 26
 Introduction to High performance Computing
Resources Balancing
✓A grid federates a large number of resources contributed by individual machines
into a large single-system image.
✓For applications that are grid-enabled, the grid can offer a resource balancing effect
by scheduling grid jobs on machines with low utilization.
✓This feature can prove invaluable for handling occasional peak loads of activity in
parts of a larger organization.
✓This can happen in two ways:
An unexpected peak can be routed to relatively idle machines in the grid.
✓If the grid is already fully utilized, the lowest priority work being performed on
the grid can be temporarily suspended or even cancelled and performed
again later to make room for the higher priority work.
✓Without a grid infrastructure, such balancing decisions are difficult to prioritize
and execute.

By Tilahun A. 27
Introduction to High performance Computing
✓ Other more subtle benefits can occur using a
grid for load balancing.
✓ When jobs communicate with each other, the
Internet, or with storage resources, an
advanced scheduler could schedule them to
minimize communications traffic or
minimize the distance of the communications.
✓ This can potentially reduce
communication and other forms of contention
in the grid
✓ a grid provides excellent infrastructure for
brokering resources
✓ Individual resources can be profiled to
determine their availability and their capacity,
and this can be factored into scheduling on the
grid.

By Tilahun A. 28
Introduction to High performance Computing

By Tilahun A. 29
Introduction to High performance Computing
Cloud Computing
✓ The term Cloud refers to a Network or Internet.
✓ In other words, we can say that Cloud is something, which is present at remote
location.
✓ Cloud can provide services over network, i.e., on public networks or on private
networks, i.e., WAN, LAN or VPN.
✓ Applications such as e-mail, web conferencing, customer relationship
management (CRM),all run in cloud.
✓ Refers to manipulating, configuring, and accessing the applications online. It
offers online data storage, infrastructure and application.

By Tilahun A. 30
Introduction to High performance Computing

By Tilahun A. 31
Introduction to High performance Computing
USES

✓ Helps to use applications without installations.

✓ Access the personal files and data from any computer
with internet access.
✓ This technology allows much more efficient computation
by
✓ Centralizing storage,
✓ Shared Memory and processing units .

By Tilahun A. 32
Introduction to High performance Computing
Essential Cloud Characteristics
✓ On-demand self-service
✓ Broad network access
✓ Resource pooling
✓ Location independence
✓ Rapid elasticity
✓ Measured service
Pay as you go.
✓ Elasticity and the ability to scale up and down
Self-service provisioning and automatic deprovisioning
✓ on demand self-service.
.
By Tilahun A. 33
Introduction to High performance Computing
Discuss the pros and cons of cloud computing
Pros
✓ Cost effective, scalable, pay as per use, easily available
Cons
✓ Link based infrastructure, data unsecured, limited features depending on
provider, Backup with provider
Cloud deployment models
✓Define the type of access to the cloud, i.e., how the cloud is
located?
✓Cloud can have any of the four types of access:
✓Public, Private, Hybrid and Community.
By Tilahun A. 34
.
Introduction to High performance Computing

By Tilahun A. 35
Introduction to High performance Computing
✓ The Public Cloud allows systems and services to be easily accessible to the
general public.
✓ Public cloud may be less secure because of its openness, e.g., e-mail.
✓ The Private Cloud allows systems and services to be accessible within an
organization.
✓ It offers increased security because of its private nature.
✓ The Community Cloud allows systems and services to be accessible by
group of organizations.
✓ The Hybrid Cloud is mixture of public and private cloud. However, the
critical activities are performed using private cloud while the non-critical
activities are performed using public cloud.

By Tilahun A. 36
 Introduction to High performance Computing
SERVICE MODELS
✓ Service Models are the reference
models on which the
Cloud Computing is based.
✓ These can be categorized into
three basic service models as listed below:
1. Infrastructure as a Service (IaaS)
2. Platform as a Service (PaaS)
3. Software as a Service (SaaS)

By Tilahun A. 37
 Introduction to High performance Computing
Cloud Infrastructure:

Cloud infrastructure includes: Server, storage, management, network,

management software and deployment software virtualization

By Tilahun A. 38
 Introduction to High performance Computing
Infrastructural constraints:

By Tilahun A. 39
 Introduction to High performance Computing
Challenges of cloud computing (Security)
✓ Trusted computing is a broad term that refers to technologies and proposals for
resolving computer security problems through hardware enhancements and
associated software modifications.
✓ Several major hardware manufacturers and software vendors, collectively known
as the Trusted Computing Group (TCG), are cooperating in this venture and have
come up with specific plans.
✓ The TCG develops and promotes specifications for the protection of computer
resources from threats posed by malicious entities without infringing on the rights
of end users.

By Tilahun A. 40
 Introduction to High performance Computing
Conclusions
✓ SaaS represents an architectural model that is built on the
foundation of:
✓ Massive scalability,
✓ Multi-tenant efficiency,
✓ Metadata-driven configurability to provide great software
inexpensively to both existing and potential clients.
✓ Adopting these principles can help in implementing SaaS maturity model and place
you on the right path to completely transforming the manner in which you depict
the long-tail business

By Tilahun A. 41