Distributed systems

and Cloud Computing

Lecture 1
We will very likely cover topics such as

● MPI - Message Passing Interface

● Actor Model for Distributed Programming
● Apache Spark
● Amazon AWS
● Google Cloud
● Microsoft Azure
Introduction to
Distributed
Systems
What is Distributed Systems?

Also known as distributed computing and distributed databases, a

distributed system is a collection of independent components
located on different machines that share messages with each other in
order to achieve common goals.

Distributed computing is the method of making multiple

computers work together to solve a common problem.
Parallelism is the new
order of the day
Advantages of Parallel computing

• Serial implementation

• Parallel implementation
Evolution of
Distributed
Systems
 In the early days, computers
operated in isolation. The birth of
networking marked a transformative
era as machines started connecting
The Birth of for basic communication. This
allowed data exchange and paved
Networking the way for collaborative computing.

From Isolated Machines to

Connected Worlds
 The client-server architecture
emerged, introducing a
Client-Server centralized model where clients
requested services from a central
Architecture server.

This paradigm shift streamlined

Enter the Client-Server Era processes, making information
and services more accessible
across networks.
 With the proliferation of the
internet, distributed systems

Proliferation of became ubiquitous. Mass adoption

occurred, connecting the world in

the Internet
ways previously unimaginable.
This global network laid the
The Web Unleashed foundation for distributed
computing on a larger scale.
 Peer-to-peer networks gained
prominence, promoting

Peer-to-Peer decentralization. Devices began

sharing resources directly,
Networks reducing reliance on a central
authority. This shift brought about
The Rise of Equality increased resilience and
scalability.
 The era of distributed computing

Distributed began, leveraging the collective

power of multiple machines. Tasks

Computing were divided and processed in

parallel, optimizing performance
Harnessing Collective Power and efficiency.
 Cloud computing revolutionized

Cloud Computing the landscape by introducing

virtualization. This allowed remote

Emerges access to computing resources on-

demand, offering scalability,
The Sky's the Limit flexibility, and cost-efficiency.
Elements of a Distributed System

The most important functions of distributed computing are:

● Resource sharing - whether it’s the hardware, software or data
that can be shared
● Openness - how open is the software designed to be developed
and shared with each other
● Concurrency - multiple machines can process the same function at
the same time
● Scalability - how do the computing and processing capabilities
multiply when extended to many machines
● Fault tolerance - how easy and quickly can failures in parts of the
system be detected and recovered
● Transparency - how much access does one node have to locate
and communicate with other nodes in the system.
Different ways to achieve Distributed computing?

Distributed computing involves the use of multiple computers or servers

to work together on solving a problem or executing a task.

There are several ways to achieve distributed computing, each with its
own advantages and use cases.

Here are some common approaches:

1. Client-Server Architecture
Description: In this model, one central server Use Case: Common in applications where
provides services or resources, and multiple centralized control is needed, such as web
client machines request and use these services.
servers or database systems.

2. Peer-to-Peer Networks
Description: In a peer-to-peer network, Use Case: Popular for file sharing (e.g.,
all nodes (computers or devices) are
BitTorrent) and decentralized applications.
considered equal, and they can share
resources directly with one another
without relying on a central server.

3. Distributed Computing Clusters

Description: Multiple computers are Use Case: High-performance computing
interconnected and work together as a tasks like scientific simulations or data
cluster. Tasks are divided among the analysis.
cluster nodes, allowing parallel processing
and improved performance.
4. Grid Computing
Description: Similar to clusters, but often Use Case: Research projects, scientific
involves geographically distributed computing, and projects that require
resources connected over a network. It massive computational power.
aims to solve large-scale problems by
utilizing idle resources from multiple
locations.

5. Distributed Databases
Description: Data is distributed across Use Case: Large-scale applications with
multiple nodes, allowing for improved high read and write demands, such as
scalability and fault tolerance. Different
types include sharded databases and social media platforms.
NoSQL databases.

6. Cloud Computing
Description: Resources are provided as a Use Case: General-purpose computing,
service over the internet. Users can scalable web applications, and data
access computing power, storage, and storage.
other services on a pay-as-you-go basis.
 ●Message
Distributed
Computing Passing
Models Model
●Actor
Model
The Message-Passing Model

● Based on the notion of multiple processes

○ A process is an instance of a running program, together with the program’s data
● Parallelism is achieved by having many processes co-operate on the
same task
● Each process has access only to its own data
○ all variables are private
● Processes communicate with each other by sending and receiving
messages
○ using library calls from a conventional sequential language for synchronization
○ sending data from one process's memory space to another
MPI: Message Passing Interface

The message passing interface (MPI) is a standardized means of

exchanging messages between multiple computers running a
parallel program across distributed memory.

MPI is not…
● a language or
compiler
specification
● a specific
implementation or
product
Reasons for Using MPI

● Standardization: MPI has replaced other message passing libraries,

becoming a generally accepted industry standard.
● Portability: MPI has been implemented for many distributed memory
architectures, meaning users don't need to modify source code when
porting applications over to different platforms that are supported by the
MPI standard.
● Speed: Implementation is typically optimized for the hardware the MPI
runs on. Vendor implementations may also be optimized for native
hardware features.
● Functionality: MPI is designed for high performance on massively parallel
machines and clusters. The basic MPI-1 implementation has more than
100 defined routines.
● Availability: A variety of implementations are available, both commercial
and public domain
○ OpenMPI and MPICH are popular open-source and free implementations of MPI
○ Microsoft has an open source implementation for Windows called MSMPI
SIMD / SPMD

● Most message passing programming systems use the Single-

Instruction-Multiple-Data (SIMD)
● Sometimes is this also called SPMD: Single Program Multiple
Data
● All processes run their own identical copy of the same program
● Each process work on different parts of same data or on
different data
● Each process has a unique identifier
● Processes can follow different control paths through the
program, depending on their process ID
● Usually run one process per processor / core / machine
Communication Messages

● A message usually transfers a number of data items of a certain type

from the memory of one process to the memory of another process
● A message typically contains
○ the ID of the sending process
○ the ID of the receiving process
○ the type of the data items
○ the number of data items
○ the data itself
○ a message type identifier
● Sending a message can either be synchronous or asynchronous
● A synchronous send is not completed until the message has started
to be received
● An asynchronous send completes as soon as the message has been
accepted into the system
● Receives are usually synchronous - the receiving process must wait
until the message arrives
Process Identification

● MPI processes can belong to one or more groups called

"communicators" (communication channels)
● Processes within a communicator can only communicate with other
processes in that communicator
● When an MPI application starts, the group of all processes is initially
given a predefined name called MPI_COMM_WORLD
● A process is identified by a unique number within each communicator,
called rank
● The process of rank zero (0), is the starting process or the first
process. In older documentation, it is refer to as the "master" node.
Other documentation may refer to it as "controller" node or
"supervisor" node.
● For two different communicators, the same process can have two
different ranks: so the meaning of a “rank” is only defined when you
specify the communicator
Data Communication

● Data communication in MPI is like email exchange

● One process sends a copy of the data to another process, and the
other process receives it

Communication requires the following information:

● Sender has to know:
○ Who to send the data to (receiver’s process rank)
○ What kind of data to send (100 integers or 200 characters, etc)
○ A user-defined “tag” for the message (like an email subject which allows
the receiver to understand what type of data is being received)
● Receiver might have to know:
○ Who is sending the data. It's OK if the receiver does not know; in this
case sender rank will be MPI_ANY_SOURCE, meaning anyone can send
○ What kind of data is being received
○ What the user-defined “tag” of the message is. It's OK if the receiver
does not know; in this case tag will be MPI_ANY_TAG
Using Ranks for Communication

● When sending data, the sender has to specify the destination

process’ rank (process ID)
○ specifies where the message should go
● The receiver has to specify the source process’ rank
○ indicates where the message will come from
● The ID of MPI_ANY_SOURCE is a special “wild-card” source ID
that can be used by the receiver to match any source
Point to Point Messaging

● Sender calls a SEND routine

○ specifying the data that is to be sent
○ this is called the send buffer
● Receiver calls a RECEIVE routine
○ specifying where the incoming data should be stored
○ this is called the receive buffer
● Data goes into the receive buffer
● Metadata describing message also transferred
○ this is received into separate storage
○ this is called the status
MPI Basic Send Message

MPI_SEND(buf,count,datatype,dest,tag,comm)
● The message buffer is described by buf, count, datatype.
● The target process is specified by dest and comm.
○ dest is the rank of the target process in the communicator specified by
comm.
● tag is a user-defined “type” for the message
● When this function returns, the data has been delivered to the
system and the buffer can be reused.
○ Thus this function is "blocking"
○ However, the message might not have been received by the target
process, yet.
MPI Basic Receive Message

MPI_RECV(buf,count,datatype,source,tag,comm,status)
● Waits until a matching on source, tag, comm message is received
from the system, and the buf buffer can be read.
● source is rank in communicator comm, or MPI_ANY_SOURCE.
● Receiving fewer than count occurrences of datatype is OK, but
receiving more is an error.
● status is a structure containing further information:
○ Who sent the message, which is useful if you used MPI_ANY_SOURCE
○ How much data was actually received
○ What tag was used with the message, which is useful if you used
MPI_ANY_TAG
○ MPI_STATUS_IGNORE can be used if we don’t need any additional
information
Running MPI Programs

● MPI programs can either run on the same computer or they can be
distributed to other computers(nodes) to share the
workload.
● In order to run MPI programs on other nodes, they have to be
copied to all the node.
Assessment

This assessment is a Portfolio for 5CS022 Distributed and Cloud Systems Programming,
which accounts for 100% of the module marks.

There are several components to the Portfolio:

Part 1 – Workshop tasks

The workshop tasks will contribute 20% of the marks to the Portfolio. These

will be clearly identified within the workshop instructions.

Part 2 – Quizzes

Quiz will contribute a total of 30% of the marks to the Portfolio.

Part 3 – Coursework

The coursework will consist of a number of questions that you will have to answer by
writing a short

research-based report and a number of tasks which you will have to carry out by
creating a number

of specified programs. The coursework will contribute 50% of the marks to the Portfolio.