Version 15/09/23

Distributed Systems

Introduction and
basic concepts

[email protected] / [email protected] / [email protected]

 Contents
● Distributed System concepts and examples
● Features, pros and cons.
● Distributed computing vs. Parallel computing
● Nomenclature: protocols, clients, services and servers
● Middleware
● Distributed computing paradigms

Distributed Systems :: Introduction 2

 Definitions

A distributed system is one in which components

located at networked computers communicate and
coordinate their actions only by passing messages.
G. Coulouris, J. Dollimore & T. Kindberg (2001)

The main characteristics of distributed systems:

● Concurrent execution of processes
● Lack of a global clock
● Independent failures
Most of today's applications: social networking, cloud applications, grid
computing, mobile computing, IoT, etc. are distributed systems.

Distributed Systems :: Introduction 3

 Independent failures?
● Client and server are independent processes potentially
running on different nodes. They may have particular
circumstances and restrictions.
● Example:

RPC termination
In gRPC, both the client and server make independent and local
determinations of the success of the call, and their conclusions may
not match. This means that, for example, you could have an RPC
that finishes successfully on the server side (“I have sent all my
responses!”) but fails on the client side (“The responses arrived after
my deadline!”). It’s also possible for a server to decide to complete
before a client has sent all its requests.
https://grpc.io/docs/what-is-grpc/core-concepts/

Distributed Systems :: Introduction 4

 Distributed System
System consists of:
● computer resources (hardware and software)
● physically distributed
● connected through a computer network
● that communicate and coordinate their actions by passing
messages and
● cooperate to perform a task, providing an integrated service.

Distributed Systems :: Introduction 5

 The 8 Fallacies of DS
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn’t change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous
— Peter Deutsch

Distributed Systems :: Introduction 6

 Origin: Resource Sharing

The procedure call model would elevate the task of creating application protocols to that
of defining procedures and their calling sequences. It would also provide the foundation for a
true DPS that encourages and facilitates the work of the applications programmer by
gracefully extending the local programming environment, via the RTE, to embrace modules
on other machines.

This integration of local and network programming environments can even be carried as far
as modifying compilers to provide minor variants of their normal procedure-calling
constructs for addressing remote procedures.

James E. White
A High-Level Framework for
Network-Based Resource Sharing
RFC 707, 1976

DPS: Distributed Programming System

RTE: Run-Time Envinroment

Distributed Systems :: Introduction 7

Example
Internet and web search

●
63 · 109 web pages
●
1 · 1012 URLs

http://www.internetlivestats.com/google-search-statistics/
Distributed Systems :: Introduction 8
 Example
Cloud Computing
Cloud computing provides arbitrary software and
hardware IT resources over the Internet:
● On-demand.

● That scale with the workload

(they are "elastic" resources).

● The customer pays only for what they use.

Distributed Systems :: Introduction 9

 Example
Cloud Computing types
Infrastructure as a Service (IaaS): It provides IT
resources (nodes, storage, networks), but the customer
manages them.
● Examples: AWS, Azure, Google Cloud
Platform as a Service (PaaS): The customer is only
concerned with the application build. The provider manages
the IT resources.
● Examples: Heroku, Google App Engine
Software as a Service (SaaS): Both the infrastructure and
the application itself are managed by the provider.
● Examples: Microsoft 365, Slack, Dropbox

Distributed Systems :: Introduction 10

 Pros and cons
Pros
● Sharing resources (hardware, software, data)
● Good cost/performance ratio
● Growth capacity (scalability)
● Higher availability (fault tolerance)
● Concurrency: simultaneous service to multiple users
● Speed: increased processing capacity
Cons
● Interconnection of resources (cost, reliability, saturation, etc.)
● Communications security
● More complex software

Distributed Systems :: Introduction 14

 DS performance
Network/infrastructure metrics
● Delay and latency
● Bandwidth
● Performance
● Response time
Application Specific
● Transactions per second in the database
● Frames Per Second (FPS) in a game
● Response time in a web application
How are they measured?
● Tools (ping, iperf3, wireshark, etc.)
● Test/benchmarks (standard)
Source: Distributed Systems : Performance. P.O. Östberg, Umea Universiy
Distributed Systems :: Introduction 15
 Distributed vs. Parallel
Distributed Systems:
● Goal: resource sharing
● Heterogeneous computer networks
● Set of computers connected by an heterogeneous inter-network
Parallel Systems:
● Goal: high performance (speedup) and high productivity
● Dedicated architectures (parallel machines):
● Multiprocessors
● Multicomputers

Distributed Systems :: Introduction 16

 Nomenclature
● SERVICE – a component managing a collection of related
resources and provides functionality to users and applications
● The only access to the service is via a well-defined set of
operations that it exports.
– Example: A file server just provides read, write and delete
operations.
● For efective sharing, each resource must be managed by a
program that offers a communication interface. This
communication interface composed by a set of operations is
often called Application Programming Interface (API)
● The APIs are a key concept in new data-based companies
● Example: Instagram
https://developers.facebook.com/docs/instagram-basic-display-api

Distributed Systems :: Introduction 17

 Nomenclature
● SERVER – A running program (a process) on a networked
computer that accepts requests from another program running
on other computer to perform a service and responds
appropriately.
● The requesting processes are referred as clients, and the overall
approach is known as client-server computing.
● When a client sends a request for an operation, we say that the
client invokes an operation upon the server.
● A server can implement/provide one or several services using,
usually, an API for each service.
● ¿How many servers are running in your laptop now?

Distributed Systems :: Introduction 18

 Nomenclature
● A protocol is the set of rules that enable two entities to connect
and transmit data to one another (message exchange).
● The set of messages that a server can accept refers to a
concrete protocol.
● A protocol specification requires:
●Syntax
● Semantics significado del campo
● Synchronization / Timing Se manda el mensaje adecuado cuando toca
● Many RFCs contains protocol specifications

Distributed Systems :: Introduction 19

 Client and server is a ...
program | role
DIY
● Rol: En determinado momento actúa como cliente/servidor

● process | device proceso: Código en ejecución que controla la lógica

● always running | just a while El servidor esta en marcha siempre

● passive | active
● invokes | replies invoca: llamada de un método de un objeto

● heavy | light
● simple | complex deben poder gestionar la concurrencia (múltiples clientes simultáneamente)

Distributed Systems :: Introduction 20

 Client or Server?
ping
DIY
●

● Chrome (web browser)

● Squid (web proxy)
● IoT device
● Dropbox (desktop)
● Other?

Distributed Systems :: Introduction 21

 Challenges
● Heterogeneity of hardware, software, networks and uses.
● Openness: ready to extend, provides open interfaces.
● Security (CIA): Confidentiality, Integrity, Availability
● Scalability: Remain effective when system grows (more
resources and users)
● Failure handling: Ready to manage failures in any
process, computer or network.
● Concurrency: Resources may be requests concurrently.
● Transparency: Show the system as a whole rather than a
set of components. Hiding details to users/programmers.
● Quality of service: how the service should be provided:
● Reliability, security and performance.
Distributed Systems :: Introduction 22
 Heterogeneity

Variety and difference in its components:

● Networks: topology, technology, configuration, etc.
● Hardware: servers, mobiles, laptops, printers, robots, motors,
sensors, etc.
● Operating Systems: Windows, Linux, Android, FreeRTOS,
VxWorks, etc.
● Programming languages: Java, C++, Python, C, Lisp, etc.
● Programmers, experience, know-how...
● E.g. KPI: Time of integration of a new technology.

How do we solve heterogeneity?

● Using open systems

Distributed Systems :: Introduction 23

 Heterogeneity
Open Systems
Protocolos documentados y estandarizados

Property that determines if the system can be

extended and reimplemented
● Specifications, APIs, public, accesible, well documented
protocol stacks.
● Public and open standards
– E.g. RFCs
● Object Management Group (OMG)

http://www.omg.org/spec/index.htm
Distributed Systems :: Introduction 24
 Scalability
capacidad para adaptarse a la demanda

A system has the capacity to grow (scalability) if it maintains its

effectiveness when the number of resources or users increases significantly
● A system with n users is scalable if the amount of resources needed to
support it is proportional to n (O(n))
● Example: Internet growth
● Ex. KPI: Number of users served per minute/hour/.
Aspects to be considered:
● Cost/Performance
● Resource availability (ex. IPv4 addresses)
The concept of cloud was born, among other reasons, to provide ad-hoc
scalability.
● Resource outsourcing

https://www.crn.com
Distributed Systems :: Introduction 25
 Concurrency
se ejecutan procesos en cada instante

● Simultaneous execution of processes

● A resource can be shared by several clients concurrently.
● Same issues addressed in concurrent and real-time
scheduling
– If the resource is replicated, it must preserve
consistency
● In general, for p processors and n processes,
concurrency is:
● Apparent if n > p
● Real if n <= p

● Ex. KPI: Number of users served concurrently

Distributed Systems :: Introduction 26

 Concurrency

● Concurrent processes must communicate and synchronize

● Basic form of communication and synchronization in a distributed
system
● Passing messages
– send (message, destination)
– receive (message, source)
● Synchronous and asynchronous mechanisms
● Direct or indirect communication
● Fixed or variable message size
● Group communication
– Multicast, broadcast

Distributed Systems :: Introduction 27

 Concurrency → Consistency

The consistency problem (coherence) arises when several

processes access and update data concurrently:
● Consistency of updates
● Coherence of replication
● Cache consistency
● Coherence in the face of failures
● Consistent clocks

Distributed Systems :: Introduction 28

 Reliability

● The probability that a system will work (meet your

requirements) for a period of time
● To achieve reliability you need to guarantee:
● Failure treatment
● Consistency

● Security

● Ex. KPI: Mean time between failures

● Ex. KPI: % service availability

Distributed Systems :: Introduction 29

 Reliability
Failure handling
Distributed systems often exhibit partial failure:
● The objective is to increase the availability of the distributed
system
● Availability: percentage of time a system is available for use
Fault tolerance:
● Detection
● Masking and insulation
● Failure recovery
● Redundancy

It must be considered from the design

Distributed Systems :: Introduction 30

 Security

Security has three components:

● Confidentiality
● Unauthorized access
● Integrity
● Data corruption or alteration
● Availability
● System or resource not accessible (DoS attacks)

Tools:
● Security at different levels
● SSL, Public Key Infrastructure, VPN, etc.
● KPI: Number of security incidents per day/week/month

Distributed Systems :: Introduction 31

 Transparency

Hiding details of the system components from the user:

● Access: The same interface (operations) to access local and remote
resources.
● Location: Access is independent of the location of the resource.
● Concurrency: Several processes access the same resource
concurrently.
● Replication: Users are not aware of the existence of replicas.
● Of failures: Fails are hidden from clients (but not all).
● Mobility: of both resources and user without affecting their operations.
● Performance: allowing optimization according to the load.
● Scalability: it works the same way regardless of the amount of
resources or clients.

Distributed Systems :: Introduction 32

 What kind(s) of transparency provides? Topología que pone varios caminos para ir de un sitio a otro de la red,
redundancia (evitar fallos) el camino se reconfigura por otra línea
El árbol que se genera a partir del grafo evitando los bucles en switches

Spanning Tree Protocol

DIY
● Replicación y tolerancia a fallos (siempre van juntas)

● e-mail Localización

Nos hacen creer que una colección

● RAID (Redundant Array of Independent Disks) de discos es uno solo, lo que hacen
depende del tipo de RAID
Mirroring (Copia de datos) -> Replicación y transparencia a fallos

● URL Localización

● DNS Nombres simbólicos (indirección). No dice como se debe hacer la elección, si una maquina esta carga le da la IP de otra maquina
Localización, Transparencia de fallos (DNS primario y secundario...). Transparencia de replicación (Replicado en un montón de
servidores base de datos distribuida)

● TCP Transparencia de fallos

● VRRP(Virtual Router Redundancy Protocol)

Distributed Systems :: Introduction 33

 Middleware

A middleware is a software infrastructure (which may include tools,

libraries, protocols, services, etc.) that aims to facilitate the
development of applications in distributed environments.
● It allows masking the heterogeneity of the underlying networks:
● Supports various programming models, programming
languages, etc.
● It supports various operating systems
● Provide a common programming model for developers
● They provide common structural services to all applications.
● Naming, security, persistence, transactions, etc.

Distributed Systems :: Introduction 34

 Middleware

Applications, services

Middleware

Operating system
Platform

Computer and network hardware

Source: Colouris

Platform: the lowest-level hardware and software layers of the

architecture (ej. Intel/x86, SPARC/SunOS)
Middleware: a software layer aimed at masking the complexity
of the platform (ej. CORBA, RMI, DCOM)
Applications, services: the software built on top of middleware

Distributed Systems :: Introduction 35

 Middleware
Increases developer productivity
The main goal of middleware is to isolate the developer from the non-
functional features of his application/service.
● Ideally, problems with resource location, failures, performance,
scalability, etc. should be transparent to the developer.
From the developer's point of view:
● Isolate yourself from the SD particularities
● Make your development transparent
● Focus on functional requirements
● Non-functional requirements should be provided by the platform

The term middleware is coined.

Distributed Systems :: Introduction 36

 Middleware
How to achieve homogeneity

Universal references
● A unique non-ambiguous way to locate resources.

● Resource Naming.

Canonical representation
● Common network-level data representation and

format.
Language bindings
● Definition of common interfaces.

● The contract between peers.

● How to map representations to different
programming languages.
Distributed Systems :: Introduction 37
 Message passing

Fundamental model of distributed applications:

● A sending process sends a request message.
● The message arrives at the receiving process, which processes the
request and returns a reply message.
● This reply can generate further requests from the sending process.

Services exchange messages defined for each specific application.

● Data Oriented
● Each message must
be interpreted
● Coupled services

Distributed Systems :: Introduction 38

 Distributed Computing paradigms

Source: Colouris
● Sockets
● Client/Server
● Remote invocation (RPC / RMI)
● Indirect communication
Distributed Systems :: Introduction 39
 Sockets
IPC mechanism (ie, provides communication between
processes) eventually running on different computers.
● It introduced TCP/IP in UNIX
● Virtually universal low-level interface
● Access to link, network and transport layers
● Very efficient, but complex and error-prone
● Communication protocol agnostic
● Distributed application features are implemented ad-hoc
● A protocol is designed for each application.

Distributed Systems :: Introduction 40

 Client/Server
● Assigns different roles to the processes that
communicate: client and server
● Client: request a service from the server
– Active entity: invokes requests
● Server: offers a service

– Manages resources (hardware, software, data)

– Passive entity: wait for incoming requests
● Clients and servers can be objects or processes running
on different computers
● Examples:
● Internet services (HTTP, FTP, DNS, etc.)

Distributed Systems :: Introduction 41

 Client/Server with sockets
Example: TCP server
TCP server
Transparency?
socket() Reliability?
Scalability?
bind()
TCP client
listen()
socket()

accept() connect()

recv() send()

processing
send() recv()

close() close()

Distributed Systems :: Introduction 42

 RPC: Remote Procedure Call
Idea: Make distributed software programming the same way
a conventional (non-distributed) application.
● Conceptually the same as invoking a local (same node)
procedure

Distributed Systems :: Introduction 43

 Google gRPC
github.com/grpc

Fuente https://geeks.ms

Fuente: https://indico.cern.ch/

Distributed Systems :: Introduction 44

 RMI: Distributed Objects
● RPC counterpart for OOP.
● Resources are encapsulated as objects that are accessed by
clients
● An object is an abstraction of operations + state

● Execute object methods as if they were local.

● Client's Stubs: Representation of a remote object in local
(including proxies).
● Server's stubs: executes the method on the invoked object
(performs the dispaching).

Distributed Systems :: Introduction 45

 RMI: Distributed Objects
ZeroC Ice
● Internet Communications Engine: distributed object
oriented middleware.
● Include “common services” for:
● persistence
● location of object references
● replication
● load balancing
● service management
● replication
● event-oriented programming, etc.

Distributed Systems :: Introduction 46

ZeroC Ice
Development model
module Example {
interface Printer {
void write(string s);
};
};

http://doc.zeroc.com/display/Ice/Slice+Compilation
Distributed Systems :: Introduction 47
 Indirect Communication
Some applications are better suited to an event-driven paradigm.
● Events distributed by publishers are generated in event channels
that are...
● filtered (source vs. destination)
● grouped
● analyzed
● replicated
● Received by subscribers
● An event management application (broker) manages the
distribution of events and subscriptions

Scalability and efficiency are the key.

Distributed Systems :: Introduction 48

Indirect Communication
● pull/push model
● Event channel
management
● Filtering:
source/destination,
type, content, etc.

Distributed Systems :: Introduction 49

Indirect communication
RabbitMQ

Source: https://cloudamqp.com
Distributed Systems :: Introduction 50
 Fundamental models
Forma en la que se especifica el funcionamiento de un sistema distribuido

Models that allow us to focus on fundamental

aspects of DS to be more specific:
● Interaction Model

● Failure Model

● Security Model

Distributed Systems :: Introduction 51

 Interaction model
Tipos de entidades

Any DS involves process communication by passing messages to

achieve a goal:
● Communication: interaction flow
●Example: Multiple servers, peer processes
● Coordination: synchronization and order
Two crucial factors in the interaction:
● Communication channel performance (latency, bandwidth, jitter)
● Lack of global clock (clock drift and synchronization):
two processes running on different computers could associate
different timestamps to the events
Two models: synchronous vs. asynchronous

Distributed Systems :: Introduction 52

 Interaction model
Synchronous distributed systems
● Minimum and maximum time limit for each process execution step
● Each transmitted message is received in a limited and known time.
● Each process has its local clock and its known drift

Asynchronous distributed systems

● In practice, most of them.
● They don’t make any assumptions about the relative speeds of the
processes, nor the delays of the messages, nor the drifts
● Channels are reliable, but there is no time limit to message delivery
● Communication between processes is the only way to synchronize

Distributed Systems :: Introduction 53

 Interaction model
Event ordering
●
The relative order of events is more important than the timestamp
●
Logical clocks, event management without physical clocks

send receive receive

X
1 m1 4
m2
send
receive
Y 2 3 Physical
receive time

send
Z
receive receive

m3 m1 m2 User A:
A
receive receive receive 1. From Z: Re: Meeting
t1 t2 t3 2. From X: Meeting
3. From Y: Re: Meeting

Distributed Systems :: Introduction 54

 Failure model
Definition and classification of the failures that can affect a DS:
● Basis for the analysis of potential effects and for the design
of fault-tolerant systems
● Types of failure:
● Omission: Process or communication channel does not
do what it is supposed to do
● Time: not done in time
– Only synchronous distributed systems
● Arbitrary (Byzantine): any kind of error can occur

Distributed Systems :: Introduction 55

 Failure model

● Channel failure vs. process failure

process p process q

send m receive

Communication channel
Outgoing message buffer Incoming message buffer

Distributed Systems :: Introduction 56

Failure model

Distributed Systems :: Introduction 57

 Failure model

http://www.cs.hut.fi/Opinnot/T-106.5250/Cmaps/Ch05/Programming%20models%20for%20distributed%20systems.html
Distributed Systems :: Introduction 58
 Security model
Definition and classification of attacks/threats that can
affect a DS:
● Basis for analysis of potential threats and for building
robust systems
Safety can be achieved:
● Protecting objects
● Authorization: only to beneficiaries of the (main) rights
● Authentication: third party/mainframe identification

● Ensuring processes and their interactions

(communication channels)

Distributed Systems :: Introduction 59

 Security model
●
Protecting resources
Access rights Object
invocation

Client
result Server

Principal (user) Network Principal (server)

●
Securing processes and their interactions
Copy of m

The enemy
m’
Process p m Process q
Communication channel

Distributed Systems :: Introduction 60

 Security model

http://www.cs.hut.fi/Opinnot/T-106.5250/Cmaps/Ch05/Programming%20models%20for%20distributed%20systems.html
Distributed Systems :: Introduction 61
 References
G. Coulouris, Distributed Systems: Concepts and
Design, Addison Wesley 2011
● Chapter 1 – Characterization of distributed

systems.
● Chapter 2 – System models

Distributed Systems :: Introduction 63