Distributed System Notes
Distributed System Notes
The first and foremost thing is that it makes our system more stable because of node replication.
It is good to have replicas of a node in a network due to following reasons:
If a node stops working, the distributed network will still work fine due to its replicas which will
be there. Thus it increases the fault tolerance of the system.
It also helps in load sharing where loads on a server are shared among different replicas.
It enhances the availability of the data. If the replicas are created and data is stored near to the
consumers, it would be easier and faster to fetch data.
Types of Replication
Active Replication
Passive Replication
Active Replication:
Advantages:
It is really simple. The codes in active replication are the same throughout.
It is transparent.
Even if a node fails, it will be easily handled by replicas of that node.
Disadvantages:
It increases resource consumption. The greater the number of replicas, the greater the memory
needed.
It increases the time complexity. If some change is done on one replica it should also be done in
all others.
Passive Replication:
The client request goes to the primary replica, also called the main replica.
There are more replicas that act as backup for the primary replica.
Primary replica informs all other backup replicas about any modification done.
The response is returned to the client by a primary replica.
Periodically primary replica sends some signal to backup replicas to let them know that it is
working perfectly fine.
In case of failure of a primary replica, a backup replica becomes the primary replica.
Advantages:
The resource consumption is less as backup servers only come into play when the primary server
fails.
The time complexity of this is also less as there’s no need for updating in all the nodes replicas,
unlike active replication.
Disadvantages:
To reduce access time of data caching is used. The effect of replication and caching
increases complexity and overhead of consistency management.
Different Consistency models offer different degrees of consistency.
Depending on the order of operations allowed and how much inconsistency can be
tolerated, so consistency models range from strong to weak.
Data centric model:
In this model it is guarantee that the results of read and write operations which is
performed on data can be replicated to various stores located nearby immediately.
Fig(a) Data Centric model
Fig (b) Client centric model
Strict consistency:
It is the strongest data centric consistency model as it requires that a write on a data be
immediately available to all replicas.
This model states that “Any read on data item x returns a value corresponding to the
result of the most recent write on x.
In a distributed system, it is very difficult to maintain a global time ordering, along with
the availability of data which is processed concurrently at different locations, which
causes delays for transferring of data from one location to another.
Thus Strict consistency model is impossible to achieve.
Eg P1 & P2 are two processes. P1 performs a write operation on its data item x and
modifies its value to a.
Update is propogated to all other replicas
Suppose if P2 now reads and finds its value to be NIL due to the propogation delay.
So the result of this value read by P2 is not strictly consistent.
But as the law of Strictly consistent model says that results should be immediately
propogated to all the replicas as shown in figure below
Client centric model explanation
Monotonic Reads:
It states that “If a process P reads the value of a data item t, any successive read operation
on x by that process at later time will always return that same or a recent value.
For eg Each time one connects to the email server (may be different replicas),the email
server guarantees that all the time will always return that same or recent value.
Here, in fig (a) operations WS(x1) at L1 before the second read operation. Thus
WS(x1,x2) at L2 would see the earlier update.
Consistency Model
A consistency model is contract between a distributed data store and processes, in which the
processes agree to obey certain rules in contrast the store promises to work correctly.
A consistency model basically refers to the degree of consistency that should be maintained for
the shared memory data.
If a system supports the stronger consistency model, then the weaker consistency model is
automatically supported but the converse is not true.
The types of consistency models are Data-Centric and client centric consistency models.
A data store may be physically distributed across multiple machines. Each process that can
access data from the store is assumed to have a local or nearby copy available of the entire store.
Any read on a data item X returns a value corresponding to the result of the most recent write
on X
This is the strongest form of memory coherence which has the most stringent consistency
requirement.
Strict consistency is the ideal model but it is impossible to implement in a distributed system. It
is based on absolute global time or a global agreement on commitment of changes.
ii.Sequential Consistency
iii.Linearizability
It that is weaker than strict consistency, but stronger than sequential consistency.
A data store is said to be linearizable when each operation is timestamped and the result of any
execution is the same as if the (read and write) operations by all processes on the data store
were executed in some sequential order
The operations of each individual process appear in sequence order specified by its program.
If tsOP1(x)< tsOP2(y), then operation OP1(x) should precede OP2(y) in this sequence.
iv.Causal Consistency
v.FIFO Consistency
vi.Weak consistency
The basic idea behind the weak consistency model is enforcing consistency on a group of
memory reference operations rather than individual operations.
A Distributed Shared Memory system that supports the weak consistency model uses a special
variable called a synchronization variable which is used to synchronize memory.
When a process accesses a synchronization variable, the entire memory is synchronized by
making visible the changes made to the memory to all other processes.
vii.Release Consistency
Release consistency model tells whether a process is entering or exiting from a critical section so
that the system performs either of the operations when a synchronization variable is accessed
by a process.
Two synchronization variables acquire and release are used instead of single synchronization
variable. Acquire is used when process enters critical section and release is when it exits a
critical section.
Release consistency can be viewed as synchronization mechanism based on barriers instead of
critical sections.
viii.Entry Consistency
In entry consistency every shared data item is associated with a synchronization variable.
In order to access consistent data, each synchronization variable must be explicitly acquired.
Release consistency affects all shared data but entry consistency affects only those shared data
associated with a synchronization variable.
Client-centric consistency models aim at providing a system wide view on a data store.
This model concentrates on consistency from the perspective of a single mobile client.
Client-centric consistency models are generally used for applications that lack simultaneous
updates were most operations involve reading data.
i.Eventual Consistency
In Systems that tolerate high degree of inconsistency, if no updates take place for a long time all
replicas will gradually and eventually become consistent. This form of consistency is called
eventual consistency.
Eventual consistency only requires those updates that guarantee propagation to all replicas.
Eventual consistent data stores work fine as long as clients always access the same replica.
Write conflicts are often relatively easy to solve when assuming that only a small group of
processes can perform updates. Eventual consistency is therefore often cheap to implement.
A data store is said to provide monotonic-read consistency if a process reads the value of a data
item x, any successive read operation on x by that process will always return that same value or
a more recent value.
A process has seen a value of x at time t, it will never see an older version of x at a later time.
Example: A user can read incoming mail while moving. Each time the user connects to a
different e-mail server, that server fetches all the updates from the server that the user
previously visited. Monotonic Reads guarantees that the user sees all updates, no matter from
which server the automatic reading takes place.
iii.Monotonic Writes
A data store is said to provide read-your-writes consistency if the effect of a write operation by a
process on data item x will always be a successive read operation on x by the same process.
A write operation is always completed before a successive read operation by the same process
no matter where that read operation takes place.
Example: Updating a Web page and guaranteeing that the Web browser shows the newest
version instead of its cached copy.
A data store is said to provide writes-follow-reads consistency if a process has write operation
on a data item x following a previous read operation on x then it is guaranteed to take place on
the same or a more recent value of x that was read.
Any successive write operation by a process on a data item x will be performed on a copy of x
that is up to date with the value most recently read by that process.
Example: Suppose a user first reads an article A then posts a response B. By requiring writes-
follow-reads consistency, B will be written to any copy only after A has been written.
Any system has two major components – Hardware and Software. Fault may occur in either of it.
So there are separate techniques for fault-tolerance in both hardware and software.
Hardware Fault-tolerance Techniques:
Making a hardware fault-tolerance is simple as compared to software. Fault-tolerance techniques
make the hardware work proper and give correct result even some fault occurs in the hardware
part of the system. There are basically two techniques used for hardware fault-tolerance:
1. BIST –
BIST stands for Build in Self Test. System carries out the test of itself after a certain
period of time again and again, that is BIST technique for hardware fault-tolerance.
When system detects a fault, it switches out the faulty component and switches in the
redundant of it. System basically reconfigure itself in case of fault occurrence.
2. TMR –
TMR is Triple Modular Redundancy. Three redundant copies of critical components are
generated and all these three copies are run concurrently. Voting of result of all redundant
copies are done and majority result is selected. It can tolerate the occurrence of a single
fault at a time.
1. N-version Programming –
In N-version programming, N versions of software are developed by N individuals or
groups of developers. N-version programming is just like TMR in hardware fault-
tolerance technique. In N-version programming, all the redundant copies are run
concurrently and result obtained is different from each processing. The idea of n-version
programming is basically to get the all errors during development only.
2. Recovery Blocks –
Recovery blocks technique is also kike the n-version programming but in recovery blocks
technique, redundant copies are generated using different algorithms only. In recovery
block, all the redundant copies are not run concurrently and these copies are run one by
one. Recovery block technique can only be used where the task deadlines are more than
task computation time.
Recovery from an error is essential to fault tolerance, and error is a component of a system that
could result in failure. The whole idea of error recovery is to replace an erroneous state with an
error-free state. Error recovery can be broadly divided into two categories.
1. Backward Recovery:
Moving the system from its current state back into a formerly accurate condition from an
incorrect one is the main challenge in backward recovery. It will be required to accomplish this
by periodically recording the system’s state and restoring it when something goes wrong. A
checkpoint is deemed to have been reached each time (part of) the system’s current state is
noted.
2. Forward Recovery:
Instead of returning the system to a previous, checkpointed state in this instance when it has
entered an incorrect state, an effort is made to place the system in a correct new state from which
it can continue to operate. The fundamental issue with forward error recovery techniques is that
potential errors must be anticipated in advance. Only then is it feasible to change those mistakes
and transfer to a new state.
These two types of possible recoveries are done in fault tolerance in distributed system.
Stable Storage :
Stable storage, which can resist anything but major disasters like floods and earthquakes, is
another option. A pair of regular discs can be used to implement stable storage. Each block on
drive 2 is a duplicate of the corresponding block on drive 1, with no differences. The block on
drive 1 is updated and confirmed first whenever a block is updated. then the identical block on
drive 2 is finished.
Suppose that the system crashes after drive 1 is updated but before the update on drive 2. Upon
recovery, the disk can be compared with blocks. Since drive 1 is always updated before drive 2,
the new block is copied from drive 1 to drive 2 whenever two comparable blocks differ, it is safe
to believe that drive 1 is the correct one. Both drives will be identical once the recovery process
is finished.
Another potential issue is a block’s natural deterioration. A previously valid block may suddenly
experience a checksum mistake without reason . The faulty block can be constructed from the
corresponding block on the other drive when such an error is discovered.
Checkpointing :
Backward error recovery calls for the system to routinely save its state onto stable storage in a
fault-tolerant distributed system. We need to take a distributed snapshot, often known as a
consistent global state, in particular. If a process P has recorded the receipt of a message in a
distributed snapshot, then there should also be a process Q that has recorded the sending of that
message. It has to originate somewhere, after all.
Each process periodically saves its state to a locally accessible stable storage in backward error
recovery techniques. We must create a stable global state from these local states in order to
recover from a process or system failure. Recovery to the most current distributed snapshot, also
known as a recovery line, is recommended in particular. In other words, as depicted in Fig., a
recovery line represents the most recent stable cluster of checkpoints.
Coordinated Checkpointing :
As the name suggests, coordinated checkpointing synchronises all processes to write their state
to local stable storage at the same time. Coordinated checkpointing’s key benefit is that the saved
state is automatically globally consistent, preventing cascading rollbacks that could cause a
domino effect.
Message Logging :
The core principle of message logging is that we can still obtain a globally consistent state even
if the transmission of messages can be replayed, but without having to restore that state from
stable storage. Instead, any communications that have been sent since the last checkpoint are
simply retransmitted and treated appropriately.
As system executes, messages are recorded on stable storage. A message is called as logged if
its data and index of stable interval that is stored are both recorded on stable storage. In above
Fig. you can see logged and unlogged images denoted by different arrows. The idea is if
transmission of messages is replayed, we can still reach a globally consistent state. so we can
recover logs of messages and continue the execution.
These measures may be insufficient, to identify attacks at the network level without help from
other sources. We can not only prevent malicious actors from gaining access to our machines
from other machines in the same firewall but can also monitor our own actions.
Reckless data sharing can significantly increase exposure to both the threats themselves and the
costs entailed in defending against them.
Security in a distributed system poses unique challenges that need to be considered when
designing and implementing systems. A compromised computer or network may not be the only
location where data is at risk; other systems or segments may also become infected with
malicious code. Because these types of threats can occur anywhere, even across distances in
networks with few connections between them, new research has been produced to help determine
how well distributed security architectures are actually performing.
In the past, security was typically handled on an end-to-end basis. All the work involved in
ensuring safety occurred “within” a single system and was controlled by one or two
administrators. The rise of distributed systems has created a new ecosystem that brings with it
unique challenges to security.
Distributed systems are made up of multiple nodes working together to achieve a common goal,
these nodes are usually called peers.
Security Requirements and Attacks Related to Distributed Systems:
A distributed system is composed of many independent units, each designed to run its own tasks
without communicating with the rest of them except through messaging service. This means that
a single point of failure can render a system completely incapable without any warning since
there is no single point that can perform all necessary operations.
Attacks related to distributed systems are an area of active research. There were two main
schools of thought, those who believed that network worms could be stopped by employing
firewalls and those who did not.
A firewall might do nothing about worms and their ability to spread across various types of
networks, especially wireless networks and the Internet. This was because although firewalls
were able to stop intruders from gaining access through the firewall, they were unable to stop a
worm from self-replicating.
To summarize, there are numerous attacks that can be used against a network worm that has to
do with breaking functionality and altering data, or simply deleting it.
Passive eavesdroppers − They monitor the messages and get hold of private
information.
Active attackers − They not only monitor the messages but also corrupt data by inserting
new data or modifying existing data.
Security measures encompass security in communications, security in data and data auditing.
Communications Security
In a distributed database, a lot of data communication takes place owing to the diversified
location of data, users and transactions. So, it demands secure communication between users and
databases and between the different database environments.
Two popular, consistent technologies for achieving end-to-end secure communications are −
Data Security
Authentication and authorization − These are the access control measures adopted to
ensure that only authentic users can use the database. To provide authentication digital
certificates are used. Besides, login is restricted through username/password combination.
Data encryption − The two approaches for data encryption in distributed systems are −
o Internal to distributed database approach: The user applications encrypt the data
and then store the encrypted data in the database. For using the stored data, the
applications fetch the encrypted data from the database and then decrypt it.
o External to distributed database: The distributed database system has its own
encryption capabilities. The user applications store data and retrieve them without
realizing that the data is stored in an encrypted form in the database.
Validated input − In this security measure, the user application checks for each input
before it can be used for updating the database. An un-validated input can cause a wide
range of exploits like buffer overrun, command injection, cross-site scripting and
corruption in data.
Data Auditing
A database security system needs to detect and monitor security violations, in order to ascertain
the security measures it should adopt. It is often very difficult to detect breach of security at the
time of occurrences. One method to identify security violations is to examine audit logs. Audit
logs contain information such as −
All the above information gives an insight of the activities in the database. A periodical analysis
of the log helps to identify any unnatural activity along with its site and time of occurrence. This
log is ideally stored in a separate server so that it is inaccessible to attackers.
Kerberos
Kerberos provides a centralized authentication server whose function is to authenticate users to
servers and servers to users. In Kerberos Authentication server and database is used for client
authentication. Kerberos runs as a third-party trusted server known as the Key Distribution
Center (KDC). Each user and service on the network is a principal.
Kerberos Overview:
Step-1:
User login and request services on the host. Thus user requests for ticket-granting service.
Step-2:
Authentication Server verifies user’s access right using database and then gives ticket-granting-
ticket and session key. Results are encrypted using the Password of the user.
Step-3:
The decryption of the message is done using the password then send the ticket to Ticket
Granting Server. The Ticket contains authenticators like user names and network addresses.
Step-4:
Ticket Granting Server decrypts the ticket sent by User and authenticator verifies the request
then creates the ticket for requesting services from the Server.
Step-5:
The user sends the Ticket and Authenticator to the Server.
Step-6:
The server verifies the Ticket and authenticators then generate access to the service. After this
User can access the services.
Kerberos Limitations
Each network service must be modified individually for use with Kerberos
It doesn’t work well in a timeshare environment
Secured Kerberos Server
Requires an always-on Kerberos server
Stores all passwords are encrypted with a single key
Assumes workstations are secure
May result in cascading loss of trust.
Scalability
Is Kerberos Infallible?
No security measure is 100% impregnable, and Kerberos is no exception. Because it’s been
around for so long, hackers have had the ability over the years to find ways around it, typically
through forging tickets, repeated attempts at password guessing (brute force/credential stuffing),
and the use of malware, to downgrade the encryption.
Despite this, Kerberos remains the best access security protocol available today. The protocol is
flexible enough to employ stronger encryption algorithms to combat new threats, and if users
employ good password-choice guidelines, you shouldn’t have a problem!
Although Kerberos can be found everywhere in the digital world, it is commonly used in secure
systems that rely on robust authentication and auditing capabilities. Kerberos is used for Posix,
Active Directory, NFS, and Samba authentication. It is also an alternative authentication system
to SSH, POP, and SMTP.
Confidentiality
Message Integrity
In the SSL Record Protocol application data is divided into fragments. The fragment is
compressed and then encrypted MAC (Message Authentication Code) generated by algorithms
like SHA (Secure Hash Protocol) and MD5 (Message Digest) is appended. After that encryption
of the data is done and in last SSL header is appended to the data.
Handshake Protocol:
Handshake Protocol is used to establish sessions. This protocol allows the client and server to
authenticate each other by sending a series of messages to each other. Handshake protocol uses
four phases to complete its cycle.
Phase-1: In Phase-1 both Client and Server send hello-packets to each other. In this IP session,
cipher suite and protocol version are exchanged for security purposes.
Phase-2: Server sends his certificate and Server-key-exchange. The server end phase-2 by
sending the Server-hello-end packet.
Phase-3: In this phase, Client replies to the server by sending his certificate and Client-exchange-
key.
Phase-4: In Phase-4 Change-cipher suite occurred and after this Handshake Protocol ends.
SSL Handshake Protocol Phases diagrammatic representation
Change-cipher Protocol:
This protocol uses the SSL record protocol. Unless Handshake Protocol is completed, the SSL
record Output will be in a pending state. After the handshake protocol, the Pending state is
converted into the current state.
Change-cipher protocol consists of a single message which is 1 byte in length and can have only
one value. This protocol’s purpose is to cause the pending state to be copied into the current
state.
Alert Protocol:
This protocol is used to convey SSL-related alerts to the peer entity. Each message in this
protocol contains 2 bytes.
This Alert breaks the connection between sender and receiver. The connection will be stopped,
cannot be resumed but can be restarted. Some of them are :
Handshake failure: When the sender is unable to negotiate an acceptable set of security
parameters given the options available.
Decompression failure: When the decompression function receives improper input.
Illegal parameters: When a field is out of range or inconsistent with other fields.
Bad record MAC: When an incorrect MAC was received.
Unexpected message: When an inappropriate message is received.
The second byte in the Alert protocol describes the error.
The advantage of this approach is that the service can be tailored to the specific needs of the
given application.
Secure Socket Layer was originated by Netscape.
SSL is designed to make use of TCP to provide reliable end-to-end secure service.
This is a two-layered protocol.
Versions of SSL:
1. Confidentiality: Information can only be accessed by the person for whom it is intended
and no other person except him can access it.
2. Integrity: Information cannot be modified in storage or transition between sender and
intended receiver without any addition to information being detected.
3. Non-repudiation: The creator/sender of information cannot deny his intention to send
information at later stage.
4. Authentication: The identities of sender and receiver are confirmed. As well as
destination/origin of information is confirmed.
Applications Of Cryptography:
1. Computer passwords
2. Digital Currencies
3. Secure web browsing
4. Electronic Signatures
5. Authentication
6. Cryptocurrencies
7. End-to-end encryption