Basic Principles
Big Data Management
Anis Ur Rahman
Faculty of Computer Science & Information Technology
University of Malaya
October 15, 2019
Borrowed from http://www.ksi.mff.cuni.cz/~svoboda/courses/171-NDBI040/
1 / 55
Lecture Outline
Different aspects of data distribution
1 Scaling
Vertical vs. horizontal
2 Distribution models
Sharding
Replication: master-slave vs. peer-to-peer architectures
3 CAP properties
Consistency, availability and partition tolerance
ACID vs. BASE guarantees
4 Consistency
Read and write quora
2 / 55
Scalability
Outline
1 Scalability
2 Distribution Models
3 CAP Theorem
4 Consistency
3 / 55
Scalability
Scalability
What is scalability?
the capability of a system to handle growing amounts of data
and/or queries without losing performance, or
its potential to be enlarged in order to accommodate such a growth
Two general approaches
Vertical scaling
Horizontal scaling
4 / 55
Scalability
Vertical Scalability
Vertical scaling (scaling up/down)
Adding resources to a single node in a system
e.g. increasing the number of CPUs, extending system memory,
using larger disk arrays, ...
i.e. larger and more powerful machines are involved
Traditional choice
In favor of strong consistency
Easy to implement and deploy
No issues caused by data distribution
...
Works well in many cases but ...
5 / 55
Scalability
Vertical Scalability: Drawbacks
Performance limits
Even the most powerful machine has a limit
Everything works well ... until start approaching the limit
Higher costs
The cost of expansion increases exponentially
In particular, it is higher than the sum of costs of equivalent
commodity hardware
Proactive provisioning
New projects/applications might evolve rapidly
Upfront budget is needed when deploying new machines
So flexibility is seriously suppressed
6 / 55
Scalability
Vertical Scalability: Drawbacks
Vendor lock-in
There are only a few manufacturers of large machines
Customer is made dependent on a single vendor
Their products, services, but also implementation details,
proprietary formats, interfaces, support, ...
e.e. it is difficult or impossible to switch to another vendor
Deployment downtime
Inevitable downtime is often required when scaling up
7 / 55
Scalability
Horizontal Scalability
Horizontal scaling (scaling out/in)
adding more nodes to a system
i.e. system is distributed across multiple nodes in a cluster
Choice of many NoSQL systems
Advantages
Commodity hardware, cost effective
Flexible deployment and maintenance
Often surpasses the vertical scaling
Often no single point of failure
...
8 / 55
Scalability
Horizontal Scalability: Consequences
Significantly increases complexity
Complexity of management, programming model, ...
Introduces new issues and problems
Data distribution
Synchronization of nodes
Data consistency
Recovery from failures
...
And there are also plenty of false assumptions ...
9 / 55
Scalability
Horizontal Scalability: Fallacies
False assumptions
Network is reliable
Latency is zero
Bandwidth is infinite
Network is secure
Topology does not change
There is one administrator
Network is homogeneous
Transport cost is zero
Source: https://www.red-gate.com/simple-talk/blogs/
the-eight-fallacies-of-distributed-computing/
10 / 55
Scalability
Horizontal Scalability: Conclusion
A standalone node still might be a better option in certain cases
e.g. for graph databases
Simply because it is difficult to split and distribute graphs
In other words
It can make sense to run even a NoSQL database system on a
single node
No distribution at all is the most preferred/simple scenario
But in general, horizontal scaling really opens new possibilities
11 / 55
Scalability
Horizontal Scalability: Architecture
What is a cluster?
A collection of mutually interconnected commodity nodes
Based on the shared-nothing architecture
Nodes do not share their CPUs, memory, hard drives, ...
Each node runs its own operating system instance
Nodes send messages to interact with each other
Nodes of a cluster can be heterogeneous
Data, queries, calculations, requests, workload, ... all distributed
among the nodes within a cluster
12 / 55
Distribution Models
Outline
1 Scalability
2 Distribution Models
3 CAP Theorem
4 Consistency
13 / 55
Distribution Models
Distribution Models
Generic techniques of data distribution
1 Sharding
Idea. different data on different nodes
Motivation. increasing volume of data, increasing performance
2 Replication
Idea. the same data on different nodes
Motivation. increasing performance, increasing fault tolerance
Both the techniques are mutually orthogonal
i.e. we can use either of them, or combine them both
Distribution model
specific way how sharding and replication is implemented
NoSQL systems often offer automatic sharding and replication
14 / 55
Distribution Models
Sharding
Sharding (horizontal partitioning)
Placement of different data on different nodes
What different data means? Usually aggregates
e.g. key-value pairs, documents, ...
Related pieces of data that are accessed together should also
be kept together
Specifically, operations involving data on multiple shards should be
avoided (if possible)
The questions are...
how to design aggregate structures?
how to actually distribute these aggregates?
15 / 55
Distribution Models
Sharding
Source: Sadalage, Pramod J. - Fowler, Martin: NoSQL Distilled. Pearson Education, Inc., 2013.
16 / 55
Distribution Models
Sharding
Objectives
Achieve uniform data distribution
Achieve balanced workload (read and write requests)
Respect physical locations
e.g. different data centers for users around the world
...
Unfortunately, these objectives...
may mutually contradict each other
may change in time
So, how to actually determine shards for aggregates?
17 / 55
Distribution Models
Sharding
Source:
https://www.digitalocean.com/community/tutorials/understanding-database-sharding
18 / 55
Distribution Models
Sharding
Sharding strategies
Based on mapping structures
Data is placed on shards in a random fashion
e.g. round-robin, ...
Knowledge of the mapping of individual aggregates to particular
shards must then be maintained
Thus usually maintained using a centralized index structures with
all the disadvantages
Based on general rules
Each shard is responsible for storing certain data
Hash partitioning, range partitioning, ...
19 / 55
Distribution Models
Sharding
Key-based sharding
Source:
https://www.digitalocean.com/community/tutorials/understanding-database-sharding
20 / 55
Distribution Models
Sharding
Range-based sharding
Source:
https://www.digitalocean.com/community/tutorials/understanding-database-sharding
21 / 55
Distribution Models
Sharding
Directory-based sharding
Source:
https://www.digitalocean.com/community/tutorials/understanding-database-sharding
22 / 55
Distribution Models
Sharding
Should I Shard?
Amount of application data grows to exceed the storage capacity
of a single database node.
Volume of writes or reads to the database surpasses what a single
node can handle,
resulting in slowed response times or timeouts.
23 / 55
Distribution Models
Sharding
Why is sharding difficult?
Not only we need to be able to determine particular shards during
write requests
i.e. when a new aggregate is about to be inserted
So that we can actually make a decision where it should be
physically stored
but also during read requests
i.e. when existing aggregate/s are about to be retrieved
So that we can actually find and return them efficiently (or detect
they are missing)
And all that only based on the search criteria provided (e.g. key, id,
...) unless all the nodes should be accessed
24 / 55
Distribution Models
Sharding
Why is sharding even more difficult?
Structure of the cluster may be changing
Nodes can be added or removed
Nodes may have incomplete/obsolete cluster knowledge
Nodes involved, their responsibilities, sharding rules, ...
Individual nodes may be failing
Network may be partitioned
Messages may not be delivered even though sent
25 / 55
Distribution Models
Replication
Replication
Placement of multiple copies of the same data (replicas) on
different nodes
Replication factor = number of such copies
Two approaches
1 Master-slave architecture
2 Peer-to-peer architecture
26 / 55
Distribution Models
Replication
Master-Slave Architecture
Source: Sadalage, Pramod J. - Fowler, Martin: NoSQL Distilled. Pearson Education, Inc., 2013.
27 / 55
Distribution Models
Replication
Master-Slave Architecture
One node is primary (master), all the other secondary (slave)
Master node bears all the management responsibility
All the nodes contain identical data
Read requests can be handled by both the master or slaves
Suitable for read-intensive applications
More read requests to deal with → more slaves to deploy
When the master fails, read operations can still be handled
28 / 55
Distribution Models
Replication
Master-Slave Architecture
Write requests can only be handled by the master
Newly written replicas are propagated to all the slaves
Consistency issue
Luckily enough, at most one write request is handled at a time
But the propagation still takes some time during which obsolete
reads might happen
Hence certain synchronization is required to avoid conflicts
In case of master failure, a new one needs to be appointed
Manually (user-defined) or automatically (cluster-elected)
Since the nodes are identical, appointment can be fast
Master might therefore represent a bottleneck (because of the
performance or failures)
29 / 55
Distribution Models
Replication
Peer-to-Peer Architecture
Source: Sadalage, Pramod J. - Fowler, Martin: NoSQL Distilled. Pearson Education, Inc., 2013.
30 / 55
Distribution Models
Replication
Peer-to-Peer Architecture
All the nodes have equal roles and responsibilities
All the nodes contain identical data once again
Both read and write requests can be handled by any node
No bottleneck, no single point of failure
Both the operations scale well
More requests to deal with → more nodes to deploy
Consistency issues
Unfortunately, multiple write requests can be initiated
independently and being executed at the same time
Hence synchronization is required to avoid conflicts
31 / 55
Distribution Models
Sharding and Replication
Observations with respect to the replication
Does the replication factor really need to correspond to the
number of nodes?
No, replication factor of 3 will often be the right choice
Consequences
Nodes will no longer contain identical data
Replica placement strategy will be needed
Do all the replicas really need to be successfully written when
write requests are handled?
No, but consistency issues have to be tackled carefully
Sharding and replication can be combined... but how?
32 / 55
Distribution Models
Sharding and Replication
Sharding and Master-Slave Replication
Source: Sadalage, Pramod J. - Fowler, Martin: NoSQL Distilled. Pearson Education, Inc., 2013.
33 / 55
Distribution Models
Sharding and Replication
Sharding and Peer-to-Peer Replication
Source: Sadalage, Pramod J. - Fowler, Martin: NoSQL Distilled. Pearson Education, Inc., 2013.
34 / 55
Distribution Models
Sharding and Replication
Combinations of sharding and replication
1 Sharding + master-slave replication
Multiple masters, each for different data
Roles of the nodes can overlap
Each node can be master for some data and/or slave for other
2 Sharding + peer-to-peer replication
Basically placement of anything anywhere (although certain rules
can still be applied)
35 / 55
Distribution Models
Sharding and Replication
Questions to figure out for any distribution model
Can all the nodes serve both read and write requests?
Which replica placement strategy is used?
How the mapping of replicas is maintained?
What level of consistency and availability is provided?
What extent of infrastructure knowledge do the nodes have?
...
36 / 55
CAP Theorem
Outline
1 Scalability
2 Distribution Models
3 CAP Theorem
4 Consistency
37 / 55
CAP Theorem
CAP Theorem
Assumptions
Distributed system with sharding and replication
Read and write operations on a single aggregate only
CAP properties
Properties of a distributed system:
Consistency, Availability, and Partition tolerance
CAP theorem
It is not possible to have a distributed system that would
guarantee all CAP properties at the same time.
Only 2 of these 3 properties can be enforced.
But, what these properties actually mean?
38 / 55
CAP Theorem
CAP Properties
Consistency
Read and write operations must be executed atomically
There must exist a total order on all operations
Each operation looks as if it was completed at a single instant
i.e. as if all operations were executed sequentially one by one on a
single standalone node
Practical consequence. after a write operation, all readers see
the same data
Since any node can be used for handling of read requests, atomicity
of write operations means that changes must be propagated to all
the replicas
As we will see later on, other ways for such a strong consistency
exist as well
39 / 55
CAP Theorem
CAP Properties
Availability
If a node is working, it must respond to user requests
Every read or write request successfully received by a non-failing
node in the system must result in a response,
i.e. their execution must not be rejected
Partition tolerance
System continues to operate even when two or more sets of
nodes get isolated
The network is allowed to lose arbitrarily many messages sent
from one node to another
i.e. a connection failure must not shut the whole system down
40 / 55
CAP Theorem
CAP Theorem Consequences
If at most two properties can be guaranteed ...
1 CA = consistency + availability
Traditional ACID properties are easy to achieve
Examples.: RDBMS, Google BigTable
Any single-node system, but even clusters (at least in theory)
However, should the network partition happen, all the nodes must be
forced to stop accepting user requests
2 CP = consistency + partition tolerance
Other examples. distributed locking
3 AP = availability + partition tolerance
New concept of BASE properties
Examples. Apache Cassandra, Apache CouchDB
Other examples. web caching, DNS
41 / 55
CAP Theorem
CAP Theorem Consequences
42 / 55
CAP Theorem
CAP Theorem Consequences
Partition tolerance is necessary in clusters
Why? Because it is difficult to detect network failures
Does it mean that only purely CP and AP systems are possible?
No...
The real meaning of the CAP theorem:
The real-world does not need to be just black and white
Partition tolerance is a must, but we can trade off consistency
versus availability
Just a little bit relaxed consistency can bring a lot of availability
Such trade-offs are not only possible, but often works very well in
practice
43 / 55
CAP Theorem
CAP Theorem Example
You are asked to design a distributed cluster of 4 data nodes.
Replication factor is 2 i.e. any data written in cluster must be written on
2 nodes; so when one goes down – second can serve the data. Now try
to apply CAP theorem on this requirement.
In distributed system, two things may happen anytime i.e. node failure
(hard disk crash) or network failure (connection between two nodes go
down).
1 CP [Consistency/Partition Tolerance] Systems
2 AP [Availability/Partition Tolerance] Systems
3 CA [Consistency/Availability] Systems
44 / 55
CAP Theorem
ACID Properties
Traditional ACID properties
1 Atomicity
Partial execution of transactions is not allowed (all or nothing)
2 Consistency
Transactions bring the database from one consistent (valid) state
to another
3 Isolation
Transactions executed in parallel do not see uncommitted effects
of each other
4 Durability
Effects of committed transactions must remain durable
45 / 55
CAP Theorem
BASE Properties
New concept of BASE properties
1 Basically Available
The system works basically all the time
Partial failures can occur, but there are no total system failures
2 Soft State
The system is in flux (unstable), non-deterministic state
Changes occur all the time
3 Eventual Consistency
Sooner or later the system will be in some consistent state
BASE is just a vague term, no formal definition
Proposed to illustrate design philosophies at the opposite ends of
the consistency-availability spectrum
46 / 55
CAP Theorem
ACID and BASE
ACID
Choose consistency over availability
Pessimistic approach
Implemented by traditional relational databases
BASE
Choose availability over consistency
Optimistic approach
Common in NoSQL databases
Allows levels of scalability that cannot be acquired with ACID
Current trend in NoSQL. strong consistency → eventual consistency
47 / 55
CAP Theorem
CAP Theorem Conclusion
48 / 55
Consistency
Outline
1 Scalability
2 Distribution Models
3 CAP Theorem
4 Consistency
49 / 55
Consistency
Consistency
Consistency in general...
Consistency is the lack of contradiction in the database
However, it has many facets ... e.g.
only assume atomic operations always manipulating just a single
aggregate,
but set operations could also be considered etc.
Strong consistency is achievable even in clusters, but eventual
consistency might often be sufficient
1 One minute obsolete article on a news portal does not matter
2 Even when an already unavailable hotel room is booked once
again, the situation can still be figured out in the real world
3 ...
50 / 55
Consistency
Consistency
Write consistency (update consistency)
Problem. write-write conflict
Two or more write requests on the same aggregate are initiated
concurrently
Context. peer-to-peer architecture only
Issue. lost update
Solution.
1 Pessimistic strategies
Preventing conflicts from occurring
Write locks, ...
2 Optimistic strategies
Conflicts may occur, but are detected and resolved later on
Version stamps, vector clocks, ...
51 / 55
Consistency
Consistency
Read consistency (replication consistency)
Problem. read-write conflict
Write and read requests on the same aggregate are initiated
concurrently
Context. both master-slave and peer-to-peer architectures
Issue. inconsistent read
When not treated, inconsistency window will exist
Propagation of changes to all the replicas takes some time
Until this process is finished, inconsistent reads may happen
Even the initiator of the write request may read wrong data!
Session consistency/read-your-writes/sticky session
52 / 55
Consistency
Strong Consistency
How many nodes need to be involved to get strong consistency?
Write quorum. W > N /2
Idea. only one write request can get the majority
W = number of nodes successfully participating in the write
N = number of nodes involved in replication (replication factor)
Read quorum. R > N − W
Idea. concurrent write requests cannot happen
R = number of nodes participating in the read
Should the retrieved replicas be mutually different, the newest
version is resolved and then returned
When a quorum is not attained rightarrow the request cannot be
handled
53 / 55
Consistency
Strong Consistency
Examples for replication factor N = 3
Write quorum W = 3 and read quorum R = 1
All the replicas are always updated
Can read any one of them
Write quorum W = 2 and read quorum R = 2
Typical configuration, reasonable trade-off
Consequence
Quora can be configured to balance read and write workload
The higher the write quorum is required lower the read quorum
can then be required
54 / 55
Consistency
Lecture Conclusion
There is a wide range of options influencing...
Scalability – how well the entire system scales?
Availability – when nodes may refuse to handle user requests?
Consistency – what level of consistency is required?
Latency – how long does it take to handle user requests?
Durability – is the committed data written reliably?
Resilience – can the data be recovered in case of failures?
it’s good to know these properties and choose the right trade-off
55 / 55
