Data sharding and replication

Genoveva Vargas Solar

French Council of scientific research, LIG-LAFMIA, France
[email protected]
http://www.vargas-solar.com
NoSql Stores: availability and 2
performance
n Replication
n Copy data across multiple servers
(each bit of data can be found in
multiple servers)
n Increase data availability
n Faster query evaluation

n Sharding
n Distribute different data across
multiple servers
n Each server acts as the single source
of a data subset

n Orthogonal techniques
Replication: pros & cons 3

n Data is more available n Increased updates cost

n Failure of a site containing E n Synchronisation: each replica
does not result in unavailability must be updated
of E if replicas exist
n Increased complexity of
n Performance concurrency control
n Parallelism: queries processed n Concurrent updates to distinct
in parallel on several nodes replicas may lead to
n Reduce data transfer for local inconsistent data unless
data special concurrency control
mechanisms are implemented
Sharding: why is it useful? 4

n Scaling applications by reducing n Improve read and write performance

Smaller amount of data in each user group implies faster
data sets in any single databases n
querying
n Segregating data n Isolating data into smaller shards accessed data is more
likely to stay on cache
n Sharing application data More write bandwidth: writing can be done in parallel
Securing sensitive data by
n
n n Smaller data sets are easier to backup, restore and
isolating it manage
Load%balancer%

n Massively work done

Web%3% Cache%1%
n Parallel work: scale out across more nodes
n Parallel backend: handling higher user loads
Web%2%
n Share nothing: very few bottlenecks
Cache%2%
Web%1%
n Decrease resilience improve availability
n If a box goes down others still operate
Cache%3%
n But: Part of the data missing
MySQL%
Master%
MySQL% Resume%database%
Master%
Site%database%
Sharding and replication
n Sharding with no replication: unique copy, distributed data sets
n (+) Better concurrency levels (shards are accessed independently)
n (-) Cost of checking constraints, rebuilding aggregates
n Ensure that queries and updates are distributed across shards

n Replication of shards
n (+) Query performance (availability)
n (-) Cost of updating, of checking constraints, complexity of concurrency control

n Partial replication (most of the times)

n Only some shards are duplicated

5
Contact: Genoveva Vargas-Solar, CNRS, LIG-LAFMIA
[email protected]
http://www.vargas-solar.com/teaching/

6
References
n Eric A., Brewer "Towards robust distributed systems." PODC. 2000

n Rick, Cattell "Scalable SQL and NoSQL data stores." ACM SIGMOD Record 39.4 (2011): 12-27

n Juan Castrejon, Genoveva Vargas-Solar, Christine Collet, and Rafael Lozano, ExSchema:
Discovering and Maintaining Schemas from Polyglot Persistence Applications, In Proceedings of
the International Conference on Software Maintenance, Demo Paper, IEEE, 2013

n M. Fowler and P. Sadalage. NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot
Persistence. Pearson Education, Limited, 2012

n C. Richardson, Developing polyglot persistence applications,

http://fr.slideshare.net/chris.e.richardson/developing-polyglotpersistenceapplications-
gluecon2013
7
 8

NOSQL STORES: AVAILABILITY AND

PERFORMANCE
Replication master - slave 9

Master'
n Helps with read scalability but does not help with
all'updates'
made'to'the'master'
write scalability

reads'can'be'done' n Read resilience: should the master fail, slaves can

from'master'or'slaves' still handle read requests
changes'propagate''
To'slaves'

n Master failure eliminates the ability to handle writes

until either the master is restored or a new master is
appointed

n Biggest complication is consistency

Slaves'
n Possible write – write conflict
n Makes one node the authoritative copy/replica that n Attempt to update the same record at the same
handles writes while replica synchronize with the master time from to different places
and may handle reeds

n Master is a bottle-neck and a point of failure

n All replicas have the same weight
n Replicas can all accept writes
n The lose of one of them does not prevent access to
the data store
Master-slave replication management
n Masters can be appointed
n Manually when configuring the nodes cluster
n Automatically: when configuring a nodes cluster one of them elected as master. The master can appoint a new master
when the master fails reducing downtime

n Read resilience
n Read and write paths have to be managed separately to handle failure in the write path and still reads can occur
n Reads and writes are put in different database connections if the database library accepts it

n Replication comes inevitably with a dark side: inconsistency

n Different clients reading different slaves will see different values if changes have not been propagated to all slaves
n In the worst case a client cannot read a write it just made
n Even if master-slave is used for hot backups, if the master fails any updates on to the backup are lost

10
Replication: peer-To-Peer 11

Master'

n Deals with inconsistencies

n Replicas coordinate to avoid
all'nodes'read'
conflict
Network traffic cost for
and'write'all'data'
nodes'communicate' n
their'writes'
coordinating writes
n Unnecessary to make all replicas
agree to write, only the majority
n Survival to the loss of the minority
of replicas nodes
n Allows writes to any node; the nodes coordinate to n Policy to merge inconsistent writes
synchronize their copies
n Full performance on writing to any
n The replicas have equal weight
replica
Sharding 12

n Puts different data on separate nodes

Each%shard%reads%and% n Each user only talks to one servicer
so she gets rapid responses
writes%its%own%data%

n The load should be balanced out

nicely between servers

n Ability to distribute both data and n Ensure that

load of simple operations over many
n data that is accessed together is
servers, with no RAM or disk shared
among servers clumped together on the same
node
n A way to horizontally scale writes
n Improve read performance n that clumps are arranged on the

n Application/data store support nodes to provide best data access

 13

Sharding

Database laws Principle

n Small databases are fast n Start with a big monolithic
database
n Big databases are slow n Break into smaller databases
n Keep databases small n Across many clusters
n Using a key value
Instead of having one million customers information
on a single big machine ….

100 000 customers on smaller and different machines

+ Sharding criteria 14

n Partitioning
n Relational: handled by the DBMS (homogeneous DBMS)
n NoSQL: based on ranging of the k-value

n Federation
n Relational
n Combine tables stored in different physical databases
n Easier with denormalized data
n NoSQL:
n Store together data that are accessed together
n Aggregates unit of distribution
 15

Sharding

Architecture Process
Pick a dimension that helps sharding easily
Each application server (AS) is
n
n (customers, countries, addresses)
running DBS/client
n Pick strategies that will last a long time as
repartition/re-sharding of data is operationally
difficult
n Each shard server is running
This is done according to two different principles
n a database server n

n Partitioning: a partition is a structure that

divides a space into tow parts
n replication agents and query
n Federation: a set of things that together
agents for supporting parallel compose a centralized unit but each individually
maintains some aspect of autonomy
query functionality

Customers data is partitioned by ID in shards using an

algorithm d to determine which shard a customer ID belongs to
Replication: aspects to consider 16

n Conditioning n Important elements to consider

n Data to duplicate
n Copies location
Fault
Performance tolerance n Duplication model (master –
slave / P2P)
n Consistency model (global –
copies)

Transparency
levels Availability
à Find a compromise !
 17

PARTITIONING
A PARTITION IS A STRUCTURE THAT DIVIDES A SPACE INTO TOW PARTS
Background: distributed relational 18
databases
n External schemas (views) are often subsets
of relations (contacts in Europe and
America)

n Access defined on subsets of relations:

80% of the queries issued in a region have
to do with contacts of that region

n Relations partition
n Better concurrency level
n Fragments accessed independently

n Implications
n Check integrity constraints

n Rebuild relations
 19

Fragmentation
n Horizontal n Hybrid
n Groups of tuples of the same relation
n Budget < 300 000 or >= 150 000
n Not disjoint are more difficult to manage

n Vertical
n Groups attributes of the same relation
n Separate budget from loc and pname of
the relation project
 20

Fragmentation: rules

Vertical Horizontal
Tuples of the same fragment must be statistically homogeneous
n Clustering n

n If t1 and t2 are tuples of the same fragment then t1 and t2 have

the same probability of being selected by a query
n Grouping elementary fragments
n Budget and location information in two n Keep important conditions
relations n Complete
n Every tuple (attribute) belongs to a fragment (without
information loss)
n Splitting n If tuples where budget >= 150 000 are more likely to be
selected then it is a good candidate
n Decomposing a relation according to n Minimum
affinity relationships among attributes n If no application distinguishes between budget >= 150 000
and budget < 150 000 then these conditions are unnecessary
 21

Sharding: horizontal partitioning

Load%balancer%
n The entities of a database are split into two or
more sets (by row) Web%3% Cache%1%

n In relational: same schema several physical Web%2%

bases/servers
Cache%2%
n Partition contacts in Europe and America shards Web%1%
where they zip code indicates where the will be found
n Efficient if there exists some robust and implicit way to
identify in which partition to find a particular entity MySQL%
Master% Cache%3%

n Last resort shard MySQL%

Master%
n Needs to find a sharding function: modulo, round
robin, hash – partition, range - partition MySQL% MySQL%
Slave%1% MySQL% Slave%n%
Slave%2%

MySQL% Odd%IDs%
MySQL% Slave%n%
Slave%1% MySQL%
Slave%2%

Even%IDs%
 22

FEDERATION
A FEDERATION IS A SET OF THINGS THAT TOGETHER COMPOSE A CENTRALIZED UNIT BUT EACH
INDIVIDUALLY MAINTAINS SOME ASPECT OF AUTONOMY
FEDERATION: vertical SHARDING 23

n Principle Load%balancer%
n Partition data according to their logical
affiliation Web%3% Cache%1%
n Put together data that are commonly accessed

Web%2%
n The search load for the large partitioned entity can
be split across multiple servers (logical and Cache%2%
Web%1%
physical) and not only according to multiple indexes
in the same logical server
MySQL%
Cache%3%
n Different schemas, systems, and physical Master%

bases/servers
MySQL%
Master%
n Shards the components of a site and not only data
Internal%
user%
MySQL%
Slave%1%

MySQL% Resume%database%
MySQL% Slave%n%
Slave%1% MySQL%
Slave%2%

Site%database%
 24

NOSQL STORES: PERSISTENCY MANAGEMENT

 «memcached»
n «memcached» is a memory management protocol based on a cache:
n Uses the key-value notion
n Information is completly stored in RAM

n «memcached» protocol for:

n Creating, retrieving, updating, and deleting information from the
database
n Applications with their own «memcached» manager (Google,
Facebook, YouTube, FarmVille, Twitter, Wikipedia)

25
Storage on disc (1)
n For efficiency reasons, information is stored using the RAM:
n Work information is in RAM in order to answer to low latency requests
n Yet, this is not always possible and desirable

Ø The process of moving data from RAM to disc is called "eviction”; this
process is configured automatically for every bucket

26
Storage on disc (2)
n NoSQL servers support the storage of key-value pairs on disc:
n Persistency–can be executed by loading data, closing and
reinitializing it without having to load data from another source
n Hot backups– loaded data are sotred on disc so that it can be
reinitialized in case of failures
n Storage on disc– the disc is used when the quantity of data is
higher thant the physical size of the RAM, frequently used
information is maintained in RAM and the rest es stored on disc

27
Storage on disc (3)
n Strategies for ensuring:
n Each node maintains in RAM information on the key-value pairs it stores.
Keys:
n may not be found, or
n they can be stored in memory or on disc
n The process of moving information from RAM to disc is asynchronous:
n The server can continue processing new requests
n A queue manages requests to disc
Ø In periods with a lot of writing requests, clients can be notified that the
server is termporaly out of memory until information is evicted

28
 29

NOSQL STORES: CONCURRENCY CONTROL

Multi version concurrency control
(MVCC)
n Objective: Provide concurrent access to the database and in programming languages to implement transactional memory

n Problem: If someone is reading from a database at the same time as someone else is writing to it, the reader could see a
half-written or inconsistent piece of data.

n Lock: readers wait until the writer is done

n MVCC:
n Each user connected to the database sees a snapshot of the database at a particular instant in time
n Any changes made by a writer will not be seen by other users until the changes have been completed (until the transaction has been
committed
n When an MVCC database needs to update an item of data it marks the old data as obsolete and adds the newer version elsewhere à
multiple versions stored, but only one is the latest
n Writes can be isolated by virtue of the old versions being maintained
n Requires (generally) the system to periodically sweep through and delete the old, obsolete data objects

30