C7 Hbase
C7 Hbase
NoSQL
Data Model
Architecture
Considerations
Highly Scalable
– Automatic partitioning (sharding) : sorted row distribution (not column-
distributed)
– Scale linearly and automatically with new nodes : uses HDFS
Low Latency (unlike Hadoop batch processing)
– Support random read/write, small range scan
Highly Available
Flexible and dynamic data model
– The schema does not need to be defined in advance which makes
HBase a natural fit for many BigData applications and some traditional
applications as well.
Strong Consistency
– All reads and writes are routed through a single RegionServer, which
guarantees that all writes happen in order, and all reads access the
most recently committed data.
Very good for “sparse data” (no fixed columns)
T” starts
https://www.geeksforgeeks.org/acid-properties-in-dbms/
HBase and other NoSQL distributed data stores are subject to the CAP
Theorem which states that distributed NoSQL data stores can only achieve
2 out of the 3 properties:
consistency, availability and partition tolerance.
Basically Available
• The system works all the time (available in case of failure).
Soft-state
• Stores don’t have to be write-consistent, nor do different replicas have to
be mutually consistent all the time.
• i.e The data source does not have to be consistent all the time
Eventual consistency
• Not immediately consistent, but at some point, data will be consistent—
no guarantee about when this will occur.
• Stores exhibit consistency at some later point (e.g., lazily at read time).
exhibit=dispose ,presente
Key
Key
All data are byte arrays, including table name, Column Family names,
and Column names (also called Column Qualifiers)
Key
HFile
HFile
HBase
Region
A subset of a Tables’ Rows
Region Server(s) Automatically sharded when
grown too large
Master Region Server
Region(s)
Hosts tables, performs reads,
buffers writes (Memstore)
Clients talk directly to them for
reads/writes
ZooKeeper HDFS
HBase Zookeeper
• Apache open source product and
part of the Hadoop ecosystem
Region Server(s) • Critical component for HBase
• Ensures one Master Server is
Master running
Region(s)
• Registers Region Servers
• Handles Region and Master Server
failures
• Integral part of HBase’s fault tolerant
architecture
• Keeps location of Hbase .META
table* (which RegionServer manage
ZooKeeper HDFS
it now)
* Hbase .META table contains metadata of all regions of all tables managed by cluster
23 © 2013 IBM Corporation
ZooKeeper
Zookeeper provides coordination service
Background services:
– LoadBalancer (move regions to balance the cluster load)
– CatalogJanitor (check and clean up the .META. Table)
– LogCleaner (clear the HLogs in the old logs directory)
When a master starts up, it races with other masters to write its
address into ZooKeeper. If it succeeds, it is the primary/active
master.
A backup master wait until the active master dies to try and become
the next active master
Region
Write-Ahead-Log
• WAL stores all the edits to the Store. There is one WAL per region server. All edits
for all regions carried by a particular region server are entered first in the WAL
• Region stores data for a certain region of a table. There are multiple stores for a
single region
• A Store holds a column family in a region. It has a memstore and a set of zero or
more Hfiles.
• MemStore is the write cache. Its main role is to store new data which has not yet
been written to disk. Memstore is flushed when data is transferred and saved in
Hfiles as blocks.
27 © 2013 IBM Corporation
Region Server
Region
Write-Ahead-Log
Store MemStore
…… …
StoreFile
… …
HFile …
Region
A region is a horizontal partition of a table with a start row and an end row
Regions are the basic element of availability and distribution for tables
A region is automatically split by the hosting region server when it reaches a
specified size
Periodically, a load balancer will move regions within the cluster to balance load
When a region server fails, its regions will be reassigned to other region servers
A Keys:[A-D]
Region
. Keys:[S-V]
Region
Table – Logical View
.
Keys:[I-M]
K
Region
.
Keys:[E-H]
. Region
T Keys:[N-R]
. Region
. Keys:[W-Z]
Region
Z
Auto-Sharded Regions
1. Write Ahead Log (WAL) is a file used to store new data that is yet to be put on permanent
storage. It is used for recovery in the case of failure. When a client issues a put request, it will
write the data to the write-ahead log (WAL).
32 © 2013 IBM Corporation
HBase Write Mechanism
2. MemStore is the write cache that stores new data that has not yet been written to disk.
There is one MemStore per column family per region. Once data is written to the WAL, it is
then copied to the MemStore.
33 © 2013 IBM Corporation
HBase Write Mechanism
3. Once the data is placed in MemStore, the client then receives the acknowledgment.
4. Hfiles store the rows of data as sorted KeyValue on disk. When the MemStore reaches the
threshold, it dumps or commits the data into an HFile.
HBase can run fine on your laptop -- but why would you…?