Lec 18
Lec 18
Lecture - 18
Design of HBase
Design of HBase,
1. What is HBase?
2. HBase architecture
3. HBase components
4. Data model
5. HBase storage hierarchy
6. Cross-data center replication
7. Auto sharding and distribution
8. Bloom filter and fold, store and shift.
51)
Let us see some of the important aspects of the HBase architecture. So, HBase has the region servers and
these region servers are basically handling the regions and there is one HBase master and this zookeeper
has to interact with the HBase master and all the other component and HBase, then as HBase also, deals
with the data nodes. So, HBase master has to communicate with the region servers and zookeeper. So, we
will see, in more detail about this. So, HBase architecture here, the table is split into the regions and
served by the region servers. So, regions are vertically divided by the column families into the stores that
we will discuss later on. And stores saved as files on HDFS. HBase utilizes zookeeper for the distributed
coordination service.
So, this is the typical layout of the data model which is also called as the, ‘Column Families’. So, data is
stored in HBase and is located by its "rowkey". So, "rowkey" is the primary key from the notion of
relational database management system. So, the records in HBase are stored in the sorted order according
to the rowkeys and the data in a row are grouped together as the column families and each column family
has one or more columns. These columns in a family are stored together in a low level storage file which
is called as, ‘HFile’.
So, the tables are divided into the sequence of rows, by the key range called ‘Regions’. So, here we can
see that, this particular key range is basically nothing but, the row 1 and this will be, stored together into
the region. So, these regions are then assigned to the data node in the cluster called the ‘Region Servers’.
Now, that is shown over here. So, again for example, let us say that the range the sequence or the key
range, the "rowkey” range R2 will store these set off the rows, sequence of the rows and this will be
stored in another region server and the region servers, these regions are managed by the data nodes, they
are called they are called the ‘Region Servers’. Region servers as the data nodes.
Now, cell in HBase. A table data is stored in HBase table cells. And cell is a combination of row, column
family, column qualifier and contains a value and a timestamp. So, the key consists of the row key,
column name and a timestamp that is shown here and the entire cell, with the added, structural
information is called the, ‘Key Value Pair’.
So, HBase data model consists of a table. HBase organizes data into the table and table names are the
strings composed of the characters that are safe for use in the file system path. And the rows within the
table, the data is stored according to its row. Rows are identified by the arrow key and row keys do not
have the data type and are always treated as the byte [ ] (byte array). So, this aspect we have already
covered. So, column family the data within the row is grouped by the column family. Every row in the
table has the same column family, although the row need not store the data in all its families. Column
families are the strings and composed of characters that are safe for use in the file system path.
And timestamp the values within the cell are versioned. Versions are identified by their version number,
which by default is the timestamp. If the timestamp is not is specified for the read, the latest one is
returned. The number of cell values versions retained by the HBase is configured for each family. The
default number of cell versions is three.
Let, us see in more detail once again the HBase architecture. So, HBase has this one a client. Client can
access to the HRegionServer and this HRegionServer are many HRegionServers. one such
HRegionServer is shown over here which has HLog and each HRegionServer is further divided into
different Hregions. One such Hregion is shown over here and Hregions will contain the store and also a
MemStore. So, within the store it will be having the StoreFile and StoreFile will contain basic storage that
is called, ‘HFile’ and ‘HFile’ is stored in HDFS. Now, as far as there is one HMaster and HMaster
communicates with the zookeeper and with the HRegionServers and HDFS, we will see, we have seen
about the HMaster and what is zookeeper? Is a small group of servers which runs consensus protocol like
Paxos and the zookeeper is the coordination service for HBase and assigns the different nodes and servers
to this particular service if zookeeper is not there then HBase will stop functioning.
So, this is about the HBase HFile. So, HFile comprises, of data and Meta file, Metafile information
indices and trailer where the data portion consists of the magic value and (key, value) pairs. So, the key
value pair is having the key length, value length, row length, row and column family length, column
family, column qualifier, timestamp, key type and value. And these rows will have SSN numbers and
column family will have the demographic information and column qualifier will have ethnicity
information and this becomes the HBase key.
Now, HBase prefers the strong consistency or availability. So, HBase prefers the write-ahead log and
whenever a client comes with the key (K 1, K 2, K 3, K 4) gives to the HRegionServer then this let us say,
(K 1 and K 2) will be on a particular HRegion and (K 3, K4) will have another H Region and then this
particular aspect will be stored in the store and these store will have the MemStore, StoreFiles and
internally they are stored in as HFile. So, right to HLog before writing to MemStore is there to ensure the
fault tolerance aspect and recovery from the failures. So, this helps recover from the failure by replaying
HLog.
Now, cross-datacenter replication now there will be a single “Master” cluster, others “Slave” cluster
replicate the same table, master cluster synchronously sends HLogs over the slaves clusters, coordination
among the cluster is done by a zookeeper. Zookeeper can be used like a file system to store the control
information and also this particular zookeeper will use different paths for invoking the state and peer
cluster number and each log all this information is there.
Now, let us see how the auto sharding is done and Auto sharding means that tables is divided into the row,
range or a range keys and they are being stored in the region servers and we and these region servers are
now solved with the client.
Now, similarly the table logical view, we can see that the rows are now split into the row keys, in the
range keys and these range keys are given charted into the region servers and we have shown here that
these rows from A to Z are stored in three different region servers.
And this layout is there and it is automatically done by that is called, ‘Auto Sharding’ and ‘Distribution’.
So, unit of the scalability in HBase is the region that we have seen which is managed by the region
servers and they are sorted and contiguous range of rows spread randomly across the RegionServers,
moved around for the load balancing and failover that we have already seen in the previous slide. So, is
split automatically or manually to scale with the growing data and capacity is only a factor of cluster
nodes versus the regions per node.
So, let us see the positioning of bloom filter into this HFile. So, at the bottom of HFile this particular the
bloom filter information is stored and this will help in identifying which of these different blocks are
basically containing the block index or information. So, using bloom filter you can access at least two at
most two blocks not the entire set of block sequentially has to be read. So, hence the access is reduced
from 6 seeks per block to the 2 seeks per block using the bloom filter and bloom filter stone we is restored
as a part of s five.
Now, as far as fold store and shift is concerned you can see here that, for a particular row, the same data
is stored, multiple instances because the time series data is also stored at different time stamp the data is
stored for a particular row key. So, that is why? Several data on a particular row key is now appearing
they may be varying at their timestamp T 1, T 2 and T 3 and they are a store, into the, into the five.
Conclusion: Traditional databases (RDBMSs) works with strong consistency and offers acid property and
in the modern workload doesn't need such strong guarantees, but do need fast response time that is the
availability. Unfortunately, CAP provides three out of two that is here given the partition tolerance, the
HBase prefers the consistency over the availability. So, key-value pair or a NoSQL system offers the
BASE property in these scenarios. So, Cassandra offers the eventual consistency and the other variety of
consistency models are striving towards the strong consistency. So, in this lecture we have covered about
the HBase architecture components, data model, storage hierarchy, cross datacenter replication auto
starting, distribution and bloom filter and fold store and shift operation. Thank you.