Technical Principles of

HBase

www.huawei.com

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved.

 Objectives
 Upon completion of this course, you will be able to know:
 System architecture of HBase
 Key features of HBase
 Basic functions of HBase
 Huawei enhanced features of HBase

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 2
 Contents
1. Introduction to HBase

2. Functions and Architecture of HBase

3. Key Processes of HBase

4. Huawei Enhanced Features of HBase

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 3
HBase Overview
 HBase is a column-based distributed storage system that
features high reliability, performance, and scalability.
 HBase is suitable for storing big table data (which contains billions of rows
and millions of columns) and allows real-time data access.

 HBase uses HDFS as the file storage system to provide a distributed

column-oriented database system that allows real-time data reading and
writing.

 HBase uses ZooKeeper as the collaboration service.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 4
HBase vs. RDB

HBase RDB

1. Distributed storage and

column-oriented. 1. Fixed data structure.
2. Dynamic extension of 2. Pre-defined data
columns. structure.
3. Supports common 3. I/O intensive and cost-
commercial hardware, consuming expansion.
lowering the expansion cost.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 5
Application Scenarios of HBase
 HBase applies to the following scenarios:
 Massive data (TB and PB)
 The Atomicity, Consistency, Isolation, Durability (ACID) feature supported
by traditional relational databases is not required.
 High throughput
 Efficient random reading of massive data
 High scalability
 Simultaneous processing of structured and unstructured data

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 6
Position of HBase in FusionInsight
Application service layer
OpenAPI/SDK REST/SNMP/Syslog

Data Information Knowledge Wisdom

DataFarm Porter Miner Farmer Manager
System
management
Hadoop API Plugin API
Service
governance
HIVE M/R Spark Storm Flink
Hadoop LibrA
YARN/ Zookeeper Security
management
HDFS/HBase

 HBase is a column-based distributed storage system that features high

reliability, performance, and scalability. It stores massive data and is designed
to eliminate limitations of relational databases in the processing of mass data.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 7
Data Stored By Row

ID Name Phone Address

 Data is stored by row in an underlying file system. Generally, a fixed amount

of space is allocated to each row.
 Advantages: Data can be added, modified, or read by row.
 Disadvantages: Some unnecessary data is obtained when data in a column is
queried.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 8
Data Stored by Column

ID Name Phone Address

 Data is stored by column in an underlying file system.

 Advantages: Data can be read or calculated by column.
 Disadvantages: When a row is read, multiple I/O operations may be
required.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 9
KeyValue Storage Model (1)
ID Name Phone Address

Key-01 Value-ID01 Key-01 Value-Name01

Key-01 Value-Phone01 Key-01 Value-Address01

 KeyValue has a specific structure. Key is used to quickly query a data record,
and Value is used to store user data.
 As a basic user data storage unit, KeyValue must store some description of
itself, such as timestamp and type information. This requires some structured
space.
 Data can be expanded dynamically, adaptive to changes of data types and
structures. Data is read and written by block. Different Columns are not
associated, so are tables.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 10
KeyValue Storage Model (2)
 Partition mode of a KeyValue Database - based on continuous Key range.

Region_01 Region_02 Region_05 Region_06 Region_09 Region_10

Region_03 Region_04 Region_07 Region_08 Region_11 Region_12

Node1 Node2 Node3

Region_01 Region_05 Region_02 Region_06 Region_03 Region_07

Region_09 Region_04 Region_10 Region_12 Region_11 Region_08

Data subregions are created based on the RowKey range (sorting based on a sorting
algorithm such as the alphabetic order based on RowKeys). Each subregion is a basic
distributed storage unit.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 11
KeyValue Storage Model (3)

 The underlying data of HBase exists in the form of KeyValue. KeyValue has a
specific format.
 KeyValue contains key information such as timestamp and type, etc.
 The same key can be associated with multiple Values. Each KeyValue has a
qualifier.
 There can be multiple KeyValues associated with the same Key and Qualifier.
In this case, they are distinguished using timestamps. This is why there are
multiple versions of the same data record.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 12
 Contents
1. Introduction to HBase

2. Functions and Architecture of HBase

3. Key Processes of HBase

4. Huawei Enhanced Features of HBase

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 13
HBase Architecture (1)

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 14
HBase Architecture (2)
 Store: A Region consists of one or

multiple Stores. Each store corresponds

to a Column Family.

 MemStore: A Store contains one MemStore.

Data inserted to a Region by client is

cached to the MemStore.

 StoreFile: The data flushed to the HDFS is stored as a StoreFile in the HDFS.

 Hfile: HFile defines the storage format of StoreFiles in a file system. HFile is underlying
implementation of StoreFile.

 Hlog: HLogs prevent data loss when a RegionServer is faulty. Multiple Regions in a
RegionServer share the same HLog.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 15
HMaster (1)

"Hey, Region A, please move to

RegionServer 1!"
“RegionServer 2 was gone! Let others take
over it’s Regions!"

RegionServer1 RegionServer2 RegionServer3

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 16
HMaster (2)
 The HMaster process manages all the RegionServers.
 Handles RegionServer failovers.

 The HMaster process performs cluster operations including creating,

modifying, and deleting tables.

 The HMaster process migrates Regions.

 Allocates Regions when a new table is created.

 Ensures load balancing during operation.

 Takes over Regions after a RegionServer failover occurs.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 17
RegionServer
 RegionServer is the data service
Region process of HBase and is responsible
RegionServer for processing reading and writing
requests of user data.

 RegionServer manages Regions. All

Region
reading and writing requests of user
data are handled based on interaction
among Regions on RegionServers.

Region  Regions can be migrated between

RegionServers.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 18
Region (1)
 A data table is divided horizontally into subtables based on the
KeyValue range to implement distributed storage. A subtable is called
a Region in HBase.
 Each Region is associated with a KeyValue range, which is described
using a StartKey and an EndKey.
 Each Region only needs to record a StartKey, because its EndKey serves as
the StartKey of the next Region.

 Region is the most basic distributed storage unit of HBase.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 19
Region (2)
Row001
Row001
Row002 Region-1
Row002 StartKey, EndKey
………..
……….. Row010
Row010
Row011
Row011 Row012 Region-2
Row012 ……….. StartKey, EndKey
……….. Row020
Row020 Row021
Row021 Row022 Region-3
Row022 ……….. StartKey, EndKey
……….. Row030
Row030 Row031
Row031 ……….. Region-4
……….. ……….. StartKey, EndKey
………..

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 20
Region (3)
META
Region

Region Region Region Region Region

 Regions are categorized as Meta Region and User Region.

 Meta Region records routing information of User Regions.
 Perform the following steps to access data in a Region:
 Search for the address of the Meta Region.
 Search for the address of the User Regions in the Meta Region.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 21
Column Family
Region Region Region Region
/HBase/table
/region-1/ColumnFamily-1
/region-1/ColumnFamily-2

/region-2/ColumnFamily-1
/region-2/ColumnFamily-2
/HBase/table
/region-1 /region-3/ColumnFamily-1
/region-2 /region-3/ColumnFamily-2
/region-3
HDFS

 A ColumnFamily is a physical storage unit of a Region. Multiple column families of the

same Region have different paths in HDFS.
 ColumnFamily information is table-level configuration information. That is, multiple
Regions of the same table have the same column family information. (For example,
each Region has two column families and the configuration information of the same
column family of different Regions is the same.)

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 22
ZooKeeper
ZooKeeper provides the following functions for HBase:
 Distributed lock service
 Multiple HMaster processes will try registering a node in ZooKeeper and the node can be
registered only by one HMaster process. The process that successfully registers the node
becomes the active HMaster process.

 Event listening mechanism

 The active Hmaster’s record is deleted after the active process fails and the standby
processes will receive an update message which indicates the Active HMaster is down.

 Micro database roles

 ZooKeeper stores the addresses of RegionServers. In this case, ZooKeeper can be regarded
as a micro database.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 23
MetaData Table
User Table 1
 The MetaData Table HBase:Meta
stores the information about
Regions to locate the Specific
Region for Client.

 The MetaData Table is splitted User Table N

into multiple Regions，and

metadata information of Region is
stored in ZooKeeper.
Mapping relation
Metadata Table
User table

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 24
 Contents
1. Introduction to HBase

2. Functions and Architecture of HBase

3. Key Processes of HBase

4. Huawei Enhanced Features of HBase

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 25
Writing Process

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 26
Client Initiating a Data Writing Request

Client

 The process of initiating a writing request by a client is like sending

books to a library by a book supplier. The book supplier must
determine to which building and floor the books should be sent.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 27
Writing Process - Locating a Region

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 28
Writing Process - Grouping Data (1)

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 29
Writing Process - Grouping Data (2)
 Data groups includes two

division steps:
 Find the information of

region and regionserver

of tables based on the

meta table

 Transfer data to specific region according to rwokey

 Data on each RegionServer is sent at the same time. In this case, the
data has been divided by Region.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 30
Writing Process - Sending a Request to
a RegionServer
 Data is sent using the encapsulated RPC
framework of HBase.

 Operations of sending requests to multiple

RegionServers are implemented concurrently.

 After sending a data writing request, a client

waits for the request processing result.

 If the client does not capture any exception, it

deems that all data has been written successfully.
If writing the data fails completely or partially,
the client can obtain a detailed KeyValue list
relevant to the failure.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 31
Writing Process - Process of Writing
Data to a Region

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 32
Writing Process - Flush
MemStore-1
（ColumnFamily-1）
HFile
Region

MemStore-2 HFile
（ColumnFamily-2）

 In either of the following scenarios, a Flush operation of Memstore is

triggered:
 The total usage of MemStore of a Region reaches the predefined Flush Size
threshold.
 The ratio of occupied memory to total memory of RegionServer reaches the
threshold.
 The number of WALs reaches the threshold.
 Memstore is updated every 1 hour by default.Hbase
 Users can flush a table or Region separately by a shell command.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 33
Impacts of Multiple HFiles

As time passes by, the number of HFiles increases and a query request
will take much more time.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 34
Compaction (1)
 Compaction aims to reduce the number of small files in a column family in a
Region, thereby increasing reading performance.
 There are two kinds of compaction: major and minor.
 Minor: compaction covering a small range. Minimum and maximum numbers of
files are specified. Small files at a consecutive time duration are combined.
 Major: compaction covering the HFiles in a column family in a Region. During
major compaction, deleted data is cleared.

 Files are selected based on a certain algorithm during minor compaction.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 35
Compaction (2)

Write
put MemStore
Flush

HFile HFile HFile HFile HFile HFile HFile

Minor Compaction

HFile HFile HFile

Major Compaction

HFile

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 36
Region Split
 A common Region splitting operation is
performed to split a Region into two subregions
if the data size of the Region exceeds the Parent
predefined threshold. Region

 During splitting, the split Region suspends

the reading and writing services. During
splitting, data files of the parent Region are
not split and rewritten to the two subregions.
Reference files are created in the new Region
to achieve quick splitting. Therefore, services
of the Region are suspended only for a short
time. DaughterRegion-2

 Routing information of the parent Region

DaughterRegion-1
cached in clients must be updated.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 37
Reading Process

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 38
Client Initiating a Data Reading Request
Get  When a precise key is provided, the
Get operation is performed to read a
single row of user data.

Scan  The Scan operation is to batch scan

user data of a specified Key range.
Client

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 39
Locating a Region

Hi, META, I want to look for books whose code ranges is

from xxx to xxx, please find the bookshelf number and the
floor information within the code range.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 40
OpenScanner
ColumnFamily-1
MemStore
HFile-11
HFile-12
Region
ColumnFamily-2
MemStore
HFile-21
HFile-22

 During the OpenScanner process, scanners corresponding to

MemStore and each HFile are created:
 The scanner corresponding to HFile is StoreFileScanner.

 The scanner corresponding to MemStore is MemStoreScanner.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 41
Filter
 Filter allows users to set filtering criteria during the Scan
 Satisfied Row
operation. Only user data that meets the criteria returns.
 There are some typical Filter types:
 Satisfied Row
 RowFilter
 SingleColumnValueFilter
 KeyOnlyFilter
 FilterList
 Satisfied Row

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 42
BloomFilter
 BloomFilter is used to optimize scenarios where data is randomly read, that is,
scenarios where the Get operation is performed. It can be used to quickly
check whether a piece of user data exists in a large dataset (most data in the
dataset cannot be loaded to the memory).

 A certain error rate exists when BloomFilter checks whether a piece of data
exits. Nevertheless, the conclusion indicated by the message "User data XXXX
does not exist" is accurate.

 The data relevant to BloomFilter of HBase is stored in HFiles.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 43
 Contents
1. Introduction to HBase

2. Functions and Architecture of HBase

3. Key Processes of HBase

4. Huawei Enhanced Features of HBase

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 44
Supporting Secondary Index
 The secondary index enables HBase to query data based on specific column
values.
Column Family A Column Family B
RowKey A:Name A:Addr. A:Age B:Mobile B:Email
01 ZhangSan Beijing 23 6875349 ……
02 LiLei Hangzhou 43 6831475 ……
03 WangWu Shenzhen 35 6809568 ……
04 …… Wuhan 28 6812645 ……
05 …… Changsha 26 6889763 ……
06 …… Jinan 35 6854912 ……

When the secondary index is not used, the mobile field needs to be matched in the entire table by row
to search for specified mobile numbers such as ‘68XXX’ which results in long time delay.
When the secondary index is used, the index table is searched first to identify the location of the
mobile number, which narrows down the search scope and reduces the time delay.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 45
HFS
 HBase FileStream (HFS) is a separate module of Hbase. As an
encapsulation of Hbase and HDFS interfaces, HFS provides
capabilities, such as storing, reading and deleting files for
upper-level applications.

 HFS provides the ability of storing massive small files and large
files in HDFS。 That is, massive small files (less than 10MB) and
some large files (larger than 10MB) can be stored in HBase.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 46
HBase MOB (1)
 MOB Data（100KB to 10MB）is directly stored in the file
system (HDFS for example)as HFile. And the information about
address and size of file is stored in HBase as a value. With tools
managing these files, the frequency of compation and split
can be highly reduced, and performance can be improved.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 47
HBase MOB (2)

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 48
 Summary
 This module describes the following information about HBase:
KeyValue Storage Model, technical architecture, reading and
writing process and enhanced features of FusionInsight HBase.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 49
 Quiz
1. Can the services of the Region in HBase be provided when splitting?

2. What are the advantages of the Region splitting of HBase?

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 50
 Quiz
1. What is Compaction used for? ( )

A. Reducing the number of files in a column family and Region

B. Improving data reading performance

C. Reducing the number of files in a column family

D. Reducing the number of files in a Region

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 51
 Quiz
1. What is the physical storage unit of HBase? ( )
A. Region

B. Column Family

C. Column

D. Cell

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 52
 More Information
 Training materials:
 http://support.huawei.com/learning/Certificate!showCertificate?lang=en&pbiPath=term100002
5450&id=Node1000011796
 Exam outline:
 http://support.huawei.com/learning/Certificate!toExamOutlineDetail?lang=en&nodeId=Node10
00011797
 Mock exam:
 http://support.huawei.com/learning/Certificate!toSimExamDetail?lang=en&nodeId=Node10000
11798
 Authentication process:
 http://support.huawei.com/learning/NavigationAction!createNavi#navi[id]=_40

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 53
 Thank You
www.huawei.com

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 54