0% found this document useful (0 votes)
36 views36 pages

C7 Hbase

Uploaded by

nesrine.chaouani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
36 views36 pages

C7 Hbase

Uploaded by

nesrine.chaouani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 36

HBase

April 29, 2024 © 2013 IBM Corporation


Agenda
 Overview

 NoSQL

 Data Model

 Architecture

 Sharding and Scalability

 Considerations

2 © 2013 IBM Corporation


What is HBase?

 Open Source, a distributed Hadoop database which has its genesis


in the Google’s Bigtable.
 A NoSQL data store
 Non-relational, column-oriented DBMS adapted to storage and fast
access to massive data.
 Hbase is now “Hadoop DBMS”
 Hive is an overlay of HBase offering similar functionalities to SQL.
 HBase powers some of the leading sites on the Web (e.g. Facebook,
Yahoo, …)

3 © 2013 IBM Corporation


Why HBase?

 Highly Scalable
– Automatic partitioning (sharding) : sorted row distribution (not column-
distributed)
– Scale linearly and automatically with new nodes : uses HDFS
 Low Latency (unlike Hadoop batch processing)
– Support random read/write, small range scan
 Highly Available
 Flexible and dynamic data model
– The schema does not need to be defined in advance which makes
HBase a natural fit for many BigData applications and some traditional
applications as well.
 Strong Consistency
– All reads and writes are routed through a single RegionServer, which
guarantees that all writes happen in order, and all reads access the
most recently committed data.
 Very good for “sparse data” (no fixed columns)

4 © 2013 IBM Corporation


HBase and the ACID Properties

The total amount before and after the


transaction must be maintained.
Total before T occurs = 500 + 200
Total after T occurs = 400 + 300
X= 500, Y = 500.

T” starts

Updates and modifications to the


database are stored in (written to disk)
and they persist even if a system
failure occurs.

https://www.geeksforgeeks.org/acid-properties-in-dbms/

5 © 2013 IBM Corporation


HBase and the ACID Properties
How does HBase adhere to the ACID Properties?
 Atomicity : guarantee that all changes inside transaction will all be
successfully applied or none in case of failure.
– All reading and writing of data in one region is done by the assigned Region
Server
– All clients have to talk to the assigned Region Server to get to the data
– Provides row level atomicity!
 Consistency and Isolation
– All rows returned via any access API will consist of a complete row that existed
at some point in the table's history
– A scan is not a consistent view of a table. Scans do not exhibit snapshot
isolation. Any row returned by the scan will be a consistent view (i.e. that
version of the complete row existed at some point in time)
 Durability
– All visible data is also durable data. That is to say, a read will never return data
that has not been made durable on disk
Note: Concurrency on writes or mutations
– HBase will automatically get a lock before a write and release that lock after the
write
– Also the user can control the locking manually
6 © 2013 IBM Corporation
HBase and the ACID Properties
CAP Theorem

HBase and other NoSQL distributed data stores are subject to the CAP
Theorem which states that distributed NoSQL data stores can only achieve
2 out of the 3 properties:
consistency, availability and partition tolerance.

Basically Available
• The system works all the time (available in case of failure).
Soft-state
• Stores don’t have to be write-consistent, nor do different replicas have to
be mutually consistent all the time.
• i.e The data source does not have to be consistent all the time
Eventual consistency
• Not immediately consistent, but at some point, data will be consistent—
no guarantee about when this will occur.
• Stores exhibit consistency at some later point (e.g., lazily at read time).
exhibit=dispose ,presente

7 © 2013 IBM Corporation


Data Model
 “...a sparse, distributed, persistent, multi-dimensional sorted map. The map
is indexed by a row key, column key, and a timestamp; each value in the map
is an uninterrupted array of bytes” – Google Bigtable paper
 Data is stored in HBase table(s)
 Tables are made of rows and columns : multidimensional map
 All columns in HBase belong to a particular column family
 Table schema only defines column families
– Can have large, variable number of columns per row

 (row key, column key, timestamp)  value


 A {row, column, version} tuple exactly specifies a cell
 Each cell value has a version
– Timestamp

 Row stored in order by row keys


– Row keys are byte arrays; lexicographically sorted

8 © 2013 IBM Corporation


Think “Map” not Spreadsheet Model
 Most literature describes HBase as a Column A Column B Column C
Column-Oriented data store which it is
Row A
but this can lead to confusion
 http://en.wikipedia.org/wiki/Associative_array Row B
– “In computer science, an associative
Row C
array, map, or dictionary is an abstract
data type composed of a collection of Note: Column Families contain Columns with time
(key,value) pairs, such that each stamped versions. Columns only exist when inserted
(i.e. sparse)
possible key appears at most once in
the collection.”
Column A Column B
Row A
 Technically HBase is a Integer Value

multidimensional sorted map Column B


Long Timestamp1
Row B
Long Timestamp2
Column C
Huge URL
Row C
Family 1 Family 2

9 © 2013 IBM Corporation


Data Model : sparse, distributed, persistent, multi-
dimensional sorted map
Multidimensional map
Map Multidimensional Map : a map of maps
{ {
"xyz" : "Kingdom", "1" : { "A" : "ooo", "B" : "uuu" },
"zzz" : "lucky", "aaa" : { "A" : "UK", "B" : "Britain" },
"aab" : "hello", "aab" : { "A" : "hello", "B" : "world" },
"1" : "ooo", "xyz" : { "A" : "Kingdom", "B" : "Dynasty" },
"aaa" : "UK" "zzz" : { "A" : "lucky", "B" : "charm" }
} }
Persistent
In the context of storing data in a computer system, this means that the data survives
after the process with which it was created has ended. client1
{
"1" : "ooo", client10
Sorted "aaa" : "UK", client11
The key/value pairs are lexicographically sorted : "aab" : "hello", client2
…} client3
Sparse
Sparse! A given row can have any number of columns in each column family, or none at all.
Distributed
HBase is built upon Hadoop's Distributed File System (HDFS) so that the underlying file
storage can be spread among an array of independent machines.
10 © 2013 IBM Corporation
HBase Data Model – LOGICAL VIEW
 Table HBTABLE
– Contains column-families
Row key Value
 Column family 11111 cf_data:
– Logical and physical grouping of {‘cq_name’: ‘name1’,
columns ‘cq_val’: 1111}
cf_info:
 Column {‘cq_desc’: ‘desc11111’}
– Exists only when inserted
– Can have multiple versions 22222 cf_data:
– Each row can have different set of {‘cq_name’: ‘name2’,
columns ‘cq_val’: 2013 @ ts = 2013,
– Each column identified by it’s key ‘cq_val’: 2012 @ ts = 2012
}
 Row key
– Implicit primary key
– Used for storing ordered rows HFileHFile HFile
– Efficient queries using row key 11111 cf_data cq_name name1 @ ts1 11111 cf_info cq_desc desc11111 @ ts1
11111 cf_data cq_val 1111 @ ts1
22222 cf_data cq_name name2 @ ts1
22222 cf_data cq_val 2013 @ ts1
Each HFile is responsible for storing data 22222 cf_data cq_val 2012 @ ts2
corresponding to a particular column family.
11 © 2013 IBM Corporation
HBase versus the Traditional RDBMS
HBase RDBMS

A sparse, distributed, persistent


Data Layout multidimensional sorted map
Row or Column Oriented

ACID Support on Single Row


Transactions Only
Yes

get/put/scan only unless


Query Language combined with Hive or other SQL
technology

Security Authentication / Authorization Authentication / Authorization

Indexes Row-Key only or special table Yes

Throughput Millions of Queries per Sec Thousands of Queries per Sec

Maximum Database Size PBs TBs

13 © 2013 IBM Corporation


For Example …

Given this sample RDBMS table

SSN – Last Name First Account Type of Timestamp


primary Name Number Account
key
01234 Smith John abcd1234 Checking 20120118
01235 Johnson Michael wxyz1234 Checking 20120118
01235 Johnson Michael aabb1234 Checking 20111123
01236 Pruski null null null null

14 © 2013 IBM Corporation


HBase Logical View (“records”) Looks Like …

Row key Value (CF, Column, Version, Cell)


01234 info: {‘lastName’: ‘Smith’,
‘firstName’: ‘John’}
acct: {‘checking’: ‘abcd1234’}
01235 info: {‘lastName’: ‘Johnson’,
‘firstName’: ‘Michael’}
acct: {‘checking’: ‘wxyz1234’@ts=2012,
‘checking’: ‘aabb1234’@ts=2011}
01236 info: {‘lastName’: ‘Pruski’}

• Good for “sparse data” since non-exist column is just ignored


- no nulls

15 © 2013 IBM Corporation


While The Physical View (“cell”) Looks Like…
info Column Family
Row Key Column Key Timestamp Cell Value
01234 info:fname 1330843130 John
01234 info:lname 1330843130 Smith
01235 info:fname 1330843345 Michael
01235 info:lname 1330843345 Johnson
01236 info:lname Pruski

acct Column Family


Row Key Column Key Timestamp Cell Value

01234 acct:checking 1330843130 abcd1234


01235 acct:checking 1330843345 wxyz1234
01235 acct:checking 1330843239 aabb1234

Key

Key/Value Row Column Family Column Qualifier Timestamp Value


16 © 2013 IBM Corporation
More on the HBase Data Model
 There is no Schema for an HBase Table in the RDBMS sense
– Except that one has to declare the Column Families
• Since it determines the physical on-disk organization
– Thus every row can have a different set of Columns

 HBase is described as a key-value store

Key

Key/Value Row Column Family Column Qualifier Timestamp Value

 Each key-value pair is versioned


– Can be a timestamp or an integer
– Update a column is just to add a new version

 All data are byte arrays, including table name, Column Family names,
and Column names (also called Column Qualifiers)

17 © 2013 IBM Corporation


Indexes in HBase
 All table accesses are via the table row key -- its primary key.
 Rows are stored in byte-lexicographical order
 Within a row, columns are stored in sorted order
 This means it is fast, cheap and easy to scan adjacent rows &
columns
– Partial key lookups also possible : Supports Range based Row Scans
 HBase does not support indexes natively
– Instead, a Table can be created that serves the same purpose

Key

Key/Value Row Column Family Column Qualifier Timestamp Value

18 © 2013 IBM Corporation


HBase Cluster Architecture ZooKeeper is
used for
Client finds region coordination /
server addresses ZooKeeper Quorum monitoring
in ZooKeeper
(META table) ZooKeeper Peer Master
Client
ZooKeeper Peer …
Client reads and Hbase master
writes row by … assigns
accessing the regions and
region server load balancing

Region Server WAL Region Server WAL


Coprocessor Coprocessor … Coprocessor Coprocessor … …
… …
Region Region … Region Region …

HFile HFile HFile HFile

HFile HFile HDFS / GPFS HFile

HFile
HFile

20 © 2013 IBM Corporation


HBase Architecture Simplified

HBase
Region
A subset of a Tables’ Rows
Region Server(s) Automatically sharded when
grown too large
Master Region Server
Region(s)
Hosts tables, performs reads,
buffers writes (Memstore)
Clients talk directly to them for
reads/writes

ZooKeeper HDFS

21 © 2013 IBM Corporation


HBase Architecture Simplified
Master
• Responsible for coordinating the
Region Server(s)
HBase • Detects status of Region Server(s),
performs Region Server load
balancing
Region Server(s) • Manages cluster operations
Assigns regions to Region Servers,
Master Handles load balancing and Region splits
Region(s) Handles Cluster metadata (table/CF
creation/altering/removal, etc)
• Multiple Master Servers supported
Multiple Master Servers do not
cooperate, instead there is a primary and
one or more backup servers
• Not part of the read/write path
ZooKeeper HDFS • Highly available with ZooKeeper and
additional backup server(s)

22 © 2013 IBM Corporation


HBase Architecture Simplified

HBase Zookeeper
• Apache open source product and
part of the Hadoop ecosystem
Region Server(s) • Critical component for HBase
• Ensures one Master Server is
Master running
Region(s)
• Registers Region Servers
• Handles Region and Master Server
failures
• Integral part of HBase’s fault tolerant
architecture
• Keeps location of Hbase .META
table* (which RegionServer manage
ZooKeeper HDFS
it now)

* Hbase .META table contains metadata of all regions of all tables managed by cluster
23 © 2013 IBM Corporation
ZooKeeper
 Zookeeper provides coordination service

 Client finds region server via ZK

 Client writes/reads directly to/from the region server (ZK provides


meta table location)

 Master assigns regions and load balancing

 Region servers send heartbeats to the ZK

 Master monitors ZK for failed region servers

24 © 2013 IBM Corporation


Master
 Monitors all region server instances in the cluster

 Initializes region server failover

 Performs all metadata changes (e.g. create table)

 Manages region assignment

 Background services:
– LoadBalancer (move regions to balance the cluster load)
– CatalogJanitor (check and clean up the .META. Table)
– LogCleaner (clear the HLogs in the old logs directory)

25 © 2013 IBM Corporation


Backup Masters
 You can configure two or more masters

 When a master starts up, it races with other masters to write its
address into ZooKeeper. If it succeeds, it is the primary/active
master.

 If it does not succeed, there is another active master and it becomes


a backup master

 A backup master wait until the active master dies to try and become
the next active master

26 © 2013 IBM Corporation


Region Server

Region
Write-Ahead-Log

Store MemStore Region server : Makes a set


…… … of regions available to clients. It
StoreFile
… … checks in with the Master
HFile …

• WAL stores all the edits to the Store. There is one WAL per region server. All edits
for all regions carried by a particular region server are entered first in the WAL
• Region stores data for a certain region of a table. There are multiple stores for a
single region
• A Store holds a column family in a region. It has a memstore and a set of zero or
more Hfiles.
• MemStore is the write cache. Its main role is to store new data which has not yet
been written to disk. Memstore is flushed when data is transferred and saved in
Hfiles as blocks.
27 © 2013 IBM Corporation
Region Server

Region
Write-Ahead-Log

Store MemStore
…… …
StoreFile
… …
HFile …

Region
A region is a horizontal partition of a table with a start row and an end row
Regions are the basic element of availability and distribution for tables
A region is automatically split by the hosting region server when it reaches a
specified size
Periodically, a load balancer will move regions within the cluster to balance load
When a region server fails, its regions will be reassigned to other region servers

28 © 2013 IBM Corporation


Auto-Sharding and Scaling in HBase

 2 levels of data distribution management:


– Sharding of data tables into Hfiles automatic partionning
=> done by Hbase Region servers
– Management/distribution/replication of Hfiles
=> done by HDFS Hadoop
 Sharding is automatic in HBase : separation of records into horizontal
partitions spread across multiple storage servers
– The basic unit of scalability and load balancing in HBase is the Region
– Initially there is 1 Region per Table
– However, when the Table exceeds a configurable limit, it is automatically split in half –
this is called Auto-Sharding in Hbase : reported to Hmaster who can do the re-balancing
by affecting new region to a new Region server
– As the HBase data store grows, Tables consist of multiple Regions that are served by
Region Servers
– Region Servers may serve multiple Regions
 As the data store grows, the administrator may need to add more Region
Servers to the cluster to improve or maintain performance
 The new Region Servers will be assigned Regions by the Master Server
automatically

29 © 2013 IBM Corporation


How are Tables Stored in HBase?
The data is “sharded”
Each shard contains all the data in a key-range

Rows Region Server 1 Region Server 2 Region Server 3

A Keys:[A-D]
Region
. Keys:[S-V]
Region
Table – Logical View

.
Keys:[I-M]
K
Region
.
Keys:[E-H]
. Region
T Keys:[N-R]
. Region

. Keys:[W-Z]
Region
Z

Auto-Sharded Regions

30 © 2013 IBM Corporation


Hbase read mechanism

31 © 2013 IBM Corporation


HBase Write Mechanism

1. Write Ahead Log (WAL) is a file used to store new data that is yet to be put on permanent
storage. It is used for recovery in the case of failure. When a client issues a put request, it will
write the data to the write-ahead log (WAL).
32 © 2013 IBM Corporation
HBase Write Mechanism

2. MemStore is the write cache that stores new data that has not yet been written to disk.
There is one MemStore per column family per region. Once data is written to the WAL, it is
then copied to the MemStore.
33 © 2013 IBM Corporation
HBase Write Mechanism

3. Once the data is placed in MemStore, the client then receives the acknowledgment.

34 © 2013 IBM Corporation


HBase Write Mechanism

4. Hfiles store the rows of data as sorted KeyValue on disk. When the MemStore reaches the
threshold, it dumps or commits the data into an HFile.

35 © 2013 IBM Corporation


When Shouldn’t You Use HBase?
 HBase isn’t suitable for every problem

1. Do you have enough data to warrant HBase usage?


– e.g. Hundreds of millions or billions of rows => HBase
– e.g. Few thousand/million rows => traditional RDBMS

2. Do you need the extra features that an RDBMS provides?


– e.g., Typed columns, secondary indexes, transactions, advanced query
languages, etc.
– applications built against an RDBMS cannot be "ported" to HBase via different
driver

3. Do you have enough hardware?


– HBase doesn't do well with anything less than 5 DataNodes due to HDFS
block replication which has a default of 3, plus a NameNode.

 HBase can run fine on your laptop -- but why would you…?

37 © 2013 IBM Corporation


When to use HBase?
 Large amounts of data (*think Petabytes*)

 Need efficient random access inside large datasets


– Remember: Hadoop is designed to store and stream extremely large
datasets in batch.
• Not intended for realtime querying
• Doesn’t support random access
• Doesn’t handle billions of small files well

 Need to scale gracefully

 Relatively simple and fixed access pattern

 When the data model or schema is sparse (as opposed to fixed)

 When ACID properties are not required across rows or tables


– HBase provides strict row level atomicity - no further guarantee or
transactional feature that spans multiple rows or across tables is provided

38 © 2013 IBM Corporation


Questions?

39 © 2013 IBM Corporation

You might also like