Scribd
Upload
4K views

Hypertable An Open Source, High Performance, Scalable Database

This document introduces Hypertable, an open source scalable database modeled after Google's Bigtable. It provides a high-level overview of Hypertable's architecture, data model, scaling capabilities, and performance. Key points include that Hypertable uses a sparse multi-dimensional table structure, supports high insert/update rates, and scales horizontally across clusters through partitioning of data into ranges managed by range servers.

Uploaded by

Oleksiy Kovyrin
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
4K views

Hypertable An Open Source, High Performance, Scalable Database

This document introduces Hypertable, an open source scalable database modeled after Google's Bigtable. It provides a high-level overview of Hypertable's architecture, data model, scaling capabilities, and performance. Key points include that Hypertable uses a sparse multi-dimensional table structure, supports high insert/update rates, and scales horizontally across clusters through partitioning of data into ranges managed by range servers.

Uploaded by

Oleksiy Kovyrin
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 37

Hypertable

Doug Judd
Zvents, Inc.
Background

hypertable.org
Web 2.0 = Data Explosion

Web 1.0 Web 2.0

Web 2.0
Web 1.0
Traditional Tools
Don’t Scale Well
 Designed for a single machine
 Typical scaling solutions
 ad-hoc
 manual/static resource allocation
The Google Stack
 Google File System (GFS)
 Map-reduce
 Bigtable

hypertable.org
Architectural Overview

hypertable.org
What is Hypertable?
 A open source high performance, scalable
database, modelled after Google's Bigtable
 Not relational
 Does not support transactions

hypertable.org
Hypertable Improvements
Over Traditional RDBMS
 Scalable
 High random insert, update, and delete
rate
Data Model
 Sparse, two-dimensional table with cell versions
 Cells are identified by a 4-part key
 Row
 Column Family
 Column Qualifier
 Timestamp

hypertable.org
Table: Visual Representation

hypertable.org
Table: Actual Representation

hypertable.org
Anatomy of a Key
 Row key is \0 terminated
 Column Family is represented with 1 byte
 Column qualifier is \0 terminated
 Timestamp is stored big-endian ones-compliment

hypertable.org
Concurrency
 Bigtable uses copy-on-write
 Hypertable uses a form of MVCC
(multi-version concurrency control)
 Deletes are carried out by inserting “delete”
records
CellStore
 Sequence of 65K
blocks of compressed
key/value pairs
System Overview
Range Server
 Manages ranges of table data
 Caches updates in memory (CellCache)
 Periodically spills (compacts) cached updates to disk
(CellStore)

hypertable.org
Client API
class Client {

void create_table(const String &name,


const String &schema);

Table *open_table(const String &name);

String get_schema(const String &name);

void get_tables(vector<String> &tables);

void drop_table(const String &name,


bool if_exists);
};
Client API (cont.)
class Table {
TableMutator *create_mutator();
TableScanner *create_scanner(ScanSpec &scan_spec);
};
class TableMutator {
void set(KeySpec &key, const void *value, int value_len);
void set_delete(KeySpec &key);
void flush();
};
class TableScanner {
bool next(CellT &cell);
};
hypertable.org
Language Bindings
 Currently C++ only
 Thrift Broker
Write Ahead Commit Log
 Persists all modifications (inserts and
deletes)
 Written into underlying DFS
Range Meta-Operation Log
 Facilitates Range meta operation
 Loads
 Splits
 Moves
 Part of Master and RangeServer
 Ensures Range state and location
consistency
Compression
 Cell Stores store compressed blocks of key/value
pairs
 Commit Log stores compressed blocks of
updates
 Supported Compression Schemes
 zlib (--best and --fast)
 lzo
 quicklz
 bmz
 none

hypertable.org
Caching
 Block Cache
 Caches CellStore blocks
 Blocks are cached uncompressed
 Query Cache
 Caches query results
 TBD

hypertable.org
Bloom Filter
 Negative Cache
 Probabilistic data structure
 Indicates if key is not present
Scaling (part I)

hypertable.org
Scaling (part II)

hypertable.org
Scaling (part III)

hypertable.org
Access Groups
 Provides control of physical data layout --
hybrid row/column oriented
 Improves performance by minimizing I/O
CREATE TABLE crawldb {
Title MAX_VERSIONS=3,
Content MAX_VERSIONS=3,
PageRank MAX_VERSIONS=10,
ClickRank MAX_VERSIONS=10,
ACCESS GROUP default (Title, Content),
ACCESS GROUP ranking (PageRank, ClickRank)
};

hypertable.org
Filesystem Broker
Architecture
 Hypertable can run on top of any distributed
filesystem (e.g. Hadoop, KFS, etc.)

hypertable.org
Keys To Performance
 C++
 Asynchronous communication
C++ vs. Java
 Hypertable is CPU intensive
 Manages large in-memory key/value map
 Alternate compression codecs (e.g. BMZ)
 Hypertable is memory intensive
 Java uses 2-3 times the amount of memory to
manage large in-memory map (e.g. TreeMap)
 Poor processor cache performance

hypertable.org
Performance Test
(AOL Query Logs)
 75,274,825 inserted cells
 8 node cluster
 1 1.8 GHz Dual-core Opteron
 4 GB RAM
 3 x 7200 RPM SATA drives
 Average row key: 7 bytes
 Average value: 15 bytes
 Replication factor: 3
 4 simultaneous insert clients
 500K random inserts/s
 680K scanned cells/s
hypertable.org
Performance Test II
 Simulated AOL query log data
 1TB data
 9 node cluster
 1 2.33 GHz quad-core Intel
 16 GB RAM
 3 x 7200 RPM SATA drives
 Average row key: 9 bytes
 Average value: 18 bytes
 Replication factor: 3
 4 simultaneous insert clients
 Over 1M random inserts/s (sustained)
hypertable.org
Weaknesses
 Range data managed by a single range
server
 Though no data loss, can cause periods of
unavailability
 Can be mitigated with client-side cache or
memcached

hypertable.org
Project Status
 Currently in “alpha”
 Just released version 0.9.0.7
 Will release “beta” version end of August
 Waiting on Hadoop JIRA 1700

hypertable.org
License
 GPL 2.0
 Why not Apache?
Questions?
 www.hypertable.org

hypertable.org

You might also like