This document discusses dictionary encoding as a technique for compressing data in in-memory databases. Dictionary encoding replaces long text values with integer IDs, reducing memory usage. It provides examples showing how dictionary encoding can compress a first name column by a factor of 17 and a gender column by a factor of 8. Sorted dictionaries can provide faster lookups but updating the dictionary is more expensive. Operations on encoded data are performed using integers, which is faster than other data types.
This document discusses dictionary encoding as a technique for compressing data in in-memory databases. Dictionary encoding replaces long text values with integer IDs, reducing memory usage. It provides examples showing how dictionary encoding can compress a first name column by a factor of 17 and a gender column by a factor of 8. Sorted dictionaries can provide faster lookups but updating the dictionary is more expensive. Operations on encoded data are performed using integers, which is faster than other data types.
This document discusses dictionary encoding as a technique for compressing data in in-memory databases. Dictionary encoding replaces long text values with integer IDs, reducing memory usage. It provides examples showing how dictionary encoding can compress a first name column by a factor of 17 and a gender column by a factor of 8. Sorted dictionaries can provide faster lookups but updating the dictionary is more expensive. Operations on encoded data are performed using integers, which is faster than other data types.
This document discusses dictionary encoding as a technique for compressing data in in-memory databases. Dictionary encoding replaces long text values with integer IDs, reducing memory usage. It provides examples showing how dictionary encoding can compress a first name column by a factor of 17 and a gender column by a factor of 8. Sorted dictionaries can provide faster lookups but updating the dictionary is more expensive. Operations on encoded data are performed using integers, which is faster than other data types.
A Course in In-Memory Data Management The Inner Mechanics of In-Memory Databases
August 30, 2013
This learning material is part of the reading material for Prof.
Plattner’s online lecture "In-Memory Data Management" taking place at www.openHPI.de. If you have any questions or remarks regarding the online lecture or the reading material, please give us a note at openhpi- [email protected]. We are glad to further improve the material. Chapter 6 Dictionary Encoding
Since memory is the new bottleneck, it is required to minimize access to it.
Accessing a smaller number of columns can do this on the one hand; so only required attributes are queried. On the other hand, decreasing the number of bits used for data representation can reduce both memory consumption and transfer times. Dictionary encoding builds the basis for several other compression tech- niques (see Chapter 7) that might be applied on top of the encoded columns. The main e↵ect of dictionary encoding is that long values, such as texts, are represented as short integer values. Dictionary encoding is relatively simple. This means not only that it is easy to understand, but also it is easy to implement and does not have to rely on complex multilevel procedures, which would limit or lessen the performance gains. First, we will explain the general algorithm how original values are translated to integers using the example presented in Figure 6.1.
Dictionary encoding is applied on each column of a table separately. In the
example, every distinct value in the first name column “fname” is replaced by a distinct integer value. The position of a text value (e.g. Mary) in the
37 38 6 Dictionary Encoding
dictionary is the representing number for that text (here: “24” for Mary). Until now, we have not saved any storage space. The benefits come to e↵ect with values appearing more than once in a column. In our tiny example, the value “John” can be found twice in the column “fname”, namely on position 39 and 42. Using dictionary encoding, the long text value (we assume 49 Byte per entry in the first name column) is represented by the short integer value (23 bit are needed to encode the 5 million di↵erent first names we assume to exist in the world). The more often identical values appear, the better dictionary encoding can compress a column. As we noted in Section 3.6, enterprise data has low entropy. Therefore, dictionary encoding is well suited and yields a good compression ratio. We will exemplify this on the first name and gender columns of our world-population example.
6.1 Compression Example
Given is the world population table with 8 billion rows, 200 Byte per row:
Attribute # of Distinct Values Size
first name 5 million 49 Byte last name 8 million 50 Byte gender 2 1 Byte country 200 49 Byte city 1 million 49 Byte birthday 40 000 2 Byte Sum 200 Byte
The complete amount of data is:
8 billion rows · 200 Byte per row = 1.6 TB
Each column is split into a dictionary and an attribute vector. Each dic- tionary stores all distinct values of the column. The valueID of each value is implicitly given by the value’s position in the dictionary and do thus not need to be stored explicitly. In a dictionary-encoded column, the attribute vectors now only store valueIDs, which correspond to the valueIDs in the dictionary. The recordID (row number) is stored implicitly via the position of an entry in the attribute vector. To sum up, via dictionary encoding, all information can be stored as integers instead of other, usually larger, data types. 6.1 Compression Example 39
Dictionary Encoding Example: First Names
How many bits are required to represent all 5 million distinct values of the first name column “fname”?
dlog2 (5, 000, 000)e = 23
Therefore, 23 bits are enough to represent all distinct values for the required column. Instead of using
8 billion · 49 Byte = 392 billion Byte = 365.1 GB
for the first name column, the attribute vector itself can be reduced to the size of
8 billion · 23 bit = 184 billion bit = 23 billion Byte = 21.4 GB
and an additional dictionary is introduced, which needs
49 Byte · 5 million = 245 million Byte = 0.23 GB.
The achieved compression factor can be calculated as follows:
uncompressed size 365.1 GB
= ⇡ 17 compressed size 21.4 GB + 0.23 GB
That means we reduced the column size by a factor of 17 and the result only consumes about 6 % of the initial amount of main memory.
Dictionary Encoding Example: Gender
Let us look on another example with the gender column. It has only 2 distinct values. For the gender representation without compression for each value (“m” or “f”) 1 Byte is required. So, the amount of data without compression is: 8 billion · 1 Byte = 7.45 GB If compression is used, then 1 bit is enough to represent the same information. The attribute vector takes:
8 billion · 1bit = 8 billion bit = 0.93 GB
The dictionary needs additionally:
2 · 1 Byte = 2 Byte
This concludes to a compression factor of:
40 6 Dictionary Encoding
uncompressed size 7.45GB
= ⇡8 compressed size 0.93 GB + 2 Byte
The compression rate depends on the size of the initial data type as well as on the column’s entropy, which is determined by two cardinalities: • Column cardinality, which is defined as the number of distinct values in a column, and • Table cardinality, which is the total number of rows in the table or column Entropy is a measure which shows how much information is contained in a column. It is calculated as column cardinality entropy = table cardinality
The smaller the entropy of the column, the better the achievable compression rate.
6.2 Sorted Dictionaries
The benefits of dictionary encoding can be further enhanced if sorting is
applied to the dictionary. Retrieving a value from a sorted dictionary speeds up the lookup process from O(n), which means a full scan through the dic- tionary, to O(log(n)), because values in the dictionary can be found using binary search. Sadly, this optimization comes at a cost: Every time a new value is added to the dictionary, which does not belong at the end of the sorted sequence, the dictionary has to be re-sorted. Consequently, the posi- tions of already present values behind the inserted value have to be moved one position up. While sorting the dictionary is not that costly, updating the corresponding attribute vector is. In our example, about 8 billion values have to be checked or updated if a new first name is added to the dictionary.
6.3 Operations on Encoded Values
The first and most important e↵ect of dictionary encoding is that all oper- ations concerning the table data are now done via attribute vectors, which solely consist of integers. This causes an implicit speedup of all operations, since a CPU is designed to perform operations on numbers, not on charac- ters. When explaining dictionary encoding, a question often asked is: “But isn’t the process of looking up all values via an additional data structure more costly than the actual savings? We understand the benefits concerning main memory, but what about the processor” – First, it has to be stated that 6.3 Operations on Encoded Values 41
the question is deemed appropriate. The processor has to take additional
load, but this is acceptable, given the fact that our bottleneck is memory and bandwidth, so a slight shift of pressure in the direction of the processor is not only accepted but also welcome. Second, the impact of retrieving the ac- tual values for the encoded columns is rather small. When selecting tuples, only the corresponding values from the query have to be looked up in the dictionary for the column scan. Generally, the result set is small compared to the total table size, so the lookup of all other selected columns to materialize the query result is not that expensive. Carefully written queries also only se- lect those columns that are really needed, which not only saves bandwidth but also further reduces the number of necessary lookups. Finally, several operations such as COUNT or NOT NULL can even be performed without retrieving the real values at all.
