Index Compression
Lecture 4
Why compression (in general)?
• Use less disk space
– Save a little money; give users more space
• Keep more stuff in memory
– Increases speed
• Increase speed of data transfer from disk to
memory
– [read compressed data | decompress] is faster than
[read uncompressed data]
• Premise: Decompression algorithms are fast
– True of the decompression algorithms we use
2
Postings compression
• A posting for our purposes is a docID.
• For Reuters (800,000 documents), we would
use 32 bits per docID when using 4-byte
integers.
• Alternatively, we can use log2 800,000 ≈ 20
bits per docID.
• Our goal: use far fewer than 20 bits per docID.
3
Compression Example
• Fixed Length encoding –clear how to decode
since number of bits is fixed
Number Fixed Length
Code (5 bit)
1 00001
01010011101011001101
2 00010
15 01111
17 10001
01010, 01110, 10110, 01101
25 11001
31 11111
10, 14, 22, 13
4
Fixed Length Encoding
• How many total bits are required for encoding
for 10 million numbers using 20 bits for each
number?
• 200 million bits = 2*108 bits
• What if most numbers are small ?
• We are wasting many bits by storing small
numbers in 20 bits.
5
Variable Length Encoding
• Decode 10110011
• 1011, 0,0,11 = 11, 0,0,3 Number Variable Length
code
1 1
• 10, 1100, 11 = 2, 12, 3 2 10
10 1010
16 10000
• 101,100, 11 = 5, 4, 3
• 10,11,0,0,11 = 2, 3, 0, 0, 3
•
6
Compression Example
Decode 0101011101100
• use unambiguous code:
Prefix free code
0, 1, 0, 3, 0, 2, 0
• which gives:
7
Postings: two conflicting forces
• A term like arachnocentric occurs in maybe
one doc out of a million – we would like to
store this posting using log2 1M ≈ 20 bits.
• A term like the occurs in virtually every doc, so
20 bits/posting ≈ 2MB is too expensive.
8
Gap encoding of postings file entries
• We store the list of docs containing a term in
increasing order of docID.
– computer: 33,47,154,159,202 …
• Consequence: it suffices to store gaps.
– 33,14,107,5,43 …
• Hope: most gaps can be encoded/stored with
far fewer than 20 bits.
– Especially for common words
9
Three postings entries
10
Index compression
• Observation of posting files
– Instead of storing docID in posting, we store gap
between docIDs, since they are ordered
– Zipf’s law again:
• The more frequent a word is, the smaller the gaps are
• The less frequent a word is, the shorter the posting list
is
– Heavily biased distribution gives us great
opportunity of compression!
Information theory: entropy measures compression difficulty.
11
Delta Encoding
• Word count data is good candidate for
compression
– many small numbers and few larger numbers
– encode small numbers with small codes
• Document numbers are less predictable
– but differences between numbers in an ordered
list are smaller and more predictable
• Delta encoding:
– encoding differences between document numbers
(d-gaps)
12
Delta Encoding
– Inverted list (without counts)
– Differences between adjacent numbers
– Differences for a high-frequency word are easier to
compress, e.g.,
– Differences for a low-frequency word are large, e.g.,
13
Practice Question
• Delta encode following numbers:
• 40, 45, 405, 411, 416
• 40, 5, 360, 6, 5
• Decode following numbers encoded using
delta encoding
• 20, 10, 30, 4, 8
• 20, 30, 60, 64, 72
14
Variable length encoding
• Aim:
– For arachnocentric, we will use ~20 bits/gap entry.
– For the, we will use ~1 bit/gap entry.
• If the average gap for a term is G, we want to use
~log2G bits/gap entry.
• Key challenge: encode every integer (gap) with
about as few bits as needed for that integer.
• This requires a variable length encoding
• Variable length codes achieve this by using short
codes for small numbers
15
Unary Codes
• Breaks between encoded numbers can occur
after any bit position
• Unary code
– Encode k by k 1s followed by 0
– 0 at end makes code unambiguous
1110111010110
1110, 1110, 10, 110
3, 3, 1, 2
16
Unary Codes
• Unary is very efficient for small numbers such
as 0 and 1, but quickly becomes very
expensive
– 1023 can be represented in 10 binary bits, but
requires 1024 bits in unary
17
Binary Codes
• Variable Length Binary Codes:
– Variable length Binary is more efficient for large
numbers, but it may be ambiguous
– E.g. 2 encoded as 10 and 5 encoded as 101
– 10101, how to decode, we don’t know the word
bound
• Fixed Length Binary Codes:
– For example use 15 bits to encode each number
– Cannot encode large numbers that required more
than 15 bits
– Too much space is wasted for encoding small numbers
18
Elias-γ Code (Elias Gamma Code)
• Encode number in minimum bits in binary
• 12,
• 1100 , 4 bits
• Encode length of the code in unary in begining
• Code of 12 = 111101100
• 23
• 10111 , 5 bits
• Code of 23 = 11111010111
19
Elias-γ Code (Elias Gamma Code)
• Decode 1110 101 110 10
• 1110101 , 11010
• 5 , 2
20
Elias-γ Code (Elias Gamma Code)
• Encode number in minimum bits in binary
• 12,
• 1100 , 4 bits
• 100, leave the leftmost 1 bit, length 3 bits
• Code of 12 = 1110100
• 23
• 10111 , 5 bits
• 0111, 4 bits, leave the leftmost 1 bit
• Code of 23 = 111100111
21
Elias-γ Code (Elias Gamma Code)
• Decode 111001111000
• 1110 011, 110 00
• 011 was actually 1011 = 11
• 00 was actually 100 = 4
• 11, 4
22
Elias-γ Code (Elias Gamma Code)
• To encode a number k, compute
– Since the leftmost bit is always 1 in binary code so we
do not encode it
– The remaining number becomes kr = k – 2 lgk
– We use Unary code for kd and binary code for kr
23
Elias-γ Code (Elias Gamma Code)
• To encode a number k, compute
Unary code for kd and binary code for kr
24
Gamma seldom used in practice
• Machines have word boundaries – 8, 16, 32, 64
bits
– Operations that cross word boundaries are slower
• Compressing and manipulating at the granularity
of bits can be too slow
• All modern practice is to use byte or word aligned
codes
– Variable byte encoding is a faster, conceptually
simpler compression scheme, with decent
compression
25
Variable Byte (VB) codes
• For a gap value G, we want to use close to the
fewest bytes needed to hold log2 G bits
• Begin with one byte to store G and dedicate 1 bit
in it to be a continuation bit c
• If G ≤127, binary-encode it in the 7 available bits
and set c =1
• Else encode G’s lower-order 7 bits and then use
additional bytes to encode the higher order bits
using the same algorithm
• At the end set the continuation bit of the last
byte to 1 (c =1) – and for the other bytes c = 0.
26
Variable Byte Code
Consider the vByte representation of the postings list L = ⟨80, 400, 431, 686⟩:
Δ(L) = ⟨80, 320, 31, 255⟩
11010000 01000000 10000010 10011111
vByte(L) =
01111111 10000001
27
Example
docIDs 824 829 215406
gaps 5 214577
VB code 00000110 10000101 00001101
10111000 00001100
10110001
28
RCV1 compression
Data structure Size in MB
collection (text, xml markup etc) 3,600.0
collection (text) 960.0
Term-doc incidence matrix 40,000.0
postings, uncompressed (32-bit words) 400.0
postings, uncompressed (20 bits) 250.0
postings, variable byte encoded 116.0
postings, g-encoded 101.0
Index and dictionary compression for Reuters-RCV1.
(Manning et al. Introduction to Information Retrieval)
29
Group Variable Integer code
• Used by Google around turn of millennium….
– Jeff Dean, keynote at WSDM 2009
– Encodes 4 integers in blocks of size 5–17 bytes
• First byte: four 2-bit binary length fields
• L1 L2 L3 L4 , Lj{1,2,3,4}
• Then, L1+L2+L3+L4 bytes (between 4–16) hold 4 numbers
– Each number can use 8/16/24/32 bits. Max gap length ~4 billion
• It was suggested that this was about twice as fast as VB
encoding
– Decoding gaps is much simpler – no bit masking
– First byte can be decoded with lookup table or switch
30
Group Variable Integer code
Consider the vByte representation of the postings list L = ⟨80, 400, 431, 686⟩:
Δ(L) = ⟨80, 320, 31, 255⟩
11010000 01000000 10000010 10011111 01111111
vByte(L) =
10000001
For the same postings list as before, the Group VarInt representation is:
GroupVarInt(L) = 00010000 01010000 01000000 00000001 00011111 11111111
31
Compression Techniques Effectiveness
(All Data from Gov2 Collection)
Decoding (ns per position) Cumulative Overhead (decoding +
disk I/O)
Gamma 12.81 32.11 ns
vByte 4.34 20.82 ns
Group VarInt 1.9 19.85 ns
Chapter 6, Information Retrieval: Implementing and Evaluating
Search Engines, by S. Büttcher, C. Clarke, and G. Cormack. 32
