Indexing

Index Construction

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Motivation: Scale
Corpus Terms Docs Entries • A term incidence matrix with V
terms and D documents has O(V x
D) entries.
Shakespeare’s ~1.1
~31,000 37
Plays million
• Shakespeare used around 31,000
distinct words across 37 plays, for
about 1.1M entries.
English ~1.7 ~4.5 ~7.65
Wikipedia million million trillion • As of 2014, a collection of Wikipedia
pages comprises about 4.5M pages
and roughly 1.7M distinct words.
English Web >2 million
>1.7
>3.4x10 Assuming just one bit per matrix
billion entry, this would consume about
890GB of memory.
Inverted Indexes - Intro
• Two insights allow us to reduce this to a
manageable size:

1. The matrix is sparse – any document

uses a tiny fraction of the vocabulary.

2. A query only uses a handful of

words, so we don’t need the rest.

• We use an inverted index instead of

using a term incidence matrix directly.

• An inverted index is a map from a term

to a posting list of documents which
use that term.
Search Algorithm
• Consider queries of the form:

t1 AND t2 AND … AND tn

• In this simplified case, we need only

take the intersections of the term
posting lists.

• This algorithm, inspired by merge sort,

relies on the posting lists being sorted
by length.

• We save time by processing the terms

in order from least common to most
common. (Why does this help?)
 Motivation
• All modern search engines rely on inverted indexes in some form.
Many other data structures were considered, but none has matched
its efficiency.

• The entries in a production inverted index typically contain many

more fields providing extra information about the documents.

• The efficient construction and use of inverted indexes is a topic of its

own, and will be covered in a later module.
 Motivation
A reasonably-sized index of the web contains many billions of
documents and has a massive vocabulary.

Search engines run roughly 5

10 queries per second over that collection.

We need fine-tuned data structures and algorithms to provide search

results in much less than a second per query. O(n) and even O(log n)
algorithms are often not nearly fast enough.

The solution to this challenge is to run an inverted index on a massive

distributed system.
 Inverted Indexes
Inverted Indexes are primarily used to allow fast, concurrent query
processing.

Each term found in any indexed document receives an independent

inverted list, which stores the information necessary to process that
term when it occurs in a query.

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Indexes
The primary purpose of a search engine
index is to store whatever information is
needed to minimize processing at query time.

Text search has unique needs compared to,

e.g., database queries, and needs its own
data structures – primarily, the inverted index.

• A forward index is a map from documents

to terms (and positions). These are used
when you search within a document.

• An inverted index is a map from terms to

documents (and positions). These are used
when you want to find a term in any
document. Is this a forward or an inverted index?
 Abstract Model of Ranking
Document Topical Features!
Indexes are created to support 9.7 fish
4.2 tropical
search, and the primary search task is 22.1 tropical fish
document ranking. 8.2 seaweed
4.2 surfboards

We sort documents according to some Quality Features!

14 incoming links
scoring function which depends on 3 days since last update
the terms in the query and the
document representation.
Query!
In the abstract, we need to store tropical fish Scoring Function
various document features to
efficiently score documents in
response to a query. Document Score!
24.5
More Concrete Model
 Inverted Lists
In an inverted index, each term has an
associated inverted list.
inverted list

At minimum, this list contains a list of

identifiers for documents which posting
contain that term.

Usually we have more detailed

information for each document as it
relates to that term. Each entry in an
inverted list is called a posting.

Simple Inverted Index

 Inverted Index with Counts
Document postings can store any
information needed for efficient
ranking.

For instance, they typically store term

counts for each document – tfw,d.

Depending on the underlying storage

system, it can be expensive to
increase the size of a posting. It’s
important to be able to efficiently scan
through an inverted list, and it helps if
they’re small.
Inverted Index with Counts
 Indexing Additional Data
The information used to support all modern search features can grow quite complex.

Locations, dates, usernames, and other metadata are common search criteria,
especially in search functions of web and mobile applications.

When these fields contain text, they are ultimately stored using the same inverted list
structure.

Next, we’ll see how to compress inverted lists to reduce storage needs and
filesystem I/O.

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Indexing Term Positions
Many scoring functions assign higher
scores to documents containing the
query terms in closer proximity.

Some query languages allow users to

specify proximity requirements, like
“tropical NEAR fish.”

In the inverted lists to the right, the

word “to” has a DF of 993,427. It is
found in five documents; its TF in doc
1 is 6, and the list of positions is given. Postings with DF, TF, and Positions
 Proximity Searching
In proximity search, you search for
documents where terms are
sufficiently close to each other.

We process terms from least to most

common in order to minimize the
number of documents processed.

The algorithm shown here finds

documents from two inverted lists
where the terms are within k words of
each other.

Algorithm for Proximity Search

 Indexing Scores
For some search applications, it’s worth storing the document’s matching score
for a term in the posting list.

Postings may be sorted from largest to smallest score, in order to quickly find
the most relevant documents. This is especially useful when you want to quickly
find the approximate-best documents rather than the exact-best.

Indexing scores makes queries much faster, but gives less flexibility in updating
your retrieval function. It is particularly efficient for single term queries.

For Machine Learning based retrieval, it’s common to store per-term scores
such as BM25 as features.
 Fields and Extents
Some indexes have distinct fields with their own inverted lists. For instance,
an index of e-mails may contain fields for common e-mail headers (from,
subject, date, …).

Others store document regions such as the title or headers using extent lists.

Extent lists are contiguous regions of a document stored using term

positions.

extent list
 Index Schemas
As the information stored in an inverted
index grows more complex, it becomes
useful to represent it using some form of
schema.

However, we normally don’t use strict

SQL-type schemas, partly due to the
cost of rebuilding a massive index.
Instead, flexible formats such as <key,
value> maps with field names
arranged by convention are used.

Each text field in the schema typically

Partial JSON Schema for Tweets
gets its own inverted lists.
 Index Construction
We have just scratched the surface of the complexities of constructing and updating large-scale
indexes. The most complex indexes are massive engineering projects that are constantly being
improved.

An indexing algorithm needs to address hardware limitations (e.g., memory usage), OS limitations
(the maximum number of files the filesystem can efficiently handle), and algorithmic concerns.

When considering whether your algorithm is sufficient, consider how it would perform on a
document collection a few orders of magnitude larger than it was designed for.

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Basic Indexing
Given a collection of documents, how can
we efficiently create an inverted index of
its contents?
Basic In-Memory Indexer
The basic steps are:

1. Tokenize each document, to convert it

to a sequence of terms

2. Add doc to inverted list for each token

This is simple at small scale and in

memory, but grows much more complex
to do efficiently as the document
collection and vocabulary grow.
 Merging Lists
The basic indexing algorithm will fail as soon as you run out of
memory.

To address this, we store a partial inverted list to disk when it grows

too large to handle. We reset the in-memory index and start over.
When we’re finished, we merge all the partial indexes.

The partial indexes should be written in a manner that facilitates later

merging. For instance, store the terms in some reasonable sorted
order. This permits merging with a single linear pass through all partial
lists.
Merging Example
 Result Merging
An index can be updated from a new batch of documents by merging the posting
lists from the new documents. However, this is inefficient for small updates.

Instead, we can run a search against both old and new indexes and merge the
result lists at search time. Once enough changes have accumulated, we can
merge the old and new indexes in a large batch.

In order to handle deleted documents, we also need to maintain a delete list of

docids to ignore from the old index. At search time, we simply ignore postings
from the old index for any docid in the delete list.

If a document is modified, we place its docid into the delete list and place the new
version in the new index.
 Updating Indexes
If each term’s inverted list is stored in a separate file, updating the index
is straightforward: we simply merge the postings from the old and new
index.

However, most filesystems can’t handle very large numbers of files, so

several inverted lists are generally stored together in larger files. This
complicates merging, especially if the index is still being used for query
processing.

There are ways to update live indexes efficiently, but it’s often simpler to
simply write a new index, then redirect queries to the new index and
delete the old one.
 Compressing Indexes
The best any compression scheme can do depends on the entropy of the
probability distribution over the data. More random data is less compressible.

Huffman Codes meet the entropy limit and can be built in linear time, so are a
common choice. Other schemes can do better, generally by interpreting the
input sequence differently (e.g. encoding sequences of characters as if they
were a single input symbol – different distribution, different entropy limit).

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Index Size
Inverted lists often consume a large amount of space.

• e.g., 25-50% of the size of the raw documents for TREC collections
with the Indri search engine

• much more than the raw documents if n-grams are indexed

Compressing indexes is important to conserve disk and/or RAM

space. Inverted lists have to be decompressed to read them, but there
are fast, lossless compression algorithms with good compression
ratios.
 restricted variable length codes
• an extension of multicase encodings (“shift key”) where different code lengths are
used for each case. Only a few code lengths are chosen, to simplify encoding and
decoding.

• Use first bit to indicate case.

• 8 most frequent characters fit in 4 bits (0xxx).

• 128 less frequent characters fit in 8 bits (1xxxxxxx)

• In English, 7 most frequent characters are 65% of occurrences

• Expected code length is approximately 5.4 bits per character, for a 32.8%
compression ratio.

• average code length on WSJ89 is 5.8 bits per character, for a 27.9% compression ratio
restricted variable length codes: more symbols
• Use more than 2 cases.
3
• 1xxx for 2 = 8 most frequent symbols, and
6
• 0xxx1xxx for next 2 = 64 symbols, and
9
• 0xxx0xxx1xxx for next 2 = 512 symbols, and

• ...

• average code length on WSJ89 is 6.2 bits per

symbol, for a 23.0% compression ratio.

• Pro: Variable number of symbols.

• Con: Only 72 symbols in 1 byte.

restricted variable length codes : numeric data
• 1xxxxxxx for 2 7 = 128 most frequent symbols

• 0xxxxxxx1xxxxxxx for next 2 14 = 16,384 symbols

• ...

• average code length on WSJ89 is 8.0 bits per symbol, for a 0.0%
compression ratio (!!).

• Pro: Can be used for integer data

• Examples: word frequencies, inverted lists

 restricted variable –length codes : word based
encoding
• Restricted Variable-Length Codes can be used on words (as opposed to symbols)

• build a dictionary, sorted by word frequency, most frequent words first

• Represent each word as an offset/index into the dictionary

• Pro: a vocabulary of 20,000-50,000 words with a Zipf distribution requires 12-13

bits per word

• compared with a 10-11 bits for completely variable length

• Con: The decoding dictionary is large, compared with other methods.

 restricted variable-length codes: summary

• Four methods presented. all are

• simple

• very effective when their assumptions are correct

• No assumptions about language or language models

• all require an unspecified mapping from symbols to numbers (a

dictionary)

• all but the basic method can handle any size dictionary
 Entropy and Compressibility
The entropy of a probability
distribution is a measure of its
randomness.

!
( )= log

The more random a sequence of data

is, the less predictable and less
compressible it is.

The entropy of the probability

distribution of a data sequence
provides a bound on the best possible
compression ratio.
Entropy of a Binomial Distribution
 Huffman Codes
In an ideal encoding scheme, a symbol Symbol p Code 𝔼[length]
with probability pi of occurring will be
assigned a code which takes log(pi) bits. a 1/2 0 0.5

b 1/4 10 0.5
The more probable a symbol is to occur,
the smaller its code should be. By this c 1/8 110 0.375
view, UTF-32 assumes a uniform
distribution over all unicode symbols; d 1/16 1110 0.25
UTF-8 assumes ASCII characters are more
common. e 1/16 1111 0.25

Huffman Codes achieve the best possible Plaintext:!! aedbbaae (64 bits in UTF-8)
compression ratio when the distribution is Ciphertext:! 0111111101010001111
known and when no code can stand for
multiple symbols.
 Building Huffman Codes
Huffman Codes are built using a binary tree
which always joins the least probable 1
remaining nodes. 1

1. Create a leaf node for each symbol, 1/2

weighted by its probability. 1
0
2. Iteratively join the two least probable 1/4
nodes without a parent by creating a 0 1
parent whose weight is the sum of the
childrens’ weights. 0
1/8

3. Assign 0 and 1 to the edges from each 0 1

parent. The code for a leaf is the
sequence of edges on the path from the a: 1/2 b: 1/4 c: 1/8 d: 1/16 e: 1/16
root.
0 10 110 1110 1111
 Can We Do Better?
Huffman codes achieve the theoretical limit for compressibility, assuming
that the size of the code table is negligible and that each input symbol
must correspond to exactly one output symbol.

Other codes, such as Lempel-Ziv encoding, allow variable-length

sequences of input symbols to correspond to particular output symbols
and do not require transferring an explicit code table.

Compression schemes such as gzip are based on Lempel-Ziv encoding.

However, for encoding inverted lists it can be beneficial to have a 1:1
correspondence between code words and plaintext characters.
 Lempel-Ziv
• an adaptive dictionary approach to variable length coding.

• Use the text already encountered to build the dictionary.

• If text follows Zipf's laws, a good dictionary is built.

• No need to store dictionary; encoder and decoder each know how to build it on the fly.

• Some variants: LZ77, Gzip, LZ78, LZW, Unix compress

• Variants differ on:

• how dictionary is built,

• how pointers are represented (encoded), and

• limitations on what pointers can refer to.

 Lempel Ziv: encoding
• 0010111010010111011011
 Lempel Ziv: encoding
• 0010111010010111011011

• break into known prefixes

• 0|01 |011|1 |010|0101|11|0110|11

 Lempel Ziv: encoding
• 0010111010010111011011

• break into known prefixes

• 0|01 |011|1 |010|0101|11|0110|11

• encode references as pointers

• 0|1,1|1,1 |0,1|3,0 |1,1 |3,1|5,0 |2,?

 Lempel Ziv: encoding
• 0010111010010111011011
!

• break into known prefixes

• 0|01 |011|1 |010|0101|11|0110|11

• encode references as pointers

• 0|1,1|1,1 |0,1|3,0 |1,1 |3,1|5,0 |2,?

• encode the pointers with log(?)bits

• 0|1,1|01,1 |00,1|011,0 |001,1 |011,1|101,0 |0010,?

 Lempel Ziv: encoding
• 0010111010010111011011

• break into known prefixes: 0|01 |011|1 |010|0101|11|0110|11

• encode references as pointers : 0|1,1|1,1 |0,1|3,0 |1,1 |3,1|5,0 |2,?

• encode the pointers with log(?)bits :

0|1,1|01,1 |00,1|011,0 |001,1 |011,1|101,0 |0010,?

• final string : 01101100101100011011110100010

 Lempel Ziv: decoding
• 01101100101100011011110100010
 Lempel Ziv: decoding
• 01101100101100011011110100010

• decode the pointers with log(?)bits

• 0|1,1|01,1 |00,1|011,0 |001,1 |011,1|101,0 |0010,?

 Lempel Ziv: decoding
• 01101100101100011011110100010

• decode the pointers with log(?)bits

• 0|1,1|01,1 |00,1|011,0 |001,1 |011,1|101,0 |0010,?

• encode references as pointers

• 0|1,1|1,1 |0,1|3,0 |1,1 |3,1|5,0 |2,?

 Lempel Ziv: decoding
• 01101100101100011011110100010

• decode the pointers with log(?)bits

• 0|1,1|01,1 |00,1|011,0 |001,1 |011,1|101,0 |0010,?

• encode references as pointers

• 0|1,1|1,1 |0,1|3,0 |1,1 |3,1|5,0 |2,?

• decode references

• 0|01 |011|1 |010|0101|11|0110|11

 Lempel Ziv: decoding
• 01101100101100011011110100010!

• decode the pointers with log(?) bits : 0|1,1|01,1 |00,1|011,0 |001,1 |011,1|101,0 |0010,?!

• encode references as pointers : 0|1,1|1,1 |0,1|3,0 |1,1 |3,1|5,0 |2,?!

• decode references : 0|01 |011|1 |010|0101|11|0110|11!

• original string : 0010111010010111011011

 Lempel Ziv optimality
• LempelZiv compression rate approaches (asymptotic) entropy

• When the strings are generated by an ergodic source

[CoverThomas91].

• easier proof : for i.i.d sources

• that is not a good model for English

 • !"# x $ !%!&...!n ' (")*"+," -. !"+/#0 + /"+1
LempelZiv optimality "2'#"3 45 ' 663 (-*2," '+3 789: $ #0" ;2-4'1
46!6#5 #- ("" (*,0 ' (")*"+,"
–i.i.d source
• ('5 <"=;"!>6? 42"'@( 6+#- c ;02'("( x $
y%y&...yc '+3 ,'!! cl $ A -. ;02'("( -. !"+/#0 l
P
#0"+ ! !-/ Q8x: " cl !-/ cl
l
P " % c
8;2--.: Q8yi : < % (- Q8yi: < 8 c : l
|yi |$l |yi|$l l

• 6. pi 6( #0" (-*2," ;2-4'4 .-2 !i #0"+ 45 !'B

-. !'2/" +*=4"2( x B6!! 0'?" 2-*/0!5 npi -,,*21
2"+,"( -. !i '+3 #0"+
" npi P
logQ8x: $ ! !-/ pi # n pi !-/ pi $ nHsource
i

P
• +-#" #0'# cl !-/ cl 6( 2-*/0!5 #0" <"=;"!>6?
l
"+,-36+/ !"+/#0 (- #0 "6+")*'!6#5 2"'3(
nH "# LZencoding B06,0 6( #- ('5 H #" LZrateC
 Bit-aligned Codes
Bit-aligned codes allow us to minimize the storage used to encode integers.

We can use just a few bits for small integers, and still represent arbitrarily large
numbers.

Inverted lists can also be made more compressible by delta-encoding their

contents.

Next, we’ll see how to encode integers using a variable byte code, which is
more convenient for processing.

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Compressing Inverted Lists
An inverted list is generally represented
as multiple sequences of integers.

• Term and document IDs are used

instead of the literal term or
document URL/path/name.

• TF, DF, term position lists and other

data in the inverted lists are often
integers.

We’d like to efficiently encode this

integer data to help minimize disk and Postings with DF, TF, and Positions
memory usage. But how?
 Unary
The encodings used by processors for
integers (e.g., two’s complement) use a
fixed-width encoding with fixed upper
bounds. Any number takes 32 (say) bits, decimal binary unary
with no ability to encode larger numbers.
0 00000000 0
Both properties are bad for inverted lists.
Smaller numbers tend to be much more 1 00000001 10
common, and should take less space. But
very large numbers can happen – 7 00000111 11111110
consider term positions in very large files,
or document IDs in a large web collection. 13 00001101 11111111111110

What if we used a unary encoding? This

encodes k by k 1s, followed by a 0.
 Elias-ɣ Codes
Unary is efficient for small numbers, Decimal k k Code
but very inefficient for large numbers.
1 0 0 0
There are better ways to get a variable
bit length. 2 1 0 10 0

With Elias-ɣ codes, we use unary to 3 1 1 10 1

encode the bit length and then store 6 2 2 110 10
the number in binary.
15 3 7 1110 111
To encode a number k, compute:
16 4 0 11110 0000

! = log 255 7 127 11111110 1111111

log
= 1111111110
1023 9 511
111111111
 Elias-δ Codes
Elias-ɣ codes take log + bits. Decimal k k k k Code
We can do better, especially for large
1 0 0 0 0 0
numbers.
2 1 1 0 0 10 0 0
Elias-δ codes encode kd using an
3 1 1 0 1 10 0 1
Elias-ɣ code, and take approximately
log log + log bits. 6 2 1 1 2 10 1 10

We split kd into: 15 3 2 0 7 110 00 111

= log 16 4 2 1 0 110 01 0000

! log
= 255 7 3 0 127 1110 000
1111111
1023 9 3 2 511 1110 010
111111111
Python Implementation
 Delta Encoding
We now have an efficient variable bit
length integer encoding scheme which
uses just a few bits for small numbers,
Raw positions: 1, 5, 9, 18, 23, 24, 30, 44, 45, 48
and can handle arbitrarily large numbers
Deltas: 1, 4, 4, 9, 5, 1, 6, 14, 1, 3
with ease.

To further reduce the index size, we want High-frequency words compress more easily:
to ensure that docids, positions, etc. in 1, 1, 2, 1, 5, 1, 4, 1, 1, 3, ...
our lists are small (for smaller encodings)
and repetitive (for better compression). Low-frequency words have larger deltas:
109, 3766, 453, 1867, 992, ...
We can do this by sorting the lists and
encoding the difference, or delta,
between the current number and the last.
 Byte-Aligned Codes
In production systems, inverted lists are stored using byte-aligned
codes for delta-encoded integer sequences.

Careful engineering of encoding schemes can help tune this process

to minimize processing while reading the inverted lists. This is essential
for getting good performance in high-volume commercial systems.

Next, we’ll look at how to produce an index from a document collection.

CS6200: Information Retrieval

Slides by: Jesse Anderton
 Byte-Aligned Codes
We’ve looked at ways to encode integers with bit-aligned codes.
These are very compact, but somewhat inconvenient.

Processors and most I/O routines and hardware are byte-aligned, so

it’s more convenient to use byte-aligned integer encodings.

One of the commonly-used encodings is called vbyte. This encoding,

like UTF-8, simply uses the most significant bit to encode whether the
number continues to the next byte.
 Vbyte

k Bytes Used k Binary Hexadecimal

k 1 1 1 0000001 81

2 2 6 1 0000110 86

2 3 127 1 1111111 FF

2 4 128 0 0000001 1 0000000 01 80

130 0 0000001 1 0000010 01 82

20000 0 0000001 0 0011100 1 0100000 01 1C A0

Java Implementation
 Bringing It Together
Let’s see how to put together a compressed inverted list with delta encoding.
We start with the raw inverted list: a sequence of tuples containing (docid,
tf, [pos1, pos2, …]).

(1,2,[1,7]), (2,3,[6,17,197]), (3,1,[1])

We delta-encode the docid and position sequences independently.

(1,2,[1,6]), (1,3,[6,11,180]), (1,1,[1])

Finally, we encode the integers using vbyte.

81 82 81 86 81 82 86 8B 01 B4 81 81 81
 Alternative Codes
Although vbyte is often adequate, we can do better for high-performance
decoding.

Vbyte requires a conditional branch at every byte and a lot of bit shifting.

Google’s Group VarInt encoding achieves much better decoding

performance by storing a two bit continuation sequence for each of the
next 4-16 bytes.

Decimal: 1 15 511 131071

Encoded: 00000110 00000001 00001111 11111111 00000001 11111111 11111111 00000001