Chapter Three

Indexing structure
 Indexing: Basic Concepts
• Indexing is an arrangement of index terms to permit fast searching and
reducing memory space requirement. It is an offline process of organizing
document using keywords extracted from the collection
• used to speed up access to desired information from document collection as per
users query such that
– It enhances efficiency in terms of time for retrieval.
o Relevant documents are searched and retrieved quick
– Index file usually has index terms in a sorted order.
Example: author catalog in library
• Example: Consider the following list of terms:
 A: fox pig zebra hen ant cat dog lion ox
 B : ant cat dog fox hen lion ox pig zebra
Which list is easier to search? A or B? •2
 Cont….
• An index file consists of records, called index entries.
• Index files are much smaller than the original file.
– This size is reduced by Linguistic pre-processing(or text
operations).

• The usual unit for indexing is the word

– Index terms - are used to look up records in a file.

Indexing structure By: Abenet A. •3

 Major Steps in Index Construction
• Source file: Collection of text document
–A document can be described by a set of representative keywords called
index terms.

• Index Terms Selection: apply text operations or preprocessing

–Tokenize: identify words in a document, so that each document is
represented by a list of keywords or attributes

–Stop words removal: words with high frequency are non-content

bearing and needs to be removed from text collection

Indexing structure By: Abenet A. •4

 Cont…
–Word stem and Normalization : reduce words with
similar meaning into their stem/root word.
• Suffix stripping is the common method

–Term relevance weight: Different index terms have varying relevance

when used to describe document contents.

o This effect is captured through the assignment of numerical weights

to each index term of a document. There are different index terms
weighting methods: including TF, IDF,TF*IDF, …

• Output: a set of index terms (vocabulary) are organized in Index File to

easily identify documents in which each term occurs in.

Indexing structure By: Abenet A. •5

 Basic Indexing Process
Documents to
be indexed. Friends, Romans, countrymen.

Token Tokenizer
stream. Friends Romans countrymen

Modified Linguistic friend roman countryman

tokens. preprocessor

Index File Indexer

friend 2 4

roman 1 2
Inverted file countryman 13 16
•6
Indexing structure By: Abenet A.
 Building Index file
•An index file of a document is a file consisting of a list of index terms
and a link to one or more documents that has the index term

 Example:

• A good index file maps each keyword Ki to a set of documents Di

that
contain the keyword

o i.e. Ki = term1, term2,….,termn, and Di=doc1,doc2,…..docn

•Index file usually has index terms in a sorted order.

–The sort order of the terms in the index file provides an order on a
•7
physical file
 Cont…
•An index file is list of search terms that are organized for associative
look- up, i.e., to answer user's query:

–In which documents does a specified search term appear?

–Where within each document does each term appear? (There may be
several occurrences.)

•For organizing index file for a collection of documents, there are various
options available:

–Decide what data structure and/or file structure to use. Is it sequential

file, inverted file, suffix array, signature file, etc. ?

Indexing structure By: Abenet A. •8

 Index file Evaluation Metrics
1) Running time of the main operations

• Access/search time: : is that allows sequential or random searching/access?

• Indexing time
• How much is the running time to find the required search key from the list?

2) Update time (Insertion time, Deletion time, modification time….)

– How much time does it take to update existing records in an attempt to add new
terms or delete existing unnecessary terms?

– Does the indexing structure allows incremental update or re-indexing?

3)Space overhead

– Computer storage space consumed for keeping the list.

4) Access types supported efficiently.
 Is the indexing structure allows to access:
• records with a specified term, or
• records with terms falling in a specified range of values.
•9
 Inverted file
 Inverted file is a word oriented indexing mechanism based on
sorted list of keywords, with each keyword having links to the
documents containing it

– Building and maintaining an inverted index is a relatively

low cost risk.
– On a text of n words an inverted index can be built in O(n)
time,

where, n is number of keywords

– This list is inverted from a list of terms in location order to a

list of
terms in alphabetical order.
•10
Inverted file(Cont…)
 Content of the inverted file:
• Data to be held in the inverted file includes:
– The vocabulary (List of terms): is the set of all distinct
words (index terms) in the text collection.
o Having information about vocabulary (list of terms) speeds

searching for relevant documents

– The occurrence(Location and frequency of terms in a
document collection)

Indexing structure By: Abenet A. •11

Inverted file(Cont…)
• For each term: the inverted file contains information related to:
– frequency of occurrence of each term in a document collection, i.e.
count number of occurrences of keywords in a document

• TFij, number of occurrences of term tj in document di

• DFj, number of documents containing tj

• maxi, maximum frequency of any term in di
• N, total number of documents in a collection

• CFj,, collection frequency of tj in nj

o total frequency of tj in the corpus n
– Location: all the locations/positions of words in the text where the
word occurs
Indexing structure By: Abenet A. •12
Inverted file(Cont…)
 Why vocabulary is needed?
– Having information about vocabulary (list of terms) speeds searching
for relevant documents

 Why location is needed?

– Having information about the location of each term within the
document helps for:
• user interface design: highlight location of search term
• proximity based ranking: adjacency and near operators (in Boolean
searching)
 Why frequencies are needed?
– Having information about frequency is used for:
• calculating term weighting (like TF, TF*IDF, …)
• optimizing query processing
Indexing structure By: Abenet A. •13
Inverted file(Cont…)
 Documents are organized by the terms/words they contain

Term CF Doc ID TF Location

term 1 3 2 1 66
This is called an
auto 19 1 213 index file.
29 1 45
term 2 4 3 1 94
bus 19 2 7, 212 Text operations
are performed
22 1 56
before building
term 3 1 5 1 43 the index.
taxi
term 4 3 11 2 3, 70
train 34 1 40

• Is it possible to keep all these information during searching? •14

 Inverted file Construction
 An inverted index consists of two files: vocabulary and posting files

– Vocabulary file And Posting file

• A vocabulary file (Word list):

– stores all of the distinct terms (keywords) that appear in any of the
documents (in lexicographical order) and

– For each word a pointer to posting file

• Records kept for each term j in the word list contains the following:

– term j

– number of documents in which term j occurs (DFj)

– Collection frequency of term j (Cf)

– pointer to inverted (postings) list for term j

Indexing structure By: Abenet A. •15
 Inverted file Construction (Cont…)

• Postings File (Inverted List):

─ For each distinct term in the vocabulary, stores a list of pointers
to the documents that contain that term.
─ Each element in an inverted list is called a posting, i.e., the
occurrence
of a term in a document
• It is stored as a separate inverted list for each column, i.e., a
list corresponding to each term in the index file.

o Each list consists of one or many individual postings related to

Document ID, TF and location information about a given term i

Indexing structure By: Abenet A. •16

 Inverted file Construction (Cont…)
 Advantage of dividing inverted file:
• Keeping a pointer in the vocabulary to the list in the posting file
allows:
– the vocabulary to be kept in memory at search time even for
large text collection, while
– the Posting file is kept on disk for accessing to documents

Indexing structure By: Abenet A. •17

Organization of Index File
An inverted index consists of two files: vocabulary file Posting file

•18
Cont…
Example:
 Inverted index storage
• Separation of inverted file into vocabulary and posting
file is a good idea.
• Vocabulary: For searching purpose we need only word list. This
allows the vocabulary to be kept in memory at search time since
the space required for the vocabulary is small.
• The vocabulary grows by O(nβ), where β is a constant between 0 – 1.
• Example: from 1,000,000,000 documents, there may be 1,000,000 distinct
words. Hence, the size of index is 100 MBs, which can easily be held in
memory of a dedicated computer.
• Posting file requires much more space.
• For each word appearing in the text we are keeping statistical information
related to word occurrence in documents.
• Each of the postings pointer to the document requires an extra space of O(n).
• How to speed up access to inverted file?
 Example: inverted File
• Given a collection of documents, they are parsed to
extract words and these are saved with the Document
ID.
I did enact Julius
Doc 1 Caesar I was killed
I the Capitol;
Brutus killed me.

So let it be with
Doc 2 Caesar. The noble
Brutus has told you
Caesar was ambitious
Indexing structure By: Abenet A. •21
Example(Cont…)
 Steps to build Inverted Files
• To build inverted files, follow the following steps:

 Step 1: Tokenize the given documents

• After all documents have been tokenized the inverted
file is sorted by terms as shown below:

Indexing structure By: Abenet A. •22

Sorting the Vocabulary Term Doc #
ambitious 2
Term Doc # be 2
I 1
brutus 1
did 1
brutus 2
enact 1
julius 1 capitol 1
caesar 1 caesar 1
I 1 caesar 2
was 1 caesar 2
killed 1 did 1
I 1 enact 1
the 1 has 1
capitol 1 I 1
brutus 1
I 1
killed 1
I 1
me 1
so 2 it 2
let 2 julius 1
it 2 killed 1
be 2 killed 1
with 2 let 2
caesar 2 me 1
the 2 noble 2
noble 2 so 2
brutus 2
the 1
hath 2
the 2
told 2
you 2 told 2
caesar 2 you 2
was 2 was 1
ambitious 2 was 2
with 2
 Example(Cont…)
 Step 2: Remove stop words, stemming & compute frequency
• After all stop words have been removed and then stemmed from
all documents:

o Multiple term entries in a single document are merged and

frequency information added

o Counting number of occurrence of terms in the collections

helps to compute TF as shown below

Indexing structure By: Abenet A. •24

 Example(Cont…)
Term Doc # Term Doc # TF
ambition 2 ambition 2 1
brutus 1 brutus 1 1
brutus 2 brutus 2 1
capitol 1 capitol 1 1
caesar 1 caesar 1 1
caesar 2 caesar 2 2
caesar 2 enact 1 1
enact 1 julius 1 1
julius 1 kill 1 2
kill 1 noble 2 1
kill 1
noble 2

Indexing structure By: Abenet A. •25

Example(Cont…)
 Step 3: Create Vocabulary and postings file
• The file is commonly split into a Vocabulary and a Posting file

•26
 Complexity Analysis
 The inverted index can be built in O(n) + O(n log n) time, where

– n is the number of vocabulary terms

 Since terms in vocabulary file are sorted searching takes
logarithmic time.
 To update the inverted index it is possible to apply Incremental

indexing which requires O(k) time, where

– k is number of new index terms

Indexing structure By: Abenet A. •27

 Searching on Inverted File
 Since the whole index file is divided into two, searching can be
done faster by loading vocabulary list which takes less memory
even for large document collection

 Using binary Search the searching takes logarithmic time

– The search is in the vocabulary lists

 Updating inverted file is complex.

– We need to update both vocabulary and posting files

Indexing structure By: Abenet A. •28

 Exercise
 Construct the inverted index file for the following document
collections:

Doc 1: The Department of Information Technology was established in 2002.

Doc 2 : The Department launched its first BSc in Computer Studies in 2002.

Doc 3 : Followed by the MSc in Information Technology which was started in 2003.
Doc 4 : The Department also produced its first MSc graduate in 2005.
Doc 5: Our staff have contributed intellectually and professionally to the

advancements in these fields.

Indexing structure By: Abenet A. •29

 Sequential File
Sequential file is the most primitive file structures.
It has no vocabulary as well as linking pointers.
The records are generally arranged serially, one after another, but
in lexicographic order on the value of some key field.
 a particular attribute is chosen as primary key whose value will
determine the order of the records.
 when the first key fails to discriminate among records, a second
key is chosen to give an order.
 Example:
Given a collection of documents, they are parsed to extract words and
these are saved with the Document ID.
 Sorting the Vocabulary
Term Doc #
I 1
After all documents did 1 Doc
enact 1
have been tokenized, julius 1 Term No.
stopwords are caesar 1
1 ambit 2
I 1
removed, and was 1 2 brutus 1
normalization and killed 1
stemming are applied, I 1 3 brutus 2
the 1
to generate index capitol 1
4 capitol 1
terms brutus 1 5 caesar 1
These index terms in killed 1
me 1 6 caesar 2
sequential file are so 2
sorted in alphabetical let 2 7 caesar 2
order it 2 8 enact 1
be 2
with 2 9 julius 1
caesar 2
the 2 10 kill 1
noble 2
brutus 2
11 kill 1
hath 2 12 noble 2
told 2
you 2
caesar 2
was 2
ambitious 2
 Complexity Analysis

• Creating sequential file requires O(n log n) time, n is the

total number of content-bearing words identifies from the
corpus.

• Since terms in sequential file are sorted, the search time is

logarithmic using binary tree.

• Updating the index file needs re-indexing; that means

incremental indexing is not possible
 Sequential File
• Its main advantages are:
• easy to implement;
• provides fast access to the next record using lexicographic order.
• Instead of Linear time search, one can search in logarithmic time
using binary search
• Its disadvantages:
• difficult to update. Index must be rebuilt if a new term is added.
Inserting a new record may require moving a large proportion of
the file;
• random access is extremely slow.
• The problem of update can be solved :
• by ordering records by date of acquisition, than the key value;
hence, the newest entries are added at the end of the file &
therefore pose no difficulty to updating. But searching becomes
very tough; it requires linear time
Suffix Treeand SuffixArray

Indexing structure By: Abenet A.

 Suffix tree
 What is Suffix?
• A suffix is a substring that exists at the end of the given string.
– Each position in the text is considered as a text suffix
– If txt=t1t2...ti...tn is a string, then Ti=ti, ti+1...tn is the suffix of txt
that starts at position i, where 1≤ i ≤ n.
• Example: txt = mississippi txt = GOOGOL
T1 = mississippi; T1 = GOOGOL
T2 = ississippi; T2 = OOGOL
T3 = ssissippi; T3 = OGOL
T4 = sissippi; T4 = GOL
T5 = issippi; T5 = OL
T6 = ssippi; T6 = L
T7 = sippi;
T8 = ippi; Exercise: generate suffix of “technology” ?
T9 = ppi;
T10 = pi;
T11 = i;
Indexing structure By: Abenet A. •36
 Suffix trie
 What is suffix trie?
• A suffix trie is an ordinary trie in which the input strings are all
possible suffixes.

– Principles: The idea behind suffix TRIE is to assign to each symbol

in a text an index corresponding to its position in the text.
o i.e: First symbol has index 1, last symbol has index n
(number of symbols in text).
• To build the suffix TRIE we can use these indices instead of the
actual object.

Indexing structure By: Abenet A. •37

 Suffix trie(Cont…)
 The structure of suffix trie has several advantages:

– It requires less storage space.

– We do not have to worry how the text is represented (binary, ASCII,

etc.).

– We do not have to store the same object twice (no duplicate).

– Whatever the size of index terms, the search time is also linear in the length
of string S.

Indexing structure By: Abenet A. •38

  Example: suffix trie
 Construct suffix trie for the following string: “GOOGOL”
• To construct suffix trie, we begin by giving a position to every suffix in
the text starting from left to right as per characters occurrence in the
string.

TEXT: G O O G O L $

POSITION: 1 2 3 4 5 6 7

• Build a SUFFIX TRIE for all n suffixes of the text.

• Note: The resulting a tree has n leaves and height n.

Indexing structure By: Abenet A. •39

 Example (Cont…)
• This structure is
particularly
useful for any
application
requiring prefix
based ("starts
with") pattern
matching.

Indexing structure By: Abenet A. •40

 Suffix tree
 What is Suffix tree?

• A suffix tree is a member of the

trie family.
• It is a Trie of all the proper
suffixes of S

– The suffix tree is created by

compacting unary nodes of the
suffix TRIE as shown in the
figure.

•35
Suffix tree(Cont…)
• We store pointers rather than words Example
in the leaves.

–It is also possible to replace

strings in every edge by a pair
(a,b), where a & b are the
beginning and end index of the
string. i.e.

(3,7) for OGOL$

(1,2) for GO
(7,7) for $ as shown in the
figure. •36
 Example: Suffix tree
 Let s=abab, a suffix tree of s is a compressed trie of all suffixes
of s=abab$
• We label each leaf with the starting
•{ point of the corresponding suffix.
• $ •$
• b$ •ab
•b •5
$

• ab$ •$ ab

abab

• bab$ •$ •ab$ •4 b

• abab$ •ab$
bab

•3
•} •2
•1

Indexing structure By: Abenet A. •43

 Complexity Analysis

• The suffix tree for a string has been built in O(n2) time.

• The search time is proportional to the length of string S;

i.e. O(|S|).

• Searching for a substring[1..m], in string[1..n], can be

solved in O(m) time
• It requires to search for the length of the string O(|S|).

• Updating the index file can be done incrementally

without affecting the existing index
 Generalized suffix tree
• Given a set of strings S, a generalized suffix tree of S is a
compressed trie of all suffixes of s  S

•To make suffixes prefix-free we add a special char, $, at the end of

s.

•To associate each suffix with a unique string in S add a different

special symbol to each s

• Build a suffix tree for the string s1$s2#, where `$' and `#' are a
special terminator for s1,s2.

Indexing structure By: Abenet A. •45

  Example: Generalized suffix tree
 Let s1=abab & s2=aab, a generalized suffix tree for s1 & s2 is:

•{
•a •$ •#
• $ #
•b
• b$ b#
•# •5 •4
• ab$ ab# •b
•ab$ •ab$ •$
• bab$ aab#
•3
• abab$ •ab$ •# •4
•$ •1
•} •2
•1 •3 •2

Indexing structure By: Abenet A. •46

 Search in suffix tree
• Searching for all instances of a substring S in a suffix tree is easy since any
substring of S is the prefix of some suffix.

• Pseudo-code for searching in suffix tree:

– Start at root

– Go down the tree by taking each time the corresponding path

– If S correspond to a node, then return all leaves in sub-tree

• the places where S can be found are given by the pointers in all the
leaves in the subtree rooted at x.
– If S encountered a NIL pointer before reaching the end, then S is not in
the tree
Indexing structure By: Abenet A. •47
 Cont…

 Example:

• If S = "GO" we take the GO path and return:

GOOGOL$,GOL$.

• If S = "OR" we take the O path and then we hit a NIL pointer

so "OR" is not in the tree.

Indexing structure By: Abenet A. •48

 Exercise
Given the following index terms:
worker, word, world, run & information
construct index file using suffix tree?

Indexing structure By: Abenet A. •49

 Suffix TreeApplications
• Suffix Tree can be used to solve a large number of string problems that occur in:

–text-editing,

–free-text search,

–etc.

• Some examples of string problems are given below.

–String matching

–Longest Common Substring

–Longest Repeated Substring

–Palindromes

–etc..
Indexing structure By: Abenet A. •50
 Suffix tree Drawbacks
 Suffix trees consume a lot of space
– Even if word beginnings are indexed, space overhead of 120% -
240% over the text size is produced.

– Because depending on the implementation each nodes of the suffix

tree takes a space (in bytes) equivalent to the number of symbols
used.

– How much space is required at each node for English word indexing
based on alphabets a to z.

 Example:

• How many bytes required to store MISSISSIPI ?

•51
 Suffix array
 A suffix array is more compact than a suffix tree.

– Suffix arrays are a space efficient implementation of suffix trees

 Like suffix tree, a suffix array is a sorted list of the suffixes of a given string
in lexicographical order.

– The sorted list is presented as an array of integers that identify the

suffixes in order.

– This allows a binary search or fast substring search.

 The main drawbacks of Suffix array are:

– Its costly construction process.

– The need for the document/text to be readily available at query time

•52
 Building suffix array
 To build suffix array follow the following procedures:

– Identify suffixes of the given string

– Sort the suffixes lexicographically

– Store indices of all the suffixes in a table.

 The suffix array gives the indices of the suffixes in sorted order

 A suffix array can be constructed in O(n log n) time, where n is

the length of the string, by sorting the suffixes.

•53
 Example: Building suffix array

 Consider the following string “GOOD”.

– At the end, a special character($) is usually appended to the
string.

– In lexicographical order, the suffixes are "d$", "good$", "od$”

and "ood$".

– The suffix array is [4, 1, 3, 2, 5].

•54
  Example 2:Building a suffix array
 Given the string S =“GOOGOL”, construct suffix array
• Sort the suffixes in lexicographical order and store in a table all the
indices of the given string S.

Suffixes Indices Ptr

GOL$ S[0] 4
GOOGOL$ S[1] 1
L$ S[2] 6
OGOL$ S[3] 3
OL$ S[4] 5
OOGOL$ S[5] 2
$ S[6] 7

Not stored Stored

•55
Exercise
 How can you build suffix array for multiple strings?

• Construct suffix array for the following strings:

“GOOD” and “GOOGOL”

•50
 How do we search for a pattern?
 If P occurs in T then all its occurrences are consecutive(sequential) in the
suffix array.

 Do a binary search on the suffix array

 Takes O(logn) time

• Example 1: search for „good‟ in the suffix array constructed for

„GOOGOL‟.
• Exercise: Let the string given is S = mississippi, construct suffix array and
search for:
(i) ppi
(ii) issa

•51
 Example
•Let S = mississippi
L 11 i
8 ippi
5 issippi
•Let P = issa
2 ississippi
1 mississippi
M 10 pi
9 ppi
7 sippi
4 sisippi
6 ssippi
R 3 ssissippi •52
 Thank you

Indexing structure By: Abenet A.