S TR U C T U R E

I ND E X I N G

 Designing an IR system
 Inverted Index
 Suffix Tree

10/20/24 1
 Designing an IR System
Our focus during IR system design:
In improving Effectiveness of the system
• The concern here is retrieving more relevant documents for users
query
• Effectiveness of the system is measured in terms of precision,
recall, …
• Main emphasis: Stemming, stop words removal, weighting
schemes, matching algorithms
In improving Efficiency of the system
• The concern here is reducing storage space requirement,
enhancing searching time, indexing time, access time…
• Main emphasis: Compression, indexing structures, space – time
tradeoffs
10/20/24 2
 Subsystems of IR system
The two subsystems of an IR system: Indexing and Searching
 Indexing:
• is an offline process of organizing documents using
keywords extracted from the collection
• Indexing is used to speed up access to desired
information from document collection as per users query
 Searching
• Is an online process that scans document corpus to find
relevant documents that matches users query

10/20/24 3
 Indexing Subsystem
documents
Documents Assign document identifier

document document
Tokenization
IDs
tokens
Stopword removal
non-stoplist tokens
Stemming &
stemmed terms
Normalization
Term weighting

Weighted index
terms Index File
10/20/24 4
 Searching Subsystem
query parse query
query tokens
ranked
Stop word non-stoplist
document
tokens
set
Ranking
Stemming & Normalize
relevant stemmed terms
document set
Similarity Query Term weighting
Measure terms
Index terms
Index

10/20/24 5
 Basic assertion
Indexing and searching: (inexorably connected)
you cannot search that was not first indexed in some
manner or other.
indexing of documents or objects is done in order to be
searchable .
• there are many ways to do indexing
o to index one needs an indexing language
• there are many indexing languages
o even taking every word in a document is an
indexing language

“Knowing
10/20/24 searching is knowing indexing” 6
 Implementation Issues
 Storage of text:
o The need for text compression: to reduce storage space
 Indexing text
o Organizing indexes
•What techniques to use ? How to select it ?
o Storage of indexes
•Is compression required? Do we store on memory or in a disk ?
 Accessing text
o Accessing indexes
•How to access to indexes ? What data/file structure to use?
o Processing indexes
•How to search a given query in the index? How to update the
10/20/24index? 7
o Accessing documents
 Indexing: Basic Concepts
 Indexing is used to speed up access to desired information from
document collection as per users query such that
• It enhances efficiency in terms of time for retrieval. Relevant
documents are searched and retrieved quick
Example: author catalog in library
 An index file consists of records, called index entries.
• The usual unit for indexing is the word
o Index terms - are used to look up records in a file.

 Index files are much smaller than the original file. Do you agree?
• Remember Heaps Law: In 1 GB text collection the size of a
vocabulary is only 5 MB (Baeza-Yates and Ribeiro-Neto, 2005)
• This size may be further reduced by Linguistic pre-processing
(like stemming & other normalization methods).
10/20/24 9
 Major Steps in Index
Construction
 Source file: Collection of text document
•A document can be described by a set of representative
keywords called index terms.
 Index Terms Selection:
o Tokenize: identify words in a document, so that each
document is represented by a list of keywords or
attributes
o Stop words: removal of high frequency words
•Stop list of words is used for comparing the input text
o Stemming and Normalization: reduce words with
similar meaning into their stem/root word
10/20/24 •Suffix stripping is the common method 10
 Major Steps in Index
Construction
o Weighting terms: Different index terms have varying importance
when used to describe document contents.
• This effect is captured through the assignment of numerical
weights to each index term of a document.
• There are different index terms weighting methods (TF, DF, CF)
based on which TF*IDF weight can be calculated during
searching
 Output: a set of index terms (vocabulary) to be used for Indexing
the documents that each term occurs in.
10/20/24
11
 Basic Indexing Process
Documents to
be indexed. Friends, Romans,
countrymen.
Token Tokenizer
stream. Friends Roman countrymen
s
Modified Linguistic friend roman countryma
tokens. preprocessing
n

Index File Indexer friend 2 4

(Inverted 1 2
roman
file).
countryman 13 16
10/20/24 12
 Building Index file
 An index file of a document is a file consisting of a list of index terms and a link to one
or more documents that has the index term
o A good index file maps each keyword Ki to a set of documents Di that contain the
keyword

 Index file usually has index terms in a sorted order.

o The sort order of the terms in the index file provides an order on a physical file
 An index file is list of search terms that are organized for associative look-up, i.e., to
answer user’s query:
o In which documents does a specified search term appear?
o Where within each document does each term appear? (There may be several
occurrences.)
 For organizing index file for a collection of documents, there are various options
available:
o Decide what data structure and/or file structure to use. Is it sequential file, inverted13
file, suffix array, signature file, etc. ?
10/20/24
 Index file Evaluation
Metrics
 Running time
 Indexing time
 Access/search time: is that allows sequential or random
searching/access?
 Update time (Insertion time, Deletion time, modification
time….): can the indexing structure support re-indexing or
incremental indexing?
 Space overhead
 Computer storage space consumed.
 Access types supported efficiently.
 Is the indexing structure allows to access:
•records with a specified term, or 14
10/20/24
•
 Sequential File
• Sequential file is the most primitive file structures.
It has no vocabulary as well as linking pointers.
• The records are generally arranged serially, one after another, but in
lexicographic order on the value of some key field.
a particular attribute is chosen as primary key whose value will
determine the order of the records.
when the first key fails to discriminate among records, a second key
is chosen to give an order.

10/20/24 15
 Example:
•Given a collection of documents, they are parsed to
extract words and these are saved with the Document
ID.

I did enact Julius

Doc 1 Caesar I was killed
I the Capitol;
Brutus killed me.

So let it be with
Doc 2 Caesar. The noble
Brutus has told you
10/20/24 Caesar was ambitious 16
 Sorting the Vocabulary
Term Doc #
I 1
•After all did 1 Sequential file
documents enact
julius
1
1 Doc
have been caesar 1
Term No.
I 1
tokenized, was 1 1 ambition 2
stopwords are killed
I
1
1 2 brutus 1
removed, and the 1
3 brutus 2
normalization capitol
brutus
1
1 4 capitol 1
and stemming killed 1
5 caesar 1
me 1
are applied, to so 2
6 caesar 2
generate index let 2
it 2 7 caesar 2
terms be 2
with 2 8 enact 1
•These index caesar 2
9 julius 1
terms in the 2
noble 2
10 kill 1
sequential file brutus 2
hath 2 11 kill 1
are sorted in told 2
alphabetical you 2 12 noble 2
caesar 2
order
10/20/24 was 2 17
ambitious 2
 Complexity Analysis
 Creating sequential file requires O(n log n) time, n is the
total number of content-bearing words identifies from the
corpus.
 Since terms in sequential file are sorted, the search time is
logarithmic using binary tree.
 Updating the index file needs re-indexing; that means
incremental indexing is not possible

10/20/24 18
 Sequential File
 Its main advantages are:
• easy to implement;
• provides fast access to the next record using lexicographic order.
• Instead of Linear time search, one can search in logarithmic time using
binary search
 Its disadvantages:
• difficult to update. Index must be rebuilt if a new term is added.
Inserting a new record may require moving a large proportion of the file;
• random access is extremely slow.
 The problem of update can be solved :
• by ordering records by date of acquisition, than the key value; hence,
the newest entries are added at the end of the file & therefore pose no
difficulty to updating. But searching becomes very tough; it requires
linear time
10/20/24 19
 Inverted file
 A technique that index based on sorted list of terms, with each term having
links to the documents containing it
• Building and maintaining an inverted index is a relatively low cost risk. On a
text of n words an inverted index can be built in O(n) time, n is number of
terms
 Content of the inverted file: Data to be held in the inverted file includes :
• The vocabulary (List of terms)
• The occurrence (Location and frequency of terms in a document collection)
 The occurrence: contains one record per term, listing
• Frequency of each term in a document
 TFij, number of occurrences of term tj in document di
 DFj, number of documents containing tj
 maxi, maximum frequency of any term in di
 N, total number of documents in a collection
 CFj,, collection frequency of tj in nj
20
• Locations/Positions of words in the text
 Term Weighting: Term
Frequency (TF)
 TF (term frequency) - Count the number of
docs t1 t2 t3
times a term occurs in document.
D1 2 0 3
fij = frequency of term i in document j
D2 1 0 0
D3 0 4 7
 The more times a term t occurs in document d D4 3 0 0
the more likely it is that t is relevant to the
D5 1 6 3
document, i.e. more indicative of the topic..
D6 3 5 0
• If used alone, it favors common words and
D7 0 8 0
long documents.
D8 0 10 0
• It gives too much credit to words that
appears more frequently. D9 0 0 1
 There is a need to normalize term frequency (tf) D10 0 3 5
D11 4 0 1
10/20/24 21
 Document Frequency
 It is defined to be the number of documents in the collection that
contain a term

DF = document frequency

• Count the frequency considering the whole collection of documents.

• Less frequently a term appears in the whole collection, the more

discriminating it is.

df i (document frequency of term i) = number of documents containing

term i
10/20/24 22
 Inverted file
•Why vocabulary?
 Having information about vocabulary (list of terms) speeds searching
for relevant documents
•Why location?
 Having information about the location of each term within the
document helps for:
o user interface design: highlight location of search term
o proximity based ranking: adjacency and near operators (in Boolean
searching)
•Why frequencies?
 Having information about frequency is used for:
o calculating term weighting (like IDF, TF*IDF, …)
10/20/24 o optimizing query processing 23
 Inverted file
Documents are organized by the terms/words they
contain
Term CF Document TF Location
ID
auto 3 2 1 66 This is called an index
file.
19 1 213
29 1 45
bus 4 3 1 94
19 2 7, 212 Text operations are
22 1 56 performed before
taxi 1 5 1 43 building the index.
train 3 11 2 3, 70
34 1 40 24
 Organization of Index
Fileof two files:
An inverted index consists
•vocabulary file
•Posting file
Postings Actual
Vocabulary (word (inverted list) Documents
list)
Term No Tot Pointer
of freq To
Doc posting

Act 3 3 Inverted
lists
Bus 3 4
pen 1 1
total 2 3
25
10/20/24
 Inverted File
 Vocabulary file (Word list):
•stores all of the distinct terms (keywords) that appear in any of the
documents (in lexicographical order) and
•For each word a pointer to posting file
 Records kept for each term j in the word list contains the following:
term j, DFj, CFj and pointer to posting file
 Postings File (Inverted List)
 For each distinct term in the vocabulary, stores a list of pointers to
the documents that contain that term.
 Each element in an inverted list is called a posting, i.e., the
occurrence of a term in a document
 It is stored as a separate inverted list for each column, i.e., a list
corresponding to each term in the index file.
•Each list consists of one or many individual postings related to
Document ID, TF and location information about a given term i 26
 Construction of Inverted
file
Advantage of dividing inverted file:
•Keeping a pointer in the vocabulary to the list in the posting file
allows:
– the vocabulary to be kept in memory at search time even for
large text collection, and
– Posting file to be kept on disk for accessing to documents
•Exercise:
– In the Terabyte of text collection, if 1 page is 100KBs and each
page contains 250 words, on the average, calculate the memory
space requirement of vocabulary words? Assume 1 word
contains 10 characters. 27
 Inverted index storage
 Separation of inverted file into vocabulary and posting file is a good idea.
 Vocabulary: For searching purpose we need only word list. This allows
the vocabulary to be kept in memory at search time since the space
required for the vocabulary is small.
• The vocabulary grows by O(nβ), where β is a constant between 0 – 1.
• Example: from 1,000,000,000 documents, there may be 1,000,000
distinct words. Hence, the size of index is 100 MBs, which can easily
be held in memory of a dedicated computer.
 Posting file requires much more space.
• For each word appearing in the text we are keeping statistical
information related to word occurrence in documents.
• Each of the postings pointer to the document requires an extra space
of O(n).
 How to speed up access to inverted file? 28
 Example:
•Given a collection of documents, they are parsed to
extract words and these are saved with the Document ID.

I did enact Julius

Doc 1 Caesar I was killed
I the Capitol;
Brutus killed me.

So let it be with
Doc 2 Caesar. The noble
Brutus has told you
10/20/24 Caesar was ambitious 29
 Sorting the Vocabulary Term Doc #
ambitious 2
Term Doc # be 2
I 1 brutus 1
did 1 brutus 2
enact 1 capitol 1
julius 1 caesar 1
•After all caesar
I
1
1
caesar 2
caesar 2
documents was
killed
1
1
did 1
enact 1
have been I
the
1
1 has 1

tokenized the capitol

brutus
1
1
I
I
1
1

inverted file killed

me
1
1
I
it
1
2

is sorted by
so 2 julius 1
let 2 killed 1
it 2
terms be
with
2
2
killed
let
1
2
caesar 2 me 1
the 2 noble 2
noble 2 so 2
brutus 2 the 1
hath 2 the 2
told 2 told 2
you 2 you 2
caesar 2 was 1
was 2
10/20/24 was 2 30
ambitious 2
with 2
 Remove stopwords, apply stemming & compute term frequency

•Multiple term Term Doc #

Term Doc # TF
entries in a single ambition 2
ambition 2 1
document are brutus 1
brutus 1 1
merged and brutus 2
brutus 2 1
capitol 1
frequency capitol 1 1
caesar 1
information added caesar 2
caesar 1 1
•Counting number caesar 2 2
caesar 2
enact 1 1
of occurrence of enact 1
julius 1 1
terms in the julius 1
kill 1 2
collections helps kill 1
noble 2 1
kill 1
to compute TF
noble 2

10/20/24 31
 Vocabulary and postings file
The file is commonly split into a Dictionary and a Posting file

vocabulary posting
Term Doc # TF Term DF CF Doc # TF
ambition 2 1 ambitious 1 1 2 1
brutus 1 1 1 1
brutus 2 2
brutus 2 1 2 1
capitol 1 1 capitol 1 1 1 1
caesar 1 1 caesar 2 3 1 1
caesar 2 2 enact 1 1 2 2
enact 1 1 1 1
julius 1 1 1 1
julius 1 1 kill 1 2 1 2
kill 1 2
noble 2 1
noble 1 1 2 1

Pointers
10/20/24 32
 Suffix Trie and Tree

10/20/24 35
 Suffix trie
 What is Suffix? A suffix is a substring that exists at the end of the
given string.
 Each position in the text is considered as a text suffix
 If txt=t1t2...ti...tn is a string, then Ti=ti, ti+1...tn is the suffix of txt that
starts at position i,
 Example: txt = mississippi txt = GOOGOL
T1 = mississippi; T1 = GOOGOL
T2 = ississippi; T2 = OOGOL
T3 = ssissippi; T3 = OGOL
T4 = sissippi; T4 = GOL
T5 = issippi; T5 = OL
T6 = ssippi; T6 = L
T7 = sippi;
T8 = ippi;
T9 = ppi;
T10 = pi; 36
T11 = i;
 Suffix trie
A suffix trie is an ordinary trie in which the input strings are all
possible suffixes.
Principles: The idea behind suffix TRIE is to assign to each
symbol in a text an index corresponding to its position in the text.
(i.e: First symbol has index 1, last symbol has index n (number of
symbols in text).
• To build the suffix TRIE we use these indices instead of the
actual object.
The structure has several advantages:
We do not have to store the same object twice (no duplicate).
Whatever the size of index terms, the search time is also linear
in the length of string S. 37
 Suffix Trie
Construct SUFFIX TRIE for the following string: GOOGOL
We begin by giving a position to every suffix in the text starting from left to right as
per characters occurrence in the string.
TEXT : GOOGOL$
POSITION : 1 2 3 4 5 6 7
Build a SUFFIX TRIE for all n suffixes of the text.
•Note: The resulting tree has n leaves and height n.

This structure is
particularly useful for
any application
requiring prefix
based ("starts with")
pattern matching.

10/20/24 38
 Suffix tree
 A suffix tree is a member of the
trie family. It is a Trie of all the
O
proper suffixes of S
• The suffix tree is created by
compacting unary nodes of the
suffix TRIE.
 We store pointers rather than
words in the leaves.
• It is also possible to replace
strings in every edge by a pair
(a,b), where a & b are the
beginning and end index of the
string. i.e.
(3,7) for OGOL$
(1,2) for GO
39
(7,7) for $
 Example: Suffix tree
• Let s=abab, a suffix tree of s is a compressed trie of
all suffixes of s=abab$

{ $
1 abab$ ab
b 5
2 bab$ $
3 ab$ • We label each
$ ab$ 4 leaf with the
4 b$ ab$ starting point of
3
5 $ 2 the
1 corresponding
}
suffix.
10/20/24 40
 Generalized suffix tree
Given a set of strings S, a generalized suffix tree of S is a
compressed trie of all suffixes of s  S
• To make suffixes prefix-free we add a special char, $, at the end of s.
To associate each suffix with a unique string in S add a different
special symbol to each s
Build a suffix tree for the string s1$s2#, where `$' and `#'
are a special terminator for s1,s2.
• Ex.: Let s1=abab & s2=aab, a generalized suffix tree for s1 & s2 is:

{ a $ #
b
1. abab$ 1. aab#
2. bab$ 2. ab# # 5 4
ab#
3. ab$ 3. b# b ab$ $
4. b$ 4. # 1 3
5. $ ab$ #
$ 2 4
} 1 2 42
10/20/24 3
 Search in suffix tree
 Searching for all instances of a substring S in a suffix tree is easy
since any substring of S is the prefix of some suffix.
 Pseudo-code for searching in suffix tree:
• Start at root
• Go down the tree by taking each time the corresponding path
• If S correspond to a node then return all leaves in sub-tree
o The places where S can be found are given by the pointers in all
the leaves in the sub-tree rooted at x.
• If S encountered a NIL pointer before reaching the end, then S is
not in the tree
Example:
 If S = "GO" we take the GO path and return:
GOOGOL$, GOL$.
 If S = "OR" we take the O path and then we hit a NIL pointer so "OR"
is not in the tree. 43
 Drawbacks
 Suffix trees consume a lot of space
• Even if word beginnings are indexed, space overhead of
120% - 240% over the text size is produced. Because
depending on the implementation each nodes of the
suffix tree takes a space (in bytes) equivalent to the
number of symbols used.
• How much space is required at each node for English
word indexing based on alphabets a to z.
 How many bytes required to store MISSISSIPI ?

10/20/24 44
 S ! ?
N K NY
H A E A
T A V
U H
YO
IF
45