Introduction to Information Retrieval

Module 4

Boolean Retrieval Models

Introduction to Information Retrieval

• Boolean Model – Term Vocabulary and Posting Lists – Dictionaries and Tolerant
Retrieval – Index Construction – Index Compression – Vector Space Model – Scoring
and Term Weighting – Evaluation in Information Retrieval
Introduction to Information Retrieval

Outline

❶ Introduction

❷ Inverted index

❸ Processing Boolean queries

❹ Query optimization
Introduction to Information Retrieval

Information Retrieval

• Information Retrieval (IR) is finding material (usually

documents) of an unstructured nature (usually text) that
satisfies an information need from within large collections
(usually stored on computers).

– These days we frequently think first of web search, but there

are many other cases:
• E-mail search
• Searching your laptop
• Corporate knowledge bases
• Legal information retrieval
Introduction to Information Retrieval

Sec. 1.1
Basic assumptions of Information Retrieval

• Collection: A set of documents

– Assume it is a static collection for the moment

• Goal: Retrieve documents with information that is

relevant to the user’s information need and helps
the user complete a task
Introduction to Information Retrieval

6
Introduction to Information Retrieval

7
Introduction to Information Retrieval

Boolean retrieval

 The Boolean model is arguably the simplest model to base

an information retrieval system on.
 Queries are Boolean expressions, e.g., CAESAR AND BRUTUS
 The seach engine returns all documents that satisfy the
 Boolean expression.

Does Google use the Boolean model?

8
Introduction to Information Retrieval

Outline

❶ Introduction

❷ Inverted index

❸ Processing Boolean queries

❹ Query optimization
Introduction to Information Retrieval

Unstructured data in 1650: Shakespeare

10
Introduction to Information Retrieval

Unstructured data in 1650

 Which plays of Shakespeare contain the words BRUTUS AND

CAESAR, but not CALPURNIA?
 One could grep all of Shakespeare’s plays for BRUTUS and
CAESAR, then strip out lines containing CALPURNIA
 Why is grep not the solution?
 Slow (for large collections)
 grep is line-oriented, IR is document-oriented
 “NOT CALPURNIA” is non-trivial
 Other operations (e.g., find the word ROMANS near
COUNTRYMAN ) not feasible

11
Introduction to Information Retrieval

Term-document incidence matrix

Anthony Julius The Hamlet Othello Macbeth
and Caesar Tempest ...
Cleopatra
ANTHONY 1 1 0 0 0 1
BRUTUS 1 1 0 1 0 0
CAESAR 1 1 0 1 1 1
CALPURNIA 0 1 0 0 0 0
CLEOPATRA 1 0 0 0 0 0
MERCY 1 0 1 1 1 1
WORSER 1 0 1 1 1 0
...

Entry is 1 if term occurs. Example: CALPURNIA occurs in Julius Caesar.

Entry is 0 if term doesn’t occur. Example: CALPURNIA
doesn’t occur in The tempest.
12
Introduction to Information Retrieval

Incidence vectors

 So we have a 0/1 vector for each term.

 To answer the query BRUTUS AND CAESAR AND NOT CALPURNIA:
 Take the vectors for BRUTUS, CAESAR AND NOT CALPURNIA
 Complement the vector of CALPURNIA
 Do a (bitwise) and on the three vectors
 110100 AND 110111 AND 101111 = 100100

13
Introduction to Information Retrieval

0/1 vector for BRUTUS

Anthony Julius The Hamlet Othello Macbeth
and Caesar Tempest ...
Cleopatra
ANTHONY 1 1 0 0 0 1
BRUTUS 1 1 0 1 0 0
CAESAR 1 1 0 1 1 1
CALPURNIA 0 1 0 0 0 0
CLEOPATRA 1 0 0 0 0 0
MERCY 1 0 1 1 1 1
WORSER 1 0 1 1 1 0
...
result: 1 0 0 1 0 0

14
Introduction to Information Retrieval

Answers to query

Anthony and Cleopatra, Act III, Scene ii

Agrippa [Aside to Domitius Enobarbus]: Why, Enobarbus,
When Antony found Julius Caesar dead,
He cried almost to roaring; and he wept
When at Philippi he found Brutus slain.
Hamlet, Act III, Scene ii
Lord Polonius: I did enact Julius Caesar: I was killed i’
the Capitol; Brutus killed me.

15
Introduction to Information Retrieval

Bigger collections

 Consider N = 106 documents, each with about 1000 tokens

 total of 109 tokens
 On average 6 bytes per token, including spaces and
 punctuation size of document collection is about 6 ・ 109 =
6 GB
 Assume there are M = 500,000 distinct terms in the
collection
 (Notice that we are making a term/token distinction.)

16
Introduction to Information Retrieval

Can’t build the incidence matrix

 M = 500,000 × 106 = half a trillion 0s and 1s.

 But the matrix has no more than one billion 1s.
 Matrix is extremely sparse.
 What is a better representations?
 We only record the 1s.

17
Introduction to Information Retrieval

Inverted Index

For each term t, we store a list of all documents that contain t.

dictionary postings
18
Introduction to Information Retrieval

Inverted Index

For each term t, we store a list of all documents that contain t.

dictionary postings
19
Introduction to Information Retrieval

Inverted Index

For each term t, we store a list of all documents that contain t.

dictionary postings
20
Introduction to Information Retrieval

Inverted index construction

❶ Collect the documents to be indexed:

❷ Tokenize the text, turning each document into a list of tokens:

❸ Do linguistic preprocessing, producing a list of normalized

tokens, which are the indexing terms:

❹ Index the documents that each term occurs in by creating an

inverted index, consisting of a dictionary and postings.
Introduction to Information Retrieval

Tokenizing and preprocessing

22
Introduction to Information Retrieval

Generate posting

23
Introduction to Information Retrieval

Sort postings

24
Introduction to Information Retrieval

Create postings lists, determine document frequency

25
Introduction to Information Retrieval

Split the result into dictionary and postings file

dictionary postings
26
Introduction to Information Retrieval

Later in this course

 Index construction: how can we create inverted indexes for

large collections?
 How much space do we need for dictionary and index?
 Index compression: how can we efficiently store and process
indexes for large collections?
 Ranked retrieval: what does the inverted index look like
when we want the “best” answer?

27
Introduction to Information Retrieval

Outline

❶ Introduction

❷ Inverted index

❸ Processing Boolean queries

❹ Query optimization
Introduction to Information Retrieval

Simple conjunctive query (two terms)

 Consider the query: BRUTUS AND CALPURNIA

 To find all matching documents using inverted index:
❶ Locate BRUTUS in the dictionary
❷ Retrieve its postings list from the postings file
❸ Locate CALPURNIA in the dictionary
❹ Retrieve its postings list from the postings file
❺ Intersect the two postings lists
❻ Return intersection to user

29
Introduction to Information Retrieval

Intersecting two posting lists

 This is linear in the length of the postings lists.

 Note: This only works if postings lists are sorted.

30
Introduction to Information Retrieval

Intersecting two posting lists

31
Introduction to Information Retrieval

Query processing: Exercise

Compute hit list for ((paris AND NOT france) OR lear)

32
Introduction to Information Retrieval

Boolean queries
 The Boolean retrieval model can answer any query that is a
Boolean expression.
 Boolean queries are queries that use AND, OR and NOT to join
 query terms.
 Views each document as a set of terms.
 Is precise: Document matches condition or not.
 Primary commercial retrieval tool for 3 decades
 Many professional searchers (e.g., lawyers) still like Boolean
queries.
 You know exactly what you are getting.
 Many search systems you use are also Boolean: spotlight,
email, intranet etc.
33
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval

Commercially successful Boolean retrieval: Westlaw

 Largest commercial legal search service in terms of the
number of paying subscribers
 Over half a million subscribers performing millions of
searches a day over tens of terabytes of text data
 The service was started in 1975.
 In 2005, Boolean search (called “Terms and Connectors” by
Westlaw) was still the default, and used by a large
percentage of users . . .
 . . . although ranked retrieval has been available since 1992.

41
Introduction to Information Retrieval

Westlaw: Example queries

Information need: Information on the legal theories involved in
preventing the disclosure of trade secrets by employees formerly
employed by a competing company Query: “trade secret” /s
disclos! /s prevent /s employe! Information need: Requirements

for disabled people to be able to access a workplace Query:

disab! /p access! /s work-site work-place (employment /3 place)

Information need: Cases about a host’s responsibility for drunk

guests Query: host! /p (responsib! liab!) /p (intoxicat! drunk!)
/p guest

42
Introduction to Information Retrieval

Westlaw: Comments
 Proximity operators: /3 = within 3 words, /s = within a
sentence, /p = within a paragraph
 Space is disjunction, not conjunction! (This was the default
in search pre-Google.)
 Long, precise queries: incrementally developed, not like
web search
 Why professional searchers often like Boolean search:
precision, transparency, control
 When are Boolean queries the best way of searching?
Depends on: information need, searcher, document
collection, . . .

43
Introduction to Information Retrieval

Outline

❶ Introduction

❷ Inverted index

❸ Processing Boolean queries

❹ Query optimization
Introduction to Information Retrieval

Query optimization

 Consider a query that is an and of n terms, n > 2

 For each of the terms, get its postings list, then and them
together
 Example query: BRUTUS AND CALPURNIA AND CAESAR
 What is the best order for processing this query?

45
Introduction to Information Retrieval

Query optimization

 Example query: BRUTUS AND CALPURNIA AND CAESAR

 Simple and effective optimization: Process in order of
increasing frequency
 Start with the shortest postings list, then keep cutting further
 In this example, first CAESAR, then CALPURNIA, then BRUTUS

46
Introduction to Information Retrieval

Optimized intersection algorithm for

conjunctive queries

47
Introduction to Information Retrieval

More general optimization

 Example query: (MADDING OR CROWD) and (IGNOBLE OR STRIFE)

 Get frequencies for all terms
 Estimate the size of each or by the sum of its frequencies
(conservative)
 Process in increasing order of or sizes

48
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval