What is Information Retrieval (IR)?

Information Retrieval (IR) is a process used by software programs to organize, store, search,
and retrieve information from a collection of documents. These documents can include text,
images, videos, or other types of multimedia.

Think of IR as a smart search system that helps find the most useful information based on
what a user is looking for. When you enter a query (a search request), the system scans
through the stored documents and brings back the most relevant ones. This is done using
indexing and metadata (extra information about the content, like tags or keywords) to make
searching more efficient.

For example, when you search for something on Google, IR helps by finding and ranking web
pages based on how well they match your search terms.

Uses of Information Retrieval (IR)

IR is widely used in many fields to make searching for information easier and faster. Here are
some real-world applications:

1.​ Search Engines (Google, Bing, etc.)

○​ When you type a query into a search engine, IR techniques scan millions of
web pages and bring you the most relevant results.
○​ These search engines use algorithms to rank results based on relevance,
popularity, and context.
2.​ Digital Libraries
○​ Digital libraries store books, research papers, and articles in electronic form.
○​ IR helps users quickly find the right material by searching through a massive
collection of digital documents.
3.​ Enterprise Search (Corporate Data Management)
○​ Large companies store huge amounts of documents, emails, and reports.
○​ IR systems help employees find important files and knowledge without wasting
time.
4.​ E-commerce (Amazon, Flipkart, etc.)
○​ Online shopping websites use IR to help customers find the right products
based on search queries.
○​ When you type "wireless headphones," the system searches its product
database and shows you relevant results.
5.​ Healthcare Information Systems
○​ Doctors and researchers use IR to find medical records, research papers, and
drug information.
 ○​ For example, if a doctor searches for "treatment for diabetes," an IR system can
pull up scientific articles and case studies.
6.​ Legal Research
○​ Lawyers need to refer to past cases, laws, and legal documents to prepare
arguments.
○​ IR systems help them search through large databases of legal information
efficiently.
7.​ Social Media Search (Facebook, Twitter, Instagram, etc.)
○​ Social media platforms store billions of posts, photos, and videos.
○​ IR helps users search for people, posts, and hashtags quickly and accurately.

Information Retrieval (IR) in Natural Language Processing (NLP)

In NLP (Natural Language Processing), Information Retrieval (IR) focuses on searching and
retrieving documents written in natural language (English, Hindi, etc.) based on a user’s
query.

Imagine you type a question in Google like:​

➡ “What are the symptoms of diabetes?”

Google doesn’t generate new answers but retrieves relevant web pages that already contain
this information. This is how an IR system works in NLP.

How IR Systems Work in NLP

●​ IR systems search through a large collection of text documents.

●​ They try to find the documents that best match a user’s question.
●​ The system does not generate new answers but only tells the user where to find
relevant documents.

Key Concept: Relevance in IR

The most important goal of an IR system is to retrieve only relevant documents—those that
contain useful information for the user.

A perfect IR system would return only relevant documents and ignore all unrelated ones.
However, in reality, systems aren’t perfect, so they try to rank documents based on how well
they match the query.

For example, if you search:​

➡ "Best laptops under $1000"

A good IR system should not return results about smartphones or expensive laptops that cost
over $2000.
Steps in Information Retrieval

1.​ User enters a query in natural language.

2.​ The system searches through a collection of documents.
3.​ Relevant documents are ranked based on similarity to the query.
4.​ Results are displayed to the user (like Google’s search results).

This diagram represents the Information Retrieval (IR) process in Natural Language
Processing (NLP). Let's break it down step by step:

Key Components and Flow

1.​ Documents (Corpus)

○​ This represents the entire set of documents available for search.
○​ These documents are processed and transformed into a structured format using
a representation function.
2.​ Representation Function (Indexing)
○​ Converts raw text documents into structured representations (e.g., term
frequency vectors, TF-IDF, word embeddings).
○​ This processed data is stored in an Index, which acts as a database for quick
retrieval.
3.​ Query (User Input)
○​ A user submits a search query (e.g., "Find articles on deep learning").
 ○​ The representation function processes the query into the same structured
format as the indexed documents.
4.​ Matching Function (Retrieval & Ranking)
○​ The query representation is compared with the document representations stored
in the index.
○​ The matching function ranks the documents based on similarity (e.g., cosine
similarity, BM25, deep learning models).
5.​ Results
○​ The most relevant documents are retrieved and displayed to the user as search
results.

IR in NLP Context

●​ Text Representation: Uses NLP techniques like tokenization, stemming, lemmatization,

word embeddings (Word2Vec, BERT).
●​ Indexing: Efficiently structures document information for faster retrieval.
●​ Query Processing: Converts user input into a machine-readable format.
●​ Ranking & Retrieval: Uses traditional (TF-IDF, BM25) or modern deep learning models
(BERT-based retrieval) for ranking.

Main Components of an IR System

An IR system consists of two main processes:

1.​ Indexing – Preparing documents for quick search.

2.​ Matching – Finding and ranking documents based on similarity.

1. Indexing

Indexing is the process of organizing text so that it can be searched quickly.

Steps in Indexing:

1.​ Tokenization – Splitting text into individual words or phrases.

○​ Example:
■​ Sentence: "Artificial Intelligence is amazing!"
■​ Tokens: [‘Artificial’, ‘Intelligence’, ‘is’, ‘amazing’]
2.​ Removing Frequent Words (Stopwords Removal)
○​ Some words appear too often (e.g., is, the, and, of, in) but do not add meaning.
○​ These words are removed to reduce noise in search results.
○​ Example:
■​ Original: "The AI system is intelligent and fast."
■​ After removing stopwords: "AI system intelligent fast."
3.​ Stemming
 ○​ Converts words to their root form to improve matching.
○​ Example:
■​ ‘running’ → ‘run’
■​ ‘jumping’ → ‘jump’
■​ ‘studies’ → ‘study’

Indexing Techniques:

●​ Boolean Model
○​ Uses AND, OR, NOT to filter results.
○​ Example:
■​ Query: "Machine Learning AND Deep Learning"
■​ Returns only documents that contain both terms.
●​ Vector Space Model
○​ Represents documents as mathematical vectors to measure similarity.
○​ Used in TF-IDF ranking (explained below).

2. Matching

Matching is the process of finding how similar a document is to a query.

To do this, IR systems use mathematical formulas to measure similarity.

TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF is a common technique used to determine how important a word is in a document.

Formula:

Let’s break this formula into two parts:

1. Term Frequency (TF)

TF tells us how many times a word appears in a document.

Example:
Imagine we have this document:​
➡ "AI is the future. AI is powerful."

●​ Total words in the document = 6

●​ "AI" appears 2 times
●​ TF(AI) = 2/6 = 0.33

This means "AI" is an important word in this document.

2. Inverse Document Frequency (IDF)

IDF checks how rare a word is across all documents.

​
Why is TF-IDF Important?

●​ Helps rank search results

●​ Filters out common words
●​ Highlights important keywords
Example: How Google Search Uses IR & TF-IDF

➡ Query: "Best programming language for AI"

●​ Google scans millions of documents.

●​ Uses Indexing (tokenization, stemming, stopword removal).
●​ Matching is done using TF-IDF and other algorithms.
●​ Documents with high TF-IDF scores for "programming language" and "AI" will appear
higher in search result.

Information Retrieval (IR) Process

Information Retrieval (IR) is the process of finding relevant information from a large collection of
unstructured data (e.g., text documents, web pages) based on a user’s query.

Step-by-Step Procedure of an IR System

1. Indexing the Collection of Documents

Before searching, the system needs to prepare and organize documents efficiently. This step is
called indexing and involves:

●​ Tokenization: Breaking text into words or phrases.

●​ Stopword Removal: Removing common words like "is", "the", "and".
●​ Stemming/Lemmatization: Reducing words to their root forms (e.g., "running" → "run").
●​ TF-IDF Calculation: Measuring the importance of words in documents.
●​ Inverted Index Creation: A mapping of words to the documents where they appear.

👉 Example:​
If we have three documents:

1.​ D1: "Information retrieval is important."

2.​ D2: "Retrieval techniques involve indexing."
3.​ D3: "Indexing helps in fast search."

The system creates an inverted index like this:

 ​
Now, when a user searches for "retrieval techniques", the system can quickly find D1 and D2.

2. Query Processing

The user submits a query (e.g., "fast indexing"), which must be processed similarly to the
documents:

●​ Tokenization
●​ Stopword removal
●​ Stemming/Lemmatization
●​ Conversion to a vector representation (like TF-IDF)

👉 Example: Query: "fast retrieval"

●​ Stopword removal → ["fast", "retrieval"]
●​ Stemming → ["fast", "retrieve"]
●​ Convert to vector form.

3. Matching (Comparing Query with Documents)

Now, the system compares the transformed query with the indexed documents.​
This is done using similarity measures like:

●​ Cosine Similarity (for vector-based models)

●​ BM25 (for ranking documents)

4. Ranking & Retrieval

●​ The documents are ranked based on their similarity to the query.

●​ The system retrieves the most relevant documents and displays them to the user.

👉 Example:​
If the user searches for "retrieval techniques", the system ranks the documents like:
D2 is ranked highest because it contains both retrieval and techniques.

Vector Space Model (VSM) of Retrieval

The Vector Space Model (VSM) is a mathematical representation of documents and queries as
vectors in high-dimensional space. It allows similarity calculations between queries and
documents.

How It Works

1.​ Each document and query is represented as a vector in an n-dimensional space, where
each dimension is a unique word (term).
2.​ The importance of each word in a document is calculated using TF-IDF (Term
Frequency - Inverse Document Frequency).
3.​ The similarity between a query and a document is measured using cosine similarity.

TF-IDF Calculation (Weighting Terms)

Each word in a document is assigned a weight using TF-IDF, which is calculated as:

Where:

●​ TF (Term Frequency) = Number of times a word appears in a document.

●​ IDF (Inverse Document Frequency) = Measures how rare the word is across all
documents.

👉 Example: Let’s say we have the following three documents:

●​ D1: "machine learning and deep learning"
●​ D2: "machine learning algorithms"
●​ D3: "deep learning and AI"
Cosine Similarity (Comparing Query and Document)

To measure similarity, we calculate the cosine of the angle between query and document
vectors:

​
Where:

●​ D1⋅QD_1 \cdot QD1​⋅Q = Dot product of document and query vectors.

●​ ∣∣D1∣∣||D_1||∣∣D1​∣∣ = Magnitude of the document vector.
●​ ∣∣Q∣∣||Q||∣∣Q∣∣ = Magnitude of the query vector.

👉 Example: If the query is "deep learning", we compute cosine similarity with each
document.

●​ D1: Cosine similarity = 0.85

●​ D2: Cosine similarity = 0.30
●​ D3: Cosine similarity = 0.95

So, D3 is most relevant!

What is Term Weighting?

Term weighting is the process of assigning numerical values to words in a document or corpus
to reflect their importance in information retrieval (IR). The higher the weight of a term, the
greater its impact on document retrieval and ranking.

Importance of Term Weighting

1.​ Helps distinguish important terms from less significant ones.

2.​ Improves document ranking and relevance in IR systems.
3.​ Enhances search accuracy in text-based applications like search engines and NLP
tasks.
Common Word Statistics / Term Weighting Methods

1. Term Frequency (TF)

Definition: Measures how often a term appears in a document.

Formula:

Example: Consider a document: "Information retrieval is the process of retrieving information

from a database."

●​ TF for "information" = 2 / 10 = 0.2

●​ TF for "retrieval" = 1 / 10 = 0.1
●​ TF for "database" = 1 / 10 = 0.1

2. Document Frequency (DF)

Definition: Measures how many documents in a corpus contain a particular term.

Formula:

Example: Given a corpus with 3 documents:

1.​ "Data structures and retrieval are important in CS."

2.​ "Information retrieval focuses on fetching relevant data."
3.​ "Storage and retrieval techniques enhance performance."
●​ DF for "retrieval" = 3 (appears in all documents)
●​ DF for "data" = 2 (appears in documents 1 & 2)

3. Inverse Document Frequency (IDF)

Definition: Measures the importance of a term across multiple documents. Rare terms have
higher IDF values.

Formula:

where:

●​ NNN = total number of documents

●​ DF(t)DF(t)DF(t) = document frequency of term ttt
Example: Using the previous example where N = 3:

●​ IDF for "retrieval" = log(3/3) = log(1) = 0 (common word)

●​ IDF for "data" = log(3/2) ≈ 0.176

Rare terms get higher IDF scores.

4. TF-IDF (Term Frequency-Inverse Document Frequency)

Definition: A combination of TF and IDF that balances word frequency and uniqueness.

Formula:

Example: For "retrieval" in Document 1:

●​ TF = 1/7 ≈ 0.142
●​ IDF = log(3/3) = 0
●​ TF-IDF = 0.142 × 0 = 0 (common word, not important)

For "data" in Document 2:

●​ TF = 1/6 ≈ 0.167
●​ IDF = 0.176
●​ TF-IDF = 0.167 × 0.176 ≈ 0.029 (relatively more important)

5. TF-IDF with Document Length Normalization (TF-IDF-DLN)

●​ Accounts for document length variations.

●​ Normalizes TF-IDF scores so longer documents don't dominate.

Formula:

This ensures that documents of different lengths contribute fairly.

6. Word Frequency Distribution

Definition: A histogram showing how frequently words appear in a corpus.

Example: Given a document:​

"AI and ML are important in AI applications."

●​ "AI" appears twice → High frequency

●​ "and" appears once → Low frequency
Use Case: Helps identify stop words (common words like "and", "the") to ignore in search.

7. Zipf’s Law

Definition: The frequency of a word is inversely proportional to its rank in the corpus.

Formula:

where:

●​ f = frequency of a word
●​ r = rank of the word in terms of frequency

Observation: The most frequent term occurs twice as often as the second-most, three times
as often as the third-most, and so on.

Text Preprocessing

Text preprocessing involves cleaning and transforming raw text data into a format suitable for
analysis and modeling. It improves the quality and efficiency of downstream NLP tasks by
addressing noise, inconsistency, and irrelevant information.

Common Techniques in Text Preprocessing

●​ Lowercasing: Converting text to lowercase standardizes the text and reduces

vocabulary size.
●​ Tokenization: Splitting text into smaller units (tokens), such as words or phrases, to
facilitate further analysis.
●​ Removing Punctuation: Eliminating punctuation marks simplifies text and reduces
noise.
●​ Stemming and Lemmatization:
○​ Stemming: Reduces words to their base form by removing prefixes and suffixes.
 ○​ Lemmatization: Maps words to their dictionary form (lemma), improving
coherence.
●​ Removing Numbers and Special Characters: Eliminating numerical digits and
symbols focuses on textual content.
●​ Handling HTML Tags and URLs: Extracting only text by removing markup elements
and hyperlinks.
●​ Handling Contractions and Abbreviations: Expanding contractions (e.g., “can’t” →
“cannot”) enhances consistency.
●​ Spell Checking and Correction: Detecting and fixing spelling errors improves analysis
quality.
●​ Text Normalization: Standardizing spellings and variations to achieve consistency and
reduce vocabulary size.

Indexing

Indexing in information retrieval involves creating data structures that efficiently store and
retrieve documents based on content or metadata. It represents documents as vectors in a
high-dimensional space, where each dimension corresponds to a unique term.

●​ TF-IDF-based Indexing: Word statistics such as TF-IDF scores construct document

vectors, with TF-IDF values serving as term weights.
●​ Efficient Retrieval: Indexing enables quick identification of relevant documents by
analyzing term distributions.

Query Processing

Query processing ensures accuracy by retrieving relevant documents based on user queries.
The effectiveness of an IR system depends on how well the query is formulated.

Key Aspects of Query Processing

●​ Matching Query Terms with Indexed Documents: The system identifies relevant
documents by comparing indexed terms with query terms.
●​ Term Weighting in Query Ranking:
○​ Terms with high Inverse Document Frequency (IDF) contribute significantly to
relevance.
○​ Rare terms in the corpus but frequent in the query indicate high relevance.

Relevance Feedback

Relevance feedback is an interactive technique in IR systems where users provide feedback on

retrieved documents to refine search results iteratively.

Types of Relevance Feedback

1.​ Implicit Feedback:
○​ Inferred from user behavior (e.g., dwell time, scrolling, document selection).
○​ Example: The longer a user spends on a document, the more relevant it is
assumed to be.
2.​ Explicit Feedback:
○​ Direct user assessment of document relevance.
○​ Relevance Systems:
■​ Binary Relevance System: A document is either relevant (1) or
irrelevant (0).
■​ Graded Relevance System: Documents are rated on a scale (e.g., "not
relevant," "somewhat relevant," "very relevant").
3.​ Pseudo Feedback (Blind Feedback):
○​ Automates relevance feedback without user interaction.
○​ Enhances retrieval performance by assuming the top-ranked documents are
relevant.