Search Engines

Information Retrieval in Practice

All slides ©Addison Wesley, 2008

Search Engine Architecture
• A software architecture consists of software components, the
interfaces provided by those components, and the
relationships between them
– describes a system at a particular level of abstraction
• Architecture of a search engine determined by 2 requirements
– effectiveness (quality of results) and efficiency (response time and
throughput)
Legacy N-Tier Architecture
• Presentation Logic
– Only for the user
• Business Logic
• Data Access Logic
– Normally DB
– Search query communicates with DB
N-Tier Web Architecture
Latest N-Tier Architecture
• Presentation Logic
– Both for the user and the author
• Business Logic
• Data Access Logic
– Both for DB
• Search Access Logic
– Connects Search Engine that search query communicates with
N-Tier Web Architecture with Enterprise Search Engine
Indexing Process
Indexing Process
• Text acquisition
– identifies and stores documents for indexing
• Text transformation
– transforms documents into index terms or features
• Index creation
– takes index terms and creates data structures (indexes) to support
fast searching
Query Process
Query Process
• User interaction
– supports creation and refinement of query, display of results
• Ranking
– uses query and indexes to generate ranked list of documents
• Evaluation
– monitors and measures effectiveness and efficiency (primarily offline)
Details: Text Acquisition
• Crawler
– Identifies and acquires documents for search engine
– Many types – web, enterprise, desktop
– Web crawlers follow links to find documents
• Must efficiently find huge numbers of web pages (coverage) and keep them up-
to-date (freshness)
• Single site crawlers for site search
• Topical or focused crawlers for vertical search
– Document crawlers for enterprise and desktop search
• Follow links and scan directories
Text Acquisition
• Feeds
– Real-time streams of documents
• e.g., web feeds for news, blogs, video, radio, tv
– RSS is common standard
• RSS “reader” can provide new XML documents to search engine
• Conversion
– Convert variety of documents into a consistent text plus metadata
format
• e.g. HTML, XML, Word, PDF, etc. → XML
– Convert text encoding for different languages
• Using a Unicode standard like UTF-8
Text Acquisition
• Document data store
– Stores text, metadata, and other related content for documents
• Metadata is information about document such as type and creation date
• Other content includes links, anchor text
– Provides fast access to document contents for search engine
components
• e.g. result list generation
– Could use relational database system
• More typically, a simpler, more efficient storage system is used due to huge
numbers of documents
Text Transformation
• Parser
– Processing the sequence of text tokens in the document to recognize
structural elements
• e.g., titles, links, headings, etc.
– Tokenizer recognizes “words” in the text
• must consider issues like capitalization, hyphens, apostrophes, non-alpha
characters, separators
– Markup languages such as HTML, XML often used to specify structure
• Tags used to specify document elements
– E.g., <h2> Overview </h2>
• Document parser uses syntax of markup language (or other formatting) to
identify structure
Text Transformation
• Stopping
– Remove common words
• e.g., “and”, “or”, “the”, “in”
– Some impact on efficiency and effectiveness
– Can be a problem for some queries
• Stemming
– Group words derived from a common stem
• e.g., “computer”, “computers”, “computing”, “compute”
– Usually effective, but not for all queries
– Benefits vary for different languages
Text Transformation
• Link Analysis
– Makes use of links and anchor text in web pages
– Link analysis identifies popularity and community information
• e.g., PageRank
– Anchor text can significantly enhance the representation of pages
pointed to by links
– Significant impact on web search
• Less importance in other applications
Text Transformation
• Information Extraction
– Identify classes of index terms that are important for some
applications
– e.g., named entity recognizers identify classes such as people,
locations, companies, dates, etc.
• Classifier
– Identifies class-related metadata for documents
• i.e., assigns labels to documents
• e.g., topics, reading levels, sentiment, genre
– Use depends on application
Index Creation
• Document Statistics
– Gathers counts and positions of words and other features
– Used in ranking algorithm
• Weighting
– Computes weights for index terms
– Used in ranking algorithm
– e.g., tf.idf weight
• Combination of term frequency in document and inverse document frequency
in the collection
Index Creation
• Inversion
– Core of indexing process
– Converts document-term information to term-document for
indexing
• Difficult for very large numbers of documents
– Format of inverted file is designed for fast query processing
• Must also handle updates
• Compression used for efficiency
Index Creation
• Index Distribution
– Distributes indexes across multiple computers and/or multiple sites
– Essential for fast query processing with large numbers of documents
– Many variations
• Document distribution, term distribution, replication
– P2P and distributed IR involve search across multiple sites
 Sec. 1.1

Term-document incidence matrices

Brutus AND Caesar BUT NOT Calpurnia 1 if play contains

word, 0 otherwise
 Sec. 1.1

Incidence vectors
• So we have a 0/1 vector for each term.
• To answer query: take the vectors for Brutus, Caesar and
Calpurnia (complemented)  bitwise AND.
– 110100 AND
– 110111 AND
– 101111 =
– 100100

22
 Sec. 1.2

Inverted index
• For each term t, we must store a list of all documents that
contain t.
– Identify each doc by a docID, a document serial number
• Can we used fixed-size arrays for this?
Brutus 1 2 4 11 31 45 173 174

Caesar 1 2 4 5 6 16 57 132

Calpurnia 2 31 54 101

What happens if the word Caesar is added to document 14?

23
 Sec. 1.2

Inverted index
• We need variable-size postings lists
– On disk, a continuous run of postings is normal and best
– In memory, can use linked lists or variable length arrays
• Some tradeoffs in size/ease of insertion
Posting

Brutus 1 2 4 11 31 45 173 174

Caesar 1 2 4 5 6 16 57 132
Calpurnia 2 31 54 101

Dictionary Postings
Sorted by docID (more later on why).
24 24
 Sec. 1.2

Inverted index construction

Documents to Friends, Romans, countrymen.
be indexed

Tokenize
r
Token stream Friends Romans Countrymen

Linguistic modules

Modified tokens friend roman countryman

Indexe
r
friend 2 4
roman 1 2
Inverted index
countryman 13 16
User Interaction
• Query input
– Provides interface and parser for query language
– Most web queries are very simple, other applications may use forms
– Query language used to describe more complex queries and results
of query transformation
• e.g., Boolean queries, Indri and Galago query languages
• similar to SQL language used in database applications
• IR query languages also allow content and structure specifications, but focus
on content
User Interaction
• Query transformation
– Improves initial query, both before and after initial search
– Includes text transformation techniques used for documents
– Spell checking and query suggestion provide alternatives to original
query
– Query expansion and relevance feedback modify the original query
with additional terms
User Interaction
• Results output
– Constructs the display of ranked documents for a query
– Generates snippets to show how queries match documents
– Highlights important words and passages
– Retrieves appropriate advertising in many applications
– May provide clustering and other visualization tools
 Sec. 19.2.2

The trouble with paid search ads …

• It costs money. What’s the alternative?
• Search Engine Optimization:
– “Tuning” your web page to rank highly in the algorithmic search
results for select keywords
– Alternative to paying for placement
– Thus, intrinsically a marketing function
• Performed by companies, webmasters and consultants
(“Search engine optimizers”) for their clients
• Some perfectly legitimate, some very shady
 Sec. 19.2.2

Search engine optimization (Spam)

• Motives
– Commercial, political, religious, lobbies
– Promotion funded by advertising budget
• Operators
– Contractors (Search Engine Optimizers) for lobbies, companies
– Web masters
– Hosting services
• Forums
– E.g., Web master world ( www.webmasterworld.com )
• Search engine specific tricks
 Sec. 19.2.2

Simplest forms
• First generation engines relied heavily on tf/idf
– The top-ranked pages for the query maui resort were the ones containing the most maui’s
and resort’s
• SEOs responded with dense repetitions of chosen terms
– e.g., maui resort maui resort maui resort
– Often, the repetitions would be in the same color as the background of the web page
• Repeated terms got indexed by crawlers
• But not visible to humans on browsers

Pure word density

cannot
be trusted as an IR
signal
 Sec. 19.2.2

Variants of keyword stuffing

• Misleading meta-tags, excessive repetition
• Hidden text with colors, style sheet tricks, etc.

Meta-Tags =
“… London hotels, hotel, holiday inn, hilton,
discount, booking, reservation, sex, mp3, britney
spears, viagra, …”
Ranking
• Scoring
– Calculates scores for documents using a ranking algorithm
– Core component of search engine
– Basic form of score is  qi di
• qi and di are query and document term weights for term i
– Many variations of ranking algorithms and retrieval models
Ranking
• Performance optimization
– Designing ranking algorithms for efficient processing
• Term-at-a time vs. document-at-a-time processing
• Safe vs. unsafe optimizations
• Distribution
– Processing queries in a distributed environment
– Query broker distributes queries and assembles results
– Caching is a form of distributed searching
 Sec. 19.5

What is the size of the web ?

• Issues
– The web is really infinite
• Dynamic content, e.g., calendar
– Static web contains syntactic duplication, mostly due to mirroring
(~30%)
– Some servers are seldom connected
• Who cares?
– Media, and consequently, the user
– Engine design
– Engine crawl policy. Impact on recall.
What can we attempt to measure?
•The relative sizes of search engines
– The notion of a page being indexed is still reasonably well defined.
– Already there are problems
• Document extension: e.g. engines index pages not yet crawled, by indexing
anchortext.
• Document restriction: All engines restrict what is indexed (first n words, only
relevant words, etc.)
•The coverage of a search engine relative to another particular
crawling process.
 Sec. 19.5

New definition?
– The statically indexable web is whatever search engines index.
• Different engines have different preferences
– max url depth, max count/host, anti-spam rules, priority rules, etc.
• Different engines index different things under the same URL:
– frames, meta-keywords, document restrictions, document
extensions, ...
Evaluation
• Logging
– Logging user queries and interaction is crucial for improving search
effectiveness and efficiency
– Query logs and clickthrough data used for query suggestion, spell
checking, query caching, ranking, advertising search, and other
components
• Ranking analysis
– Measuring and tuning ranking effectiveness
• Performance analysis
– Measuring and tuning system efficiency
 Sec. 19.6

Duplicate documents
• The web is full of duplicated content
• Strict duplicate detection = exact match
– Not as common
• But many, many cases of near duplicates
– E.g., Last modified date the only difference between two copies of a
page
 Sec. 19.6

Duplicate/Near-Duplicate Detection

• Duplication: Exact match can be detected with fingerprints

• Near-Duplication: Approximate match
– Overview
• Compute syntactic similarity with an edit-distance measure
• Use similarity threshold to detect near-duplicates
– E.g., Similarity > 80% => Documents are “near duplicates”
– Not transitive though sometimes used transitively
 Sec. 19.6

Computing Similarity
• Features:
– Segments of a document (natural or artificial breakpoints)
– Shingles (Word N-Grams)
– a rose is a rose is a rose →
a_rose_is_a
rose_is_a_rose
is_a_rose_is
a_rose_is_a
• Similarity Measure between two docs (= sets of shingles)
– Set intersection
– Specifically (Size_of_Intersection / Size_of_Union)
 Sec. 19.6

Shingles + Set Intersection

• Computingexact set intersection of shingles between all pairs of
documents is expensive/intractable
– Approximate using a cleverly chosen subset of shingles from each (a
sketch)
• Estimate (size_of_intersection / size_of_union) based on a short
sketch

Doc
Doc Shingle set Sketch
AA A A
Jaccard
Doc
Doc Shingle set Sketch
BB B B
 Sec. 19.6

Set Similarity of sets Ci , Cj

Ci C j
Jaccard(Ci , C j ) 
Ci C j
• View sets as columns of a matrix A; one row for each element in the
universe. aij = 1 indicates presence of item i in set j
• Example
C1 C2

0 1
1 0
1 1 Jaccard(C1,C2) = 2/5 = 0.4
0 0
1 1
 Sec. 19.6

Key Observation
• For columns Ci, Cj, four types of rows
Ci Cj
A 1 1
B 1 0
C 0 1
D 0 0
• Overload notation: A = # of rows of type A
A
• Claim Jaccard(Ci , C j ) 
A BC
How Does It Really Work?
• This course explains these components of a search engine in more
detail
• Often many possible approaches and techniques for a given
component
– Focus is on the most important alternatives
– i.e., explain a small number of approaches in detail rather than many
approaches
– “Importance” based on research results and use in actual search engines
– Alternatives described in references