Web Search Engines

Popular Search Engines & types

Google Search by keywords

Alta Vista

Bing

Yahoo Search by categories

Ask jeeves Interview simulation

Without search engines, the web wouldn’t
work
• Without search, content is hard to find.
• Without search, there is no incentive to create content.
– Why publish something if nobody will read it?
– Why publish something if I don’t get ad revenue from it?
• Interest aggregation
– Unique feature of the web: A small number of geographically
dispersed people with similar interests can find each other
• Somebody needs to pay for the web.
– Servers, web infrastructure, content creation
– A large part today is paid by search ads.
 Issues with web search engines
• Dynamic data
• Quality is variable & user has to make
judgement
• Factual knowledge is not objective
• Scope of web is not fixed
Structure of the web

Bow-tie structure of web

 Bow-tie structure of web

Strongly connected component (SCC) in the center

 Bow-tie structure of web

Lots of pages that get linked to, but don’t link (OUT)
Lots of pages that link to other pages, but don’t get linked to (IN)
 Bow-tie structure of web

Tendrils: that either lead nowhere from IN, or from nowhere to OUT.
Tubes: small sets of pages outside SCC that lead directly from IN to OUT
 Users
• Web queries are short
Web search architecture (Components)
 Search Index (indexer)
• Inverted index
chess → [www.chess.co.uk, www.chessclub.com, www.uschess.org]

• Information of hyperlinks in link database

– Organized like inverted index
– Source URL contains all destination URLs
 Query Engine
• Algorithmic heart
• Interface between search index, the user and
the web
• Two steps:
– Retrieves the results as per matching keywords
– Ranking the web pages
 Search Interface
• Provides look and feel of search engine
• Allows user to submit queries
• Browse result list
• Click on chosen web page
• User should be able to differentiate between
sponsored links and organic links
 Paid
Search Ads

Algorithmic results.
 Sec. 19.4.1

User Needs
• Need [Brod02, RL04]
– Informational – want to learn about something (~40% / 65%)
Low hemoglobin
– Navigational – want to go to that page (~25% / 15%)
United Airlines
– Transactional – want to do something (web-mediated) (~35% / 20%)
• Access a service Seattle weather
• Downloads Mars surface images
• Shop Canon S410
– Gray areas
• Find a good hub Car rental Brasil
• Exploratory search “see what’s there”
 How far do people look for results?

(Source: iprospect.com WhitePaper_2006_SearchEngineUserBehavior.pdf)

 Users’ empirical evaluation of results
• Quality of pages varies widely
– Relevance is not enough
– Other desirable qualities (non IR!!)
• Content: Trustworthy, diverse, non-duplicated, well maintained
• Web readability: display correctly & fast
• No annoyances: pop-ups, etc
• Precision vs. recall
– On the web, recall seldom matters
• What matters
– Precision at 1? Precision within top-K?
– Comprehensiveness – must be able to deal with obscure queries
• Recall matters when the number of matches is very small
• User perceptions may be unscientific, but are significant
over a large aggregate
 Users’ empirical evaluation of engines

• Relevance and validity of results

• UI – Simple, no clutter, error tolerant
• Trust – Results are objective
• Coverage of topics for polysemic queries
• Pre/Post process tools provided
– Mitigate user errors (auto spell check, search assist,…)
– Explicit: Search within results, more like this, refine ...
– Anticipative: related searches, instant searches (next
slide)
• Impact on stemming, spell-check, etc
– Web addresses typed in the search box
 Sec. 19.2

The Web document collection

• No design/co-ordination
• Distributed content creation, linking,
democratization of publishing
• Content includes truth, lies, obsolete
information, contradictions …
• Unstructured (text, html, …), semi-
structured (XML, annotated photos),
structured (Databases)…
• Scale much larger than previous text
collections … but corporate records are
catching up
• Growth – slowed down from initial
The Web
“volume doubling every few months” but
still expanding
• Content can be dynamically generated
 Sec. 19.2.2

The trouble with paid search ads …

• It costs money. What’s the alternative?

• Search Engine Optimization (SEO):
– “Tuning” your web page to rank highly in the
algorithmic search results for select keywords
– Alternative to paying for placement
– Thus, intrinsically a marketing function
• Performed by companies, webmasters and
consultants (“Search engine optimizers”) for
their clients
• Some perfectly legitimate, some very shady
 Sec. 19.2.2

Search engine optimization (Spam)

• Motives
– Commercial, political, religious, lobbies
– Promotion funded by advertising budget
• Operators
– Contractors (Search Engine Optimizers) for lobbies,
companies
– Web masters
– Hosting services
• Forums
– E.g., Web master world ( www.webmasterworld.com )
 Sec. 19.2.2

Simplest forms: Keyword Stuffing

• First generation engines relied heavily on tf/idf
– The top-ranked pages for the query maui resort were
the ones containing the most maui’s and resort’s
• SEOs -- dense repetitions of chosen terms
– e.g., maui resort maui resort maui resort
– Often, the repetitions would be in the same color as the
background of the web page
• Repeated terms got indexed by crawlers
• But not visible to humans on browsers

Pure word density cannot

be trusted as an IR signal
 The war against spam
• Quality signals - Prefer
authoritative pages based
• Spam recognition by
on: machine learning
– Votes from authors (linkage – Training set based on known
signals) spam
– Votes from users (usage signals)
• Family friendly filters
• Policing of URL submissions – Linguistic analysis, general
– Anti robot test classification techniques, etc.
– For images: flesh tone
• Limits on meta-keywords detectors, source text analysis,
etc.
• Robust link analysis
– Ignore statistically implausible • Editorial intervention
linkage (or text) – Blacklists
– Use link analysis to detect – Top queries audited
spammers (guilt by association) – Complaints addressed
– Suspect pattern detection
 Sec. 19.6

Duplicate documents
• The web is full of duplicated content
• Strict duplicate detection = exact match
– Not as common
• But many, many cases of near duplicates
– E.g., Last modified date the only difference
between two copies of a page
 Eg, Near-duplicate videos

< Original Contrast Brightne Crop

Video> ss

Color Color TV
Enhancement Change size

Multi- Low Noise/Blur Small Logo

25
 Eg, Near-duplicate videos
Original
video

Elongated

Copied
video

26
 Sec. 19.6

Duplicate/Near-Duplicate Detection

• Duplication: Exact match can be detected with

fingerprints
• Near-Duplication: Approximate match
• Compute syntactic similarity with an edit-
distance measure
• Use similarity threshold to detect near-duplicates
– E.g., Similarity > 80% => Documents are “near
duplicates”
– Not transitive though sometimes used transitively
 Sec. 19.6

Computing Similarity
• Features:
– Segments of a document (natural or artificial breakpoints)
– Shingles (Word N-Grams)
– a rose is a rose is a rose my rose is a rose is yours
a_rose_is_a
rose_is_a_rose
is_a_rose_is
a_rose_is_a
• Similarity Measure between two docs (= sets of shingles)
– Set intersection
– Specifically (Size_of_Intersection / Size_of_Union)
 Sec. 19.6

Shingles + Set Intersection

• Issue: Computingexact set intersection of
shingles between all pairs of documents is
expensive
 Sec. 19.6

Shingles + Set Intersection

• Issue: Computingexact set intersection of
shingles between all pairs of documents is
expensive
– Solution  Approximate using a cleverly chosen
subset of shingles from each (called a sketch)
• Estimate (size_of_intersection /
size_of_union) based on a short sketch
Doc
DocAA Shingle set Sketch
A A
Jaccard
Doc
DocBB Shingle set Sketch
B B