Scribd
Upload
179 views2 pages

Exercise 1

This document provides an exercise on evaluating information retrieval systems using precision and recall. It gives the results of two systems for a query with 27 relevant documents. System 1 retrieves documents in the order of d122, d211, d150, d88, d37, d1, d501, d800, d201, d5. System 2 retrieves documents in the order of d10, d700, d6, d250, d88, d600, d59, d422, d500, d7. The exercise asks to plot the precision and recall graphs for each system and precision vs recall graph, and determine which system is better. Both precision and recall can be measured by a search engine to evaluate how

Uploaded by

sadhanamca1
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
179 views2 pages

Exercise 1

This document provides an exercise on evaluating information retrieval systems using precision and recall. It gives the results of two systems for a query with 27 relevant documents. System 1 retrieves documents in the order of d122, d211, d150, d88, d37, d1, d501, d800, d201, d5. System 2 retrieves documents in the order of d10, d700, d6, d250, d88, d600, d59, d422, d500, d7. The exercise asks to plot the precision and recall graphs for each system and precision vs recall graph, and determine which system is better. Both precision and recall can be measured by a search engine to evaluate how

Uploaded by

sadhanamca1
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Information Retrieval Evaluation Exercise 1

IST 441

This exercise is worth 6 points.

In this exercise, you will familiarize yourself with standard methods for evaluating
information retrieval systems with an emphasis on precision and recall.

1. (4 points) Suppose that an IR system contains only 1000 documents. A query is


known to generate 27 relevant documents as listed below:

{d1, d5, d7, d10, d88, d151, d200, d211, d250, d300, d399, d401, d405, d450, d473, d500, d501, d530, d545,
d590, d600, d735, d700, d720, d800, d888, d900}.

Two different IR systems are used to retrieve ranked documents for this query. Each
system only returns the top 10 ranked documents in order of ranking. Systems 1 and 2
each retrieves documents one at a time in the following order with all 10 documents
eventually returned:

System 1: d122, d211, d150, d88, d37, d1, d501, d800, d201, d5.

System 2: d10, d700, d6, d250, d88,, d600, d59, d422, d500, d7.

Answer the following and show your work:


Plot the Precision and the Recall graphs for each system as a function of the
number of documents returned (for 1 document returned, 2 documents
returned, etc).
Plot the Precision versus Recall for systems 1 and 2 using these query results
as a function of the number of documents returned. Note that n1 is the value of
precision and recall for the first document, n2 for the 2 documents.
Which IR system is better? Justify your answer.

precision
n1
n2
n3 n
4
recall
2. (2 points) What can be measured by a search engine? Precision or recall or both?
Why?

You might also like