PRESENTED TO -DEPARTMENT OF LIBRARY AND INFORMATION SCIENCE

TOPIC- CRANFIELD TEST

PRESENTED BY- PALAK YADAV
MLISC.
 THE CRANFIELD TESTS-
Cranfield 1
The first extensive evaluation of retrieval systems was carried out by the ASLIB with the financial
assistance from the National Science Foundation at the College of Aeronautics, Cranfield, UK,
under the supervision of C.W. Cleverdon in 1957.
Its objective was to investigate the comparative efficiency of four indexing systems: UDC, Faceted
Classification, Alphabetical Subject Headings List and Uniterm Indexing system.
• A) Input Elements-
Four indexing systems were taken into consideration:
• 1) An alphabetical subject catalogue based on a subject headings list and a set of rules for
construction of heading
• 2) A classified catalogue based on UDC with an alphabetical chain index to the class headings
constructed
• 3) A catalogue based on a Colon Classification and an alphabetical chain index to the class
headings constructed
• 4) A uniterm coordinate index controlled by an authority list of uniterms compiled during
indexing.
 B) Indexers: Three indexers with varied backgrounds in proficiency in indexing were
taken:

1) One indexer with experience of indexing and subject knowledge of Aeronautics

2) One with experience of indexing but with little subject knowledge

3) One indexer straight from library school (fresh student) with neither indexing
experience nor subject knowledge.

Each indexer was asked to prepare index entries by following the above-mentioned four
indexing

C) Time-period used: Documents were indexed 5 times using variable indexing times (2,
4, 8, 12, 16 minutes). The system was run through three phases with a view to find out
whether the level of performance increases with the increasing experience of the
system personnel

D) Materials Indexed: 100 technical reports and articles on the general field of
aeronautics and the specialized field of high-speed aerodynamics.

E) Items Indexed: With 3 indexers X 4 indexing systems X 5 time periods X 3 runs =

18,000 items were indexed
b) Methodology:

1) A number of people from different organizations were asked to select documents

from the collection and in each case to frame a question to which that document would
be an answer.

2) The project used manufactured queries, which were formulated before the beginning
of the actual search. Altogether 400 queries were formulated and all were processed by
the system in each of the three phases. Thus the system worked on a total of 1200
search queries.

3) The questions were put to the indexers.

c) Results:
Alphabetical
1- The average recall index
ratios for the different81.5%
systems were as follows:

Colon classification 74%

UDC scheme 76%

Uniform indexing 82%

2) Increased time spent in indexing improved the chance of recall. Recall ratios for
different timing were as follows:

Time (in minutes) Recall (in %)

2 73
4 80
8 74
12 83
16 84

3) Success rates in retrieving documents in broad fields of Aeronautics were noted to be

4-5% better than those on the specialized field like high speed Aerodynamics.

4) The success rate in the third group of 6000 items was 3-4% greater than the second
group suggesting that the documents were indexed better in the third group. That is to
say, trained indexers without indexing experience were able to perform consistently
good indexing job in spite of lack of subject knowledge.
Significance:
• Firstly, the test proved that the performance of a system does not depend on
the experience and. subject background of the indexer
• Secondly, it showed that systems where documents are organized by a
faceted classification scheme perform poorly in comparison to the
alphabetical index and uniterm system.
• it developed for the first time the methodologies that could be applied
successfully in evaluating information retrieval systems.
• Moreover, it also proved that recall and precision are the two most important'
parameters for determining the performance of information retrieval systems,
and that these two parameters are related inversely to each other
Criticisms:
One of the main criticisms against this study was the artificiality without much
relation to the real life situation. The questions used for the test were
'manufactured' from the source documents is very much artificial. In a real life
situation queries are not based on previous knowledge of availability of the
document in the store.
• Cranfield 2 -
• The drawbacks of Cranfield necessitated the conduct of further
studies. second stage of Cranfield studies, known as Cranfield 2,
began in 1963 and completed in 1966.
• The Cranfield 2 was a controlled experiment that attempted to
investigate the components of index languages and their effect
on the performance of retrieval systems.
• Some of the drawbacks of Cranfield I were eliminated in Cranfield
2 by bringing real-life situation in it and allowing feedback
mechanisms between the indexers and users.
Test collection :
1400 reports and research papers in the field of high-speed
aerodynamics and aircraft structures were taken as input.
• Query formulation :
• Each author of 200 selected research papers was asked to formulate
questions for which he had cited the reference(s) in the paper. The authors
were also asked to point out the documents that were not cited in their
works but might have been relevant for the question they had formulated.
The abstracts of the whole set of cited: references were sent to the authors
who were asked to assess the relevance of. these with .the questions they
had formulated. The authors identified 1961 papers to be fully or partially
relevant to the 279 questions obtained. Finally, 221 questions and 1400
documents were selected for the experiment.
• Indexing :
The documents were indexed in three different ways:
a) Each document was analysed and important concepts were selected and
recorded in a natural language.
b) Concepts denoted by the single words were listed :
c) Concepts with a weighting (ranging from 1 to 3) were combined to
represent the subject contents of the documents.
Five different types of indexing languages with variations were used in this study:

• Single term index language with 8 variations like uncontrolled natural language,
controlled synonyms, controlled word form, etc.

• Simple concept index language with 15 variations by applying various controls.

• Controlled term index language with 6 similar variations.

• Title index with two variations.

• Index language generated from abstracts with 2 variations.

Searching :

To conduct searches 221 questions generated by the authors of the research papers
were used. Relevancy of each document was ascertained for the questions by grading
them from 1 to 4 in the following manner:

1) Complete answer to the question.

2) High degree of relevance

3) Useful, providing general background of the work or dealing with a specific area

4) Minimum interest, providing information like historical viewpoint.

• Results:
• A best performance result was obtained by the use of natural language single term
index, such as Uniterm, based on words occurring in documents text.
• It was suggested that the terms taken from documents may be used successfully with
minimum control in a post-coordinate index and it is helpful to eliminate synonyms. But
any measures taken to control the vocabulary are likely to decrease its efficiency.
• When broader and narrower terms were included along with the controlled languages of
the thesaurus, the performance worsened .
• Index languages formed out of titles performed better than those formed out of
abstracts.

Results and Conclusion :

The results of the Cranfield 2 tests were unexpected because the test performing index
languages were composed of uncontrolled single words occurring in documents. However,
the variables used in the study were subject to criticisms. Each index language consisted of
different units of words, phrases, or combinations of both. Both the documents and queries
were formed in the same way. Thus the matching of questions' to documents would
evaluate the relative effectiveness of the languages of different specificity. Vickery
comments that the measures used in the second Cranfield project do not adequately
characterise those aspects of retrieval performance, those are of operational importance.
REFERENCES-
• Foskett, A.e. (1996). Subject approach to information. 5 th Ed. London:
The Library Association.
• Chowdhury, GG (2004). Introduction to modern information retrieval.
2 nd Ed. London: Facet Publishing.