The evaluation of library services: a concise review of the existing literature 25

The evaluation of library services:

a concise review of the existing literature
Dr. F.W. Lancaster
Profesor Emeritus University of
Illinois at Urbana-Champaign 501
East Daniel Street Champaign, Il
61820-6212 fax 952 172 44 33020

RESUMEN
Se presenta una reseña bibliográfica de los estudios más sobresalientes sobre evaluación de los
servicios bibliotecarios, en donde se señala su origen y desarrollo a partir de la década del 60.
En forma cronológica, se comentan aquellos aportes estimados por el autor como clásicos o más
significativos en lo referente a los aspectos considerados en la evaluación de los servicios biblio-
tecarios; algunos de las cuales son los relativos ala evaluación de colecciones, evaluación del ser-
vicio de referencia, análisis costo-eficiencia y costo-beneficio, así como otros incorporados
recientemente, entre los que destacan el rango y alcance de los servicios, la cooperación de re-
cursos y los estudios sobre la satisfacción de las necesidades de información de los usuarios. Se
incluye una bibliografía selectiva sobre el tema..

ABSTRACT
A bibliographical review of the most outstanding studies regarding the assessment of library
services, establishing their origin and development since the decade of the sixties. Chronologi-
cally, the contributors which the author considers to be classics, or more significant regarding
the features included in the assessment of library services, some of which address the assess-
ment, as well as other features recently included, among which emphasis is made on the range
and scope of the services, operation of resources and studies concerning the fulfillment of the in-
formation needs of the users. A select bibliography pertaining to these matters is included.

INTRODUCTION

W hile a few isolated and limited attempts

occurred earlier, the application of sig-
nificant objective approaches to the evalua-
ker and Lancaster (1991) and Lancaster
(1993).
1) Comparing parts of the collection against
bibliographies of various types.
2) Comparing strengths of the collection in
tion of library services was virtually unk- This report is note intended as another com- carious subjects areas with measures of
nown before the 1960. The first serious prehensive discussion of library-related eva- community interest (e.g., student enroll-
attempt to develop objective evaluation pro- luation procedures but, rather, a highly selec- ment in courses).
cedures emerged in studies performed for the tive review of the relevant literature. The 3) Analyzing circulation records in an at-
National Library of Medicine by Orr et al. purpose is to identify major approaches to the tempt to determine, from amount of use,
(1968). Somewhat later, Orr (1973) prepared evaluation of various facets of library service whether or not present collection develop-
a “classic” paper on the importance of evalua- and to point to the literature that this author ment policies seem appropriate.
tion procedures for library managers. feels to be of most value or importance. The-
refore, it should serve as useful guide for li- Special aspects of collection evaluation in-
brarians in Mexico who are contemplating clude studies of in-house use, the evaluation
The next milestone was the manual of perfor-
the evaluation of some aspect of their services of periodicals, the weeding of collections,
mance measures for public libraries prepare
by suggesting possible approaches and by re- studies of space utilization, and materials
by DeProspo et al. (1973), which signifi-
ferring to the most significant examples, dis- availability studies. Each of these will be dis-
cantly influenced the later manuals published
cussions or critiques of these approaches. cussed in turn.
by the American Library Association for the
evaluation of public libraries (Van House et
al., 1987) and of academic libraries (Van
House et al., 1990). Bibliographic checking
COLLECTION- RELATED
EVALUATION
This approach (sometimes referred to as “list
Since the 1960, many hundreds of evaluative checking”), which is most appropriate for
studies have been performed in libraries and Three major approaches to the evaluation of academic or other research libraries, involves
the literature has become very extensive. library collections can be identified: comparing a particular subject area of the co-
Comprehensive reviews exist in books by Ba- llection with a bibliography of items that are
26 Investigación Bibliotecológica v.9 number 18 enero/junio 95

Q
supposed to be important items in that area. for all items owned. Kountz (1991) provides
For most subject fields, no standard list of an excellent example of an analysis of this
“best” literature will exist. Therefore, the li- type. In a large university environment, he
brarian who wants to apply this approach was able to compare the strength of student
must compile a list for evaluation purposes. interests as reflected in number of the collec-
tion in various courses. The underlying as-
The most obvious sources to draw upon are sumption is that, if student enrollment in a
the lists of materials cited by scholars writing particular subject area (say bacteriology) is
recently in the field —in monographs, journal strong but the library buys few books in this
articles or other publication forms. The un- field, this may indicate a defect in the collec-
derlying assumption is that the sources cited tion development policy, as would a case in
are those needed by the scholars to support which the library buys a lot in an area of little
their research and that these sources should student interest.
appear in a strong collection in this subject
area. When a librarian applies such a list to
evaluate his own collection in this subject Circulation analysis
field, he is really asking the question “Could
this research have been supported here?” Automated circulation systems can provide
data that are of great value in collection eva-
The classic study of this kind was performed luation. Clearly, circulation figures reflect an
by Coale (1965), using sources cited in scho- important use of collection. When these figu-
larly monographs. Lopez (1983) proposed a res are broken down by classification num-
similar approach, which would actually give ber, the strength of user interest in various
a numerical score to reflect the strength of the A related and potentially subject areas is revealed.
collection in this subject, but this method is valuable approach, although it
unnecessarily complicated and cannot really has been little used, is to tale Circulation data are of most value when they
be recommended. Others have drawn referen- are related to data on the holdings of the li-
the references retrieved in
ces form the bibliographies of scholarly jour- brary in various subject areas. There are seve-
nal articles, from dissertations, or other publi- database searches as a set of ral ways in which circulation data can be rela-
cation forms. Nisonger (1992) has prepared a items appropriate to evaluate ted to holdings data. One measure is turnover
useful survey of this and other approaches to the collection of the library in rate (Van House, et al., 1987), which is the
collection evaluation. The advantages and di- average number of times that a book in a par-
sadvantages of using different types of publi- which the searches were ticular subject field is borrowed in a year,
cations as sources for bibliographic checking performed. each book is borrowed once a year on average
are discussed in detail in Chapter 2 of Lancas- and the turnover rate is 1.0. If circulation was
ter (1993). doubled, turnover would be 2.0. Classes with
very low turnover relative to the other classes
A related and potentially valuable approach, are those in which user interest is very low. At
although it has been little used, is to tale the least, users are not very interested in the
references retrieved in database searches as a books the library now owns on these subjects.
set of items appropriate to evaluate the collec- The librarian should closely examine these
tion of the library in which the searches were classes to see what corrective action is needed
performed. The justification is obvious; the (e.g., weed out obsolete material from the co-
subjects of the searches reflect the interests of llection, buy fewer books in this subject in the
at least some library users and the items re- future, or buy different books in this area).
trieved should be the ones that these users will
be looking for in the library. A rare example A second way of comparing holdings with the
of this approach, which seems particularly circulation is by counts of the proportion of
appropriate to the evaluation of special libra- the collection in various subject areas that is
ries can be found in Lancaster et al. (1991c). absent from the shelves (in circulation) at a
particular time. Consider two subject classes,
Collection analysis x and y, with exactly 500 books in each. To-
day there are 400 x books in circulation and
Automated systems of various kinds in libra- only 100 on the shelves. For class y the situa-
ries can act as management information tion is reversed: 400 books on the shelf and
systems —giving the library manager better 100 in circulation. Clearly, x is a heavily used

R
data on which to base decisions or establish class and y is not. Relative to the other classes
policies. A systematic analysis of the present in the library, x may be one of several heavily
collection can in itself be a useful evaluative overused classes while y may be one of seve-
procedure, indicating subject areas in which ral that are heavily underused. Classes at both
the collection appears unusually strong and extremes of the distribution (most overused
areas in which it appears weaker. Detailed co- and most underused) may require attention. A
llection analyses were difficult to perform in heavily overused class is one in which the co-
the pre-automation era but should be simple llection may not be strong enough to meet the
when a library has detailed automated records needs of the users. Since most of the books are
 The evaluation of library services: a concise review of the existing literature 27

S
absent from the shelves, those few remaining adopt a somewhat different approach, rela-
are likely to be books of lesser value or inte- ting interlibrary loan data to current acquisi-
rest (i.e., shelf bias is high). A heavily underu- tions data rather than to total holdings data.
sed class needs the same kind of examination Their “collection balance indicator” again,
referred to earlier for the low turnover class. can be used to identify classes that may need
This type of collection use indicator was em- to be strengthened.
ployed by McClellan (1956 before the library
automation era. A modern example can be
found in Dowlin and Magrath (1983). In-house use

A third way of comparing circulation and hol- Books and other materials can be used within
dings data, and perhaps the most useful, is in the library instead of being borrowed, so cir-
terms of relative use (Jain, 1965). Relative culation does not reflect total use. In a large
use compares actual circulation with expec- research library, in-house use may gratly ex-
ted (in a probabilistic sense) circulation. For ceed circulation.
example, if class x accounts for 3% of the to-
tal collection, probability suggests that it There are many ways of studying in-house
should receive 3% of the total circulation. If it use, from the very simple (recording mate-
accounts for 7% of the circulation, it is hea- rials left on tables or other study areas) to the
vily overused, If it accounts for 1% of the cir- complicated (interviewing samples of users
culation, it is heavily underused. The diffe- within the library). Rubin (1986) describes
rence between the expected circulation can be and contrasts most of the possible approa-
expressed in various ways, perhaps the most ches, and the problems of defining “use” in
obvious being percentage of expected use The recording of materials this context are discussed sin some detail in
(PEU). If a class has a PEU of 100, it is beha- used in the library, and left on chapter 4 of Lancaster (1993). Daiute and
ving exactly as expected (e.g., it accounts for tables or other reading areas, is Gorman (1974) describe rather elaborate pro-
0.5% of the use). A class with a PEU of 33 is cedures for the random sampling of users to
heavily underused (use is exactly one third of
greatly facilitated if bar codes
be interviewed within the library.
the expected use). Lee and Lockway (1991) are used to identify the books
provide one good example of the use of this in the library Lancaster (1993) gives several examples of
measure. forms that can be used to record in-house use
—e.g., stapled to the front of periodicals is-
The three ways of relating collection data to sues or slipped inside a bound volume.
circulation data (turnover rate, proportion of
collection absent from the shelves, relative The recording of materials used in the library,
use) all do essentially the same thing: they and left on tables or other reading areas, is
allow the librarian to identify those subject greatly facilitated if bar codes are used to
classes (most overused, most underused) that identify the books in the library. Such codes
seem to be most in need of attention. The data can then be ready by means of a scanner
do not, in themselves, reveal what the correc- (wand). Use of technique is illustrated in pa-
tive action needs to be (which requires some pers by Lee and Lockway (1991) and by Titus
human interpretation) but they at least tell the et al. (1994).
librarian which classes to focus on.
The simplest of techniques for indicating
The more data the librarian has that reflect use which volumes have been used in the library,
and demand for various subject areas, the bet- and which not, is the placing of an adhesive
ter the collection development policy is likely dot on the spine of a book before it is returned
to be. Interlibrary lending data can be useful to the shelves after being collected from ta-
when compared whit circulation data. For bles or other study areas. A complete descrip-
example, the fact that class x is underused (as tion can be found in Slote (1989), who refers
measured by circulation) but the library is bo- to it as the “spine marking” method.
rrowing many class x books from other libra-
ries to satisfy user requests, suggests that the
wrong types of books are being bought in this Evaluation of periodicals
class. Aguilar (1986) uses “ratio of borro-
wings to holdings” (RBH) as one indicator of As budgets have shrunk, librarians have be-

T
classes that need corrective action. The RBH come increasingly concerned with the eva-
for a class is simply % of borrowings over % luation of the periodicals they subscribed to in
of holdings. Thus, a class which accounts for order to decide which might be discontinued.
15% of borrowings but only 8% of holdings Many possible evaluation criteria can be
gets an RBH of 1.9. The higher the RBH, the identified, from the purely subjective (opi-
more likely it is that the library needs to buy nions of faculty or other users) to the purely
more in this area. Aguilar combines RBH quantitative (number of uses a year). These
data with relative use circulation data to deve- various criteria are discussed in detail in
lop a purchasing model. Byrd et al. (1982) Chapter 5 of Lancaster (1993).
28 Investigación Bibliotecológica v.9 number 18 enero/junio 95

U
Ideally, one should be able to give a composite volved in one type of weeding: the retirement
numerical score to each periodical in the li- of less used materials to less accessible (and
brary, based on the scores it achieves for all of less costly) storage areas such as closed-ac-
the evaluation criteria selected. For example, a cess bookstacks or even off-site warehouses.
heavily used journal will achieve a high score
for the “use” factor but a low score for the The most obvious criterion for weeding or re-
“cost” factor if it is very expensive. Broude tirement to storage is amount of use. Trues-
(1978) illustrates the scoring of periodicals in well (1966) developed an ingenious method
this way, although his method is unnecessarily for retiring books to storage on the basis of the
complicated because he takes into account too time elapsing between circulations. His met-
many evaluation factors and some of them are hod results in a general retirement rule (e.g.,
somewhat redundant. Lancaster (1993) sug- retire any book that has not circulated in the
gests a much simpler scoring procedure. last 60 months]) and the effects of applying
the rule can be rather precisely estimated
The most important evaluation criteria are un- (e.g., the books not circulating in the past 60
doubtedly cost and use. When these are combi- months constitute 40% of the entire collec-
ned, the result is a measure of cost-effective- tion; if retired to storage, they will be reques-
ness, namely cost and use. When these are ted approximately x times only in the next y
combined, the result is a measure of cost-effec- years). One example of the application of the
tiveness, namely cost per use. Chrzastowsky principle can be found in Williams (1986).
(1991) gives and excellent example of the ran-
king of periodicals in an academic special li- Other librarians have attempted to develop
brary on the basis of cost per use. general retirement rules based on criteria ot-
The most complete her than use, most obviously the age of the pu-
discussion of weeding Modern technologies (computer networks blication. The classic study in the research li-
methods and principles coupled with telefacsimile) make it increa- brary environment is that of Fussler and
singly feasible for a library to obtain photoco- Simon (1969), although this is now mainly of
appears in a book historical interest.
pies from journals held by other libraries ra-
by Slote (1989). pidly and economically. At the same time,
commercial document delivery services now Just as periodicals can be given a numerical
offer a convenient alternative to traditional li- score to allow the optimization of cancellation
brary resource sharing. Consequently, libra- decisions, books in a library could be given nu-
rians have become increasingly involved in merical scores to optimize weeding decision,
cost-effectiveness analyses relating to the ac- where the score is the sum of several compo-
cess versus ownership decision; that is, at nent scores (e.g., for age, amount of use, ap-
what level of use within the library is it more pearance on “recommended” lists, physical
economical to suscribe to a periodical rather condition). Lancaster (1993) has illustrated
than obtaining copies from elsewhere. how such a scoring method might be applied
Clearly, the breakeven point depends largely
on the cost of the journal. Analyses of this The most complete discussion of weeding
type have been performed for many years, a methods and principles appears in a book by
classic being that of Williams et al. (1968). Slote (1989).
An excellent example of a recent study is that
of Gossen and Irving (1995). In connection with weeding, it is important
that librarians understand the phenomenaon
Weeding of collections of “obsolescence” since, in the library con-
text, materials are considered to obsolesce as
While true research libraries rarely dispose of their use declines with age. The classic paper
any materials other than unwanted duplicate on obsolescence remains that of Line and
copies, most other libraries are well advised Sandison (1974), although some of their as-
to weed their collections on a regular basis. sertions have since been repudiated —e.g., by
Without weeding, the shelves of the library Stinson and Lancaster (1987), Nakamoto
will be filled with obsolete materials or mate- (1988) and Sullivan et al. (1980-1981). The
rials that have received little or no use since most complete discussion of obsolescence,
being added to the collection. Retaining these presented from many different perspectives,
materials on open access shelves gives users can be found in an issue of Library Trends
the impression that the collection is not very edited by Pao and Warner (1993).

V
useful or interesting and makes it more diffi-
cult for them to fin the better (e.g., more cu- Space utilization
rrent) materials.
This topic is closely related to that of wee-
Furthermore, most libraries find themselves ding, as discussed in the previous section,
short of space. Space is wasted if the shelves Since the materials in a library, especially lar-
are full of materials that are little if ever used. ge one, may be kept at various levels of acces-
Even the large research libraries must be in- sibility (open access shelves, closed access
 The evaluation of library services: a concise review of the existing literature 29

W
bookstacks, remote storage facilities), it is but many others have been conducted since
important that the degree of accessibility be then. A major analysis is that of Saracevic et al.
related to the probable use. Most obviously (1977) and Mansbridge (1986) gives a useful
the open access shelves of the library —the review of earlier studies. A recent example can
“prime” space— should be devoted to the ma- be found in Chaudry and Ashoor (1994).
terials most in demand.
Simulation studies try to achieve the same re-
A special space utilization situation is that re- sults —calculation of a materials availability
lated to bound periodicals. Given space to sto- rate plus analysis of reasons for nonavailabi-
re say, 3000 volumes of bounds periodicals, lity— without bothering library users. Suppo-
the librarian must decide which titles should be se that one assembles, say, 500 bibliographic
held on these shelves for how far back. references representing items —books, re-
Clearly, a blanket decision –e.g., keep all titles ports, journal articles, and so on— that are
five years back— is unlikely to result in good fully representative of the kinds of materials
use of the space since some titles may still be in that users of that library are likely to be loo-
demand when ten or more years old, while ot- king for. An investigator walks into the li-
hers may rarely be used when more than two brary on a particular day with the list of refe-
years old. rences and checks to see how many not. In
effect, the investigator is simulating 500 li-
This is a true cost effectiveness decision, the brary users, each looking for a single item. As This is a very simple method to
measure being use per metre (or foot) of shelf in the other procedure, a materials availability apply, the only essential
space occupied. The principles involved in rate can be calculated (precisely) and reasons requirement being that the list of
the decision are well covered in Groos for nonavailability can be identified.
(1969), Brookes (1970), Taylor (1976-1977),
references should represent the
Wenger and Childress(1977), and Stayner This is a very simple method to apply, the real needs and interests of library
and Richardson (1983). only essential requirement being that the list users
of references should represent the real needs
Materials availability studies and interests of library users. Possibilitites for
arriving at such lists are discussed in Chapter
Materials availability studies are performed to 8 of Lancaster (1993).
determine the probability that a book (or other
item), known to be in the library’s collection, is The materials availability simulation was
available to a user at the time he or she looks pioneered by Orr et al. (1968) for academic li-
for it. There are two possible approaches: braries and by DeProspo et al. (1973) for pu-
blic libraries.
1)Studies that require the cooperation of li-
brary users The major factors affecting the probability
2)Studies performed by means of simulations that a book will be on the shelf and available
when looked for are (1) its popularity, (2) the
number of copies owned, and (3) the length of
In the first approach, a user is asked to com- the loan period. Having these data for the
plete a brief form to indicate which items he books in a library allows one to calculate rat-
was looking for in the library and whether or her accurately the probability of availability
not he was able to find them. During the pe- of any item. “Popularity” needs not to be a ne-
riod of the survey, forms can be handed out to bulous factor; it can be quantified precisely
all users entering the library. However, it is (e.g., in terms of frequency with which a book
usually better to focus on a random sample of has been borrowed or simply the date on
users and to do everything possible to get the- which it was last borrowed). Buckland (1975)
se people to cooperate. provides a detailed account of these availabi-
lity factors and how they interact.
Van House et al. (1987, 1990) give sample
forms, together with detailed instructions on
how to perform the survey. If properly con- REFERENCE SERVICES
ducted, a study of this kind can give a mate- EVALUATION
rials availability rate (the number of items
found/the number of items looked for]), as The major functions of a reference service are
well as allowing analyses to be performed to the answering of factual-type questions and
identify reasons for non-availability of mate- the performance of database searches. Some
rials (in circulation, in use in the library, at reference departments also have responsibili-

X
binding, misshelved, waiting to be reshelved, ties in the area of bibliographic instruction.
missing and so on).
Question-answering
The first studies of this kind were performed in
British academic libraries (Urquhart and Evaluative activities related to question-ans-
Schofield, 1971, 1972; Schofield et al., 1975) wering range from the simple to the complex.
30 Investigación Bibliotecológica v.9 number 18 enero/junio 95

Y
The simplest activity is the recording, by he or she is looking for materials that will sa-
members of the reference staff, of questions re- tisfy, or help to satisfy, some information need.
ceived and action taken, including sources In most cases, the searcher would like to find a
used and whether the librarian felt confident few items, preferably of high quality, that deal
that the question was dealt with satisfactorily. with the subject of interest. Less often, a really
Hawley (1970) gives a good example of a form comprehensive search is wanted —the sear-
used for such purpose. A slightly more sophis- cher would like to find everything available on
ticated approach, described by Murfin and Gu- the subject. In a few rare cases, a single item on
gelchuk (1987), involves two forms for each the topic will be enough.
question, one completed by the reference li-
brarian and one by the user, thus allowing a The results of a subject search can be evaluated
comparison to be made of the views of each on in a purely subjective way —is the user satisfied
the success of the transaction. Clearly, this two or not? In general, however, it is better to make
faceted approach could not be applied for the evaluation more quantitive: how many of
questions arriving by telephone. the items retrieved are useful to the user, how
many are really important, how many are new
The most sophisticated approach to this as- to the user? Such results can be expressed as
pect of evaluation involves the use of a set of performance ratios: a precision ratio (number of
questions for which complete and correct useful items retrieved/total items retrieved),
answers have been established in advance. major precision ratio (number of really impor-
The set can be used to test whether the refe- tant items/total items retrieved), and a novelty
rence librarians in a particular library are able ratio (numbers of new and useful items/number
to supply complete and correct answers or of useful items retrieved).
not. The questions can be applied obstrusi-
vely or unobstrusively. To get feedback on the results of a subject
search will usually mean that the user (reques-
In an obstrusive study, the librarians know ter) must complete a brief evaluation question-
they are being tested. Usually they operate naire. Examples of such questionnaires can be The most sophisticated approach
under time constraints —i.e., they are to ans- found on chapter 11 of Lancaster (1993). to this aspect of evaluation
wer as many questions as possible within a
particular period, say four hours. Pizer and However, an evaluation based only on the
involves the use of a set of
Cain (1968) may have been the first to descri- items retrieved in a search gives an incomple- questions for which complete and
be obstrusive reference studies of this type. te picture of its success. In almost cases, there correct answers have been
will be other items in the database, items not
In the unobstrusive version, the librarians do
established in advance.
retrieved, that would be judged useful if the
not know they are being evaluated. Questions user saw them. In some cases, these might be
are posed by “volunteers” (e.g., university more important than some of the other useful
students) acting as though they are real library items that were retrieved; they may even be
users with real questions. In most studies, the more important than all of the items that were
questions are posed by telephone; in a few retrieved. For example, they could be items
they are made by personal visit. The reference important to the user’s research but not pre-
department is judged by the proportion of the viously known to him.
questions that they are able to answer comple-
tely and correctly. It follows, then, that a complete evaluation of
a subject search must make attempt to deter-
Crowley and Childers (1971) popularized this mine, or at least estimate, how many useful
approach, Childers (1972) discusses the im- items have been missed. If this were known, it
plementation problems, and Crowley (1985) would be possible to apply a further perfor-
contributes a useful review. The largest study mance ratio to the results – a recall ratio
of this type in a single academic library has (number of useful items retrieved/number of
been described by Lancaster et al. (1991a). useful items in the database). Unfortunately,
it is not at all easy to estimate how many use-
Database searching ful items were missed in a search. Although
several estimation procedures exist (see Lan-
This section of the review deals with the eva- caster and Warner, 1993), they are somewhat
luation of subject searches. In this connec- difficult to apply. A library would not want to
tion, the term database refers to any biblio- attempt some estimate on a routine basis, alt-
graphic source in which subject searches can hough it would be important to do so for a
be performed, including databases available sample of searches if the library was really se-
through remote online access, CD-ROMs, li- rious about the evaluation of a database sear-
brary catalogs, and printed indexing/abstrac- ching service (e.g., an SDI service within an
ting publications. industrial organization).

When a library user performs a search (or The most complete evaluation of a database
when a librarian performs the search for a user) searching service remains that of Lancaster
 The evaluation of library services: a concise review of the existing literature 31

(1968). Saracevic et al. (1998) present the re- services is fully dealt with in Mitchell et al.
sults of an important research project that (1978), Griffiths and King (1983), Citron and
among other things, showed that different Dodd (1984), Roberts (1984, 1985), Rosen-
searchers (even highly experienced ones) will berg (1985) and Kantor (1989). A simplified
use completely different approaches to search approach is given in Chapter 14of Lancaster
on a particular topic and will tend to retrieve (1993).
different items. McCue (1988) describes a rare
unobstrusive study of database searching: the Cost-effectiveness analyses
same searches were performed by different li-
braries without their knowing they were being Cost-effectiveness are studies that relate to
evaluated. Lancaster et al. described a study of return on investment Most often, studies of
subject searchers in a very large OPAC, using this type are performed to compare different
very stringent criteria (did the searcher find approaches to achieving particular results or
“the best” items on the subject?). to determine the optimum allocation of re-
sources available for some purpose (e.g., the
Clearly, the most difficult searches to evalua- optimum allocation of funds available for
te are those performed by library users them- providing periodical access for library users).
selves since it is very difficult to collect data
form users under these circumstances. For an Cost-effectiveness measures relate the cost of
example of a detailed evaluation of user sear- a service to some measure of its effectiveness.
ches in a CD-ROM database see Lancaster et Frequently, the measure of effectiveness is
al. (1994). simple “use”. Cost per circulation, cost per
use of a periodical, cost per question answe-
Bibliographic instruction red, and cost per search performed could all
The evaluation of bibliographic be used as cost-effectiveness measures, alt-
Bibliographic instruction —teaching people hough more refined measures (e.g., cost per
instructions is quite different form how to use the resources of the library or in- useful item retrieved and cost per questions
other facets of library-related formation resources in general— is an acti- answered completely and correctly) are to be
evaluation because the vity that has become of increasing importan- preferred.
ce in the last few years, especially in
metodologies come more from the academic libraries, and the activity is one that One type of cost-effectiveness study is per-
field of education than from may reside in the reference department. One formed to determine which periodicals
library science reason why bibliographic instruction has gai- should be discontinued because they have
ned prominence is the fact that the library has high cost-per-use figures associated with
increasingly become a self-service institution them. A good example can be found in
with users themselves doing more and more Chrzastowski (1991). Closely related are stu-
of the things formerly done by librarians. dies that compare he cost of owning a periodi-
cal with the cost of obtaining copies (from ot-
The evaluation of bibliographic instructions her libraries or from commercial suppliers)
is quite different form other facets of li- when the need arises. A recent example can
brary-related evaluation because the metodo- be found in Gossen and Irving (1995).
logies come more from the field of education
than from library science. The most complete Cost-effectiveness analyses relating to the
discussion can be found in Chapter 12 of Lan- optimum use of space can be found in Gross
caster (1993), which deals with the following (1969), Brookes (1970), Taylor (1976-1977),
aspects: reaction of instructors, reaction of Wenger and Childress (1977), and Stayner
those who are being instructed, use of inde- and Richardson (1983).
pendent observers, evaluation of learning,
behavioral changes in those instructed, eva- Analyses that have become increasingly im-
luation of program results, and cost-effective- portant in recent years are those that look at
ness aspects. cost effectiveness aspects of making informa-
tion sources available in different formats:
COST-EFFECTIVENESS AND print versus CD-ROM, CD-ROM versus online
COST-BENEFIT STUDIES access, and so on. Excellent comparison of
CD-ROM versus online access, and so on. Exce-
Several cost-effectiveness studies relating to llent comparisons of CD-ROM versus online re-
various aspects of library service have been mote access versus loading the database onto
mentioned throughout this review. The follo- the institution’s own computer can be found in
wing section will deal with cost-effectiveness

Z
the work of Meyer (1990, 1993). The same si-
analyses as a “genre” of library evaluation tuation is deal with by Halperin and Renfro
studies. (1988), and Machovec and Brunning (1991)
have identified the various factors that need to
Clearly, a prerequisite for a cost-effective- be considered in making decisions of his type.
ness study is the calculation of the cost of CD-ROM versus online access are dealt with
some library service. The costing of library variously by Welsh (1989), Huang and McHa-
32 Investigación Bibliotecológica v.9 number 18 enero/junio 95

le (1990) and Huang (1991), although these the cost of user time and the cost of libra- More sophisticated analyses of resources sha-
studies are of variable quality (see Lancaster rian time. ring activities usually take the form of
—1993—, Chapter 14, for a critique). cost-effectiveness studies. For example,
An example of the first of these approaches Lowry (1990) looks at interlibrary loan from
Cost effectiveness analyses have to do with the can be found in the work of Magson (1973). the lending institution’s point of a view as a
optimum allocation of resources. In this re- Examples of the second can be found in Ro- loss of investment, and Bonk and Pilling
gard, it is important to recognize the phenome- senberg (1969), Kramer (1971), Mason (1990) give cost data for the various compo-
non of the “90% library” (Bourne, 1965). In (1972), Nightingale (1973), Bleck (1977), Co- nent activities associated with borrowing and
essence, this means that, for any particular ser- llete and Price (1977), Johnson et al. (1977) with lending. Related cost effectiveness
vice, it will be possible to satisfy some speci- Jensen et al. (1980) and Estabrook (1986). analyses can be found in the work of Kava-
fied proportion of the total demands (85% or nagh (1988), Rutledge and Swindler (1988)
90% perhaps) efficiently and economically. A more sophisticated approach can be found and MacDougall et al. (1990).
To get much beyond this point, however, in the studies of king research Inc. (1982,
would require a completely disproportionate 1984), including Roderer et al. (1983), and Surveys of user satisfaction
level of expenditure. For example, perhaps Griffiths and King (1991). Basically, their ap-
90% of access needs can be satisfied by sub- proach compares the cost of an information
This survey has concentrated largely on stu-
scribing to 100 journals at a total cost of around service with the financial benefit associated
dies designed to gather objective data on the
$5000. To satisfy 95% of the needs, however, with such things as savings of time and avoi-
performance of library services. Neverthe-
250 journals may be needed and the cost in- dance of duplication in research.
less, the subjective impressions of users re-
creases to around $12, 000. Other examples of garding the library’s services do have value.
diminishing returns in library services can be They indicate how “happy” the customers are
found in Chapter 14 of Lancaster (1993). OTHER ASPECTS
and also can be used in a diagnostic way to
identify possible sources of dissatisfaction or
Cost-benefit studies Range and scope services types of users who are less satisfied than ot-
hers. Questionnaires can be employed with
Libraries can be compared on the basis of the samples of library users to determine their sa-
depth and breadth of the services they provi- tisfaction with the library’s services in gene-
de. Evaluations of this kind were pioneered ral or their satisfaction with the services pro-
by Orr et al. (1968), who developed a ques- vided on a particular visit.
tionnaire to be completed by library directors.
The questionnaire addressed such matters as
hours of opening, services provided, lending Samples of questionnaires can be found in
and other policies and facility availability. Van House et al. (1987, 1990) and Sumsion
They were able to assign numerical scores to (1993), among other sources. A good recent
the various service elements, allowing libra- example of a survey in an academic library is
ries to be compared on the basis of overall Mccarthy (1995). Her study was able to iden-
score and /or component scores relating to tify sources of both satisfaction and dissatis-
different facets of service (document deli- faction and to relate degree of satisfaction to
very, reference and so on). type of student user.

This and other similar studies are reviewed in

Baker and Lancaster (1991).

Resource sharing

Resource sharing activities may be the least

evaluated aspect of library services, although
Peat, Marwick, Mitchell & Co. (1975) have
identified the relevant evaluation criteria for
various aspects of resource sharing.

There have been countless studies of fill rate

and delivery times for interlibrary lending,
dealing both with groups of libraries in a par-
ticular region (e.g., Medina, 1988) and indivi-
A cost benefit study looks at the relationship dual libraries (Horton, 1989).
between the cost of some activity and the be-
nefits of arising from it. Ideally, both sides of
the cost benefit equation shoud be expressed REFERENCES
in the same unit —a monetary one. Herein lies
the problem of performing cost-benefit stu-
dies applied tothe
1. Comparing library service
cost of in monetary
maintaining the li-
terms.
brary,Consequently,
or one or more ofalthough several
Aguilar,
its activities, withW. The application of relative use and interlibrary demand in collection deve-
cost-benefit
The thetwo
cost ofstudies
most haveapproaches
promising
purchasing been
the sameconducted,lopment.
to cost
level of Collection Management, 8(1), 1986, 15-24.
particularly
benefit
service in industrial
analysis
form libraries,
involveagencies.
outside the these
following:
The have
“bene-
beenfit”conducted, particularly
of the in-house libraryinisindustrial
proven ifli- it
braries,
2. can these have
Comparing
provide not
costbeen
theservices
the afully
of more convincing.
library user than
cheaply per-
forming some service for himself with the White and Abels (1995) present a more sop-
if they were purchased elsewhere.
cost of the librarian providing the service histicated approach based on disconfirmation
to him. This type of analysis, which has theory. In essence, the method measures the
been applied most often to database sear- difference between user expectations regar-
 The evaluation of library services: a concise review of the existing literature 33

Baker, S. L. and Lancaster, F.W. The Measurement and Evaluation of Library Servi-
ces, Second edition. Arlington, V.A. Information Resources Press, 1991.

Blick, A. R. The value of measurement in decision-making in an Information Unit —a

cost benefit analysis. Aslib Proceedings, 29, 1977, 189-196.

Bonk, S. C. and Pilling, D. Modelling the economics of Interlending. Interlending and

Document Supply, 18, 1990, 52-56.

Bourne, C. P. Some user requirements stated quantitatively in terms of the 90% li-
brary. In: Electronic Information Handling, ed. by A. Kent et al. pp. 93-110.
Washington, DC, Spartan Books, 1965.

Brookes, B. C. Obsolescence of special library periodicals: sampling errors and utility

contours. Journal of the American Society for Information Science, 21, 1970,
320-329.

Broude, J. Journal deselection in an academic environment: a comparison of faculty

and librarian choices. Serials librarian, 3, 1978, 147-166.

Buckland, M. K. Book Availability and the Library User. New York, Pergamon,
Press, 1975.

Byrd, G.D. et al. Collection development using interlibrary loan borrowing and acqui-
sitions statistics. Bulletin of the Medical Library Association. 70, 1982, 1-9.

Chaudry, A. S. and Ashoor, S. Comprehensive materials availability studies in acade-

mic libraries. Journal of Academic Librarianship, 20, 1994, 300-305.

Childers, T. Managing the quality of reference/information service. Library Quar-

terly, 42, 1972, 212-217.

Chrzastowski, T. E. Journal collection cost-effectiveness in an academic chemistry li-

brary: results of a cost/use survey at the University of Illinois at Urbana-Cham-
paign. Collection Management, 14 (1/2), 1991, 85-98.

Citron, H. R. and Dood, J. B. Cost allocation and cost recovery considerations in a spe-
cial academic library: Georgia Institute of Technology. Science and Technolo-
gies Libraries, 5 (2), 1984, 1-14.

Coale, R. P. Evaluation of a research library collection- Latin America Colonial his-

tory at the Newberry. Library Quarterly, 35, 1965, 173-184.

Collete, A. D. and Price, J. A. A cost/benefit evaluation of online interactive biblio-

graphic searching in research and engineering organization. In The value of in-
formation: Collection of Papers Presented at the 6th Mid-Year Meeting [of
ASIS], May 19-21. 1977, pp. 24-34. Syracuse, NY, Syracuse University, 1977.

Crowley, T. and Childers, T. Information Service in Public Libraries: Two Studies.

Metuchen, NJ, Scarecrow Press, 1971.

Daiute, R. J. and Gorman, K. A. Library Operations Research. Dobbs Ferry; NY,

Oceana Publication, 1974.

DeProspo, E. R. et al. Performance Measures for public Libraries. Chicago, Public

Library Association, 1973.

Dowlin, K. and Magrath, L. Beyond the numbers —a decision support system. In: Li-
brary Automation as a Source of Management Information; ed. by F.W. Lancas-
ter, pp.27-58. Urbana, University of Illinois, Graduate School of Library and
Information Science., 1983.

Estabrook, L. S. Valuing a document delivery system. RQ, 26, 1986, 58-62.

34 Investigación Bibliotecológica v.9 number 18 enero/junio 95

Fussler, H.H. and Simon, J.L. Patterns in the Use of Books in Large Research libra-
ries. Chicago, University of Chicago Press, 1969.

Gossen, E.A. and Irving, S. Ownership versus access and low use periodical titles. Li-
brary Resources &Technical Services, 39, 1995, 43-52.

Griffiths, J.M. and King, D.W. Library Cost Benefit Analysis: A Manual Prepared for
the library Cost Benefit Analysis Seminar Presented at the SUNY/OCLC Net-
work Annual Directors Day on February 17, 1983. Rockville, MD, King Re-
search, Inc., 1983.

Griffiths, J.M. and King, D.W. A Manual on the Evaluation of Information Centers
and Services. Neuilly-sur-Seine, North Atlantic Treaty Organization, Advisory
group for Aerospace Research and development, 1991. AGARD-AG-310.

Gross, O.V. Citation Characteristics of astronomical literature. Journal of Documen-

tation, 25, 1969, 344-347.

Halperin, M. and Renfro, P. Online vs Cd-ROM vs. Onsite: high volume sear-
ching-considering the alternatives. Online 12 (6), 1988, 36-42.

Hawley, M. B. Reference statistics. RQ, 10, 1970, 143-147.

Horton, W., Jr. Interlibrary loan turnaround times in science and engineering. Special
Libraries 80, 1989, 245-250.

Huang, S.T. CD-ROM database searching vs traditional online database searching.

Proceedings of the National Online Meeting 1991; ed. by M.E. Williams, pp.
139-148. Medford, N.J. Learned Information, 1990.

Huang St. and McHale, T. J. A cost effectiveness comparison between print and online
versions of the same frequently-used sources of business and financial informa-
tion. Proceedings of the National Online Meeting 1990; ed. by M.E. Williams,
pp. 161-168. Medford N.J., Learned Information 1990.

Jain, A.K. A Sampled Data Study of Book Usage in the Purdue University Libraries.
Lafayette, IN, Purdue University, 1965.

Jensen, R. J. et al. Costs and benefits to industry of online literature searches. Special
Libraries, 71, 1980, 291-299.

Johnson, F. D. et al. NASA Tech Brief Program: Cost Benefit Evaluation. Denver,
University of Denver Research Institute, 1977.

Kantor, P. B. Library cost analysis. Library Trends, 38, 1989, 171-188.

Kavanagh, R. TRESNET: The Trent Resource Sharing Network. Canadian Library

Journal, 45, 1988, 283-288.

King Research Inc. A study of the Value of information and the Effect on the Value of
Intermediary Organizations, Timelines of Services and Products, and Com-
prehensiveness of the EDB. Rockville, MD, King Research, Inc., 1984.
DOE/NMB-1078- DE 85003670.

King Research Inc. Value of the Energy Data Base. Rockville, MD, 1982.

Kountz, J. What’s in a library: comparing holdings to the constituencies served. Li-

brary Hi-Tech, 9(2), 1991, 31-48, 61.

Kramer, J. How to survive in industry: cost justifying library services. Special Libra-
ries, 62, 1971, 487-489.

Lancaster, F. W. Evaluation of the MEDLARS Demand Search Service. Bethesda,

MD, National Library of Medicine, 1968.
 The evaluation of library services: a concise review of the existing literature 35

Lancaster, F. W. If you Want to Evaluate your Library… Second edition. Cham-

paign-Urbana University of Illinois, Graduate School of Library and Informa-
tion Science.

Lancaster, F. W. and Warner, A. Information Retrieval Today /third edition of Infor-

mation Retrieval Systems: Characteristics, Testing and Evaluation). Arlington,
VA, Information Resources Press, 1993.

Lancaster, F. W. et al. The diagnostic evaluation of reference service in an academia

library. In: Evaluation of Public Services and Public Services Personnel: Pro-
ceedings of the Thirty-second Allerton Park Institute; ed. by Allen, pp. 43-57.
Urbana, University of Illinois, Graduate School of Library and Information
Science, 1991a.

Lancaster, F. W. et al. Identifying barriers to effective subject access in library cata-

logs. Library Resources & Technical Services, 35, 1991b, 377-391.

Lancaster, F. W. et al. The relationship between literature scatter and journal accesibi-
lity in an academia special library. Collection Building, 11(1), 1991c, 19-22.

Lancaster, F. W. et al. Searching databases on CD-ROM: comparison of the results of

end-user searching with results from two modes of searching by skilled interme-
diares. RQ, 33, 1994, 370-386.

Lee, D. C. and Lockway, L. A. Using an online comprehensive library management

system in collection development. Collection Management, 14(3/4), 1991,
61-73.

Line, M. B. and Sandison, A. “Obsolescence” and changes in the use of literature with
time. Journal of Documentation, 30-1974, 283-350.

Lopez, M. D. The Lopez citation technique of in depth collection evaluation explica-

ted. College & Research library, 44, 1983, 251-255.

Lowry, C. B. Resource sharing or cost shifting? The unequal burden of cooperative

cataloging and ILL in network. College and Research Libraries, 51, 1990,
11-19.

MacDougall, A. F. et al. Effectiveness of a local inter-loan system for five academic li-
braries: an operational research approach. Journal of Documentation, 46, 1990,
353-358.

Machovec, G.S. and Brunning, D. R. Choices: collection management issues of the

IOLS. Integrated Online Library Systems (Proceedings), 1991, 83-91.

Magson, M.S. Techniques for the measurement of cost-benefit in information centres.

ASLIB Proceedings, 25, 1973, 164-185.

Mansbridge, J. Availability studies in libraries. Library and Information Science Re-

search, 8, 1986, 299-314.

Mason, D. PPBS: application to an industrial information and library service. Journal

of Librarianship, 4, 1972, 91-105.

McCarthy, C.A. Students’ perceived effectiveness using the university library. Colle-
ge and Research Libraries, 56, 1995, 221-234.

McClellan, A. W. New concepts of service, Library Association Record. 58, 1956,

299-305.

McCue, J. H. Online Searching in Public Libraries: a Comparative Study of Perfor-

mance. Metuchen, N. J., Scarecrow Press, 1988.

Medina, S. O. Network of Alabama Academic Libraries interlibrary loan turnaround

time survey. Southeastern Librarian, 38, 1988, 105-107.
36 Investigación Bibliotecológica v.9 number 18 enero/junio 95

Meyer, R.W. Management, cost, and behavioral issues with locally mounted databa-
ses. Information Technology and Libraries, 9, 1990, 226-241.

Mitchell B. J. et al. Cost Analysis of Library Functions; a Total System Approach.

Greenwich, CT, JAI Press Inc., 1978.

Murfin, M.E. and Gugelchuk, G.M. Development and testing on a reference transac-
tion assessment instrument. College & Research Libraries, 48, 1987, 314-338.

Nakamoto, H, Synchronous and diachronous citation distributions. In: Informetrics

88; ed. by L. Egghe and R. Rousseau, pp. 157-163, Amsterdam, Elsevier, 1988.

Nightingale, R.A. A cost-benefit study of a manually-produced current awareness bu-

lletin. Aslib Proceedings, 25, 1973, 153-157.

Nisonger, T.E. Collection Evaluation in Academic Libraries; A Literature Guide and

Annotated Bibliography. Engelwood, CO, Libraries Unlimited, 1992.

Orr, R. H, Measuring the goodness of library services: a general framework for consi-
dering quantitative measures. Journal of Documentation, 29, 1973, 315-332.

Orr, R. H, et al. Development of methodologic tools for planning and managing li-
brary services. Bulletin of the medical Library Association, 56, 1968, 235-403.

Pao, M. L and Warner, AJ., eds. The Depreciation of Knowledge, Library Trends,
41(4), Spirng, 1993 (complete issue).

Peat, Marwick, Mitchell & Co. California Public Library Systems: a Comprehensive
Review with Guidelines for the Next Decade. Los Angeles, 1975.

Pizer, I. H., and Cain, A. M. Objective tests of library performance. Special Libraries,
59, 1968, 704-711.

Roberts, S. A., ed. Costing and the Economics of Library and Information Services.
London, Butterworths, 1985.

Roderer, N. K. et al. The Use and Value of Defense Technical Information Center Pro-
ducts and Services, Rockville, MD. King Research Inc., 1983. AD-A 130805/5.

Rosenberg, K. C. Evaluation of an industrial library: a simple minded technique. Spe-

cial Libraries, 60, 1969, 635-638.

Rosenberg, P. Cost Finding for Public Libraries. Chicago, American Library Asso-
ciation, 1985.

Rubin, R. Inhouse Use of Material in Public Libraries. Urbana, University of Illinois,

Graduate School of Library and Information Science.

Rutledge, J. and Swindler, L. Evaluating membership in a resource-sharing program:

the Center for Research Libraries. College & Research Libraries, 49, 1988,
409-424.

Saracevic, T. et al. Causes and dynamics of user frustration in an academic library. Co-
llege & Research Libraries, 38, 1977, 7-18.

Saracevic, T. et al. Study of information seeking and retrieving. Journal of the Ameri-
can Society for Information Science, 39, 1988, 161-216.

Schofield, J. L. et al. Evaluation of an academic library’s stock effectiveness. Journal

of Librarianship, 7, 1995, 207-227.

Slote, S. J. Weeding Library Collections. Third edition. Littleton, CO, Libraries Unli-
mited, 1989.
 The evaluation of library services: a concise review of the existing literature 37

Stayner, R. A. and Richardson, V.E. The Cost effectiveness of Alternative Library Sto-
rage Programs. Clayton, Victoria, Monash University, Graduate School of Li-
brarianship, 1983.

Stinson, E. R. and Lancaster, F. W. Synchoronous versus diachronous methods in the

measurements of obsolescence by citation studies. Journal of Information
Science, 13, 1987, 65-74.

Sullivan, M.V. et al. Obsolescence in biomedical journals: not an artifact of literature

growth. Library Research, 2, 1980-1981, 29-45.

Sumsion, J. Practical Performance Indicators – 1992. Loughborough, Loughborough

University of Technology, 1993.

Taylor, C. R. A practical solution to weeding university library periodicals collec-

tions. Collections Management, 1(3/4), 1976-1977, 27-45.

Titus, E. M. Barcoding as a tool for collection use analysis: a pilot project. Information
Technology and Libraries, 13, 1994, 257-265.

Trueswell, R. W. Determining the optimal number for volumes for a library’s core co-
llection. Libri, 16, 1966, 49-60.

Urquhart, J. A. and Schofield, J.L. Measuring reader’s failure at the shelf. Journal of
Documentation, 27, 1971, 272-286.

Urquhart, J. A. and Schofield, J.L. Measuring reader’s failure at the shelf in three uni-
versity libraries, Journal of Documentation, 28, 1972, 233-241.

Van House, N. A. et al. Measuring Academic Library Performance: a Practical

Approach. Chicago, American Library Association, 1990.

Van House, N. A. et al. Output Measures for Public Libraries: a Manual of Standardi-
zed Procedures. Second edition. Chicago, American Library Association, 1987.

Welsh, J. J. Evaluation of CD-ROM use in a government research library. Laserdisk

Professional, 2(6), 1989, 55-61.

Wenger, C. B. and Childress, J. Journal Evaluation in a large research library. Journal

of the American Society of Information Science, 28, 1977, 293-299.

White, M. D. and Abels, E.G. Measuring service quality in special libraries: lessons
from service marketing. Special Libraries, 86, 1995, 36-45.

Williams, R. Weeding an academic lending library using the Slote method. British
Journal of Academic Librarianship, 1, 1986, 147-159.