Paper Number 10 August 2002

Beneficiary Assessment An Approach Described

Lawrence F. Salmen

This publication was developed and produced by the Social Development Family of the World Bank. The Environment, Rural Development, and Social Development Families are part of the Environmentally and Socially Sustainable Development (ESSD) Network. The Social Development Family is made up of World Bank staff working on social issues. This is an updated version of the Paper Beneficiary Assessment: An Approach Described, Environment Department Papers, Social Assessment Series, Paper No. 023, July 1995. Papers in the Social Development series are not formal publications of the World Bank. They are published informally and circulated to encourage discussion and comment within the development community. Copies of this paper are available from: Social Development The World Bank 1818 H Street, N.W.

Social Development Papers

Paper Number 10 August 2002

Beneficiary Assessment An Approach Described

Lawrence F. Salmen

Contents
1. Introduction
1 Rationale Beneficiary Assessment Usage at the Bank 4

2. Involvement of Managers and Related Authorities 3. Assessment Design

5 Understanding the Context Target Population and Sampling Selection of the Assessors Orientation for Assessors Preparation of Final Report 8

4. Methodology

Diret Observation Conversational Interviews Participant Observation

5. Monitoring and Evaluating the Assessments 6. Impact of Beneficiary Assessments 7. Timing and Cost 8. Conclusion Annexes
1. 2. 3. 4. 16 15 13

12

18 Interview Themes Conversational Interview Method Participant ObservationResidence in a Community List of Projects/ESW Utilizing Beneficiary Assessment 23

References

ii

1. Introduction
Beneficiary assessment is a tool for managers who wish to improve the quality of development operations. This is an approach to information gathering which assesses the value of an activity as it is perceived by its principal users. The approach is qualitative in that it attempts to derive understanding from shared experience as well as observation, and gives primacy to the centrality of the other persons point of view. As the Bank and others engaged in development activities seek to do their work better, one key indicator will need to be how the ultimate customer, or intended beneficiary, assesses the value of this work, project or policy, as it affects his or her life. The illumination of how an intended beneficiary appreciates a planned or ongoing developmental activity is the primary objective of this approach. Beneficiary assessment is a systematic inquiry into peoples values and behavior in relation to a planned or ongoing intervention for social and economic change. This method draws heavily from the tradition in social science known as "qualitative research...that fundamentally depends on watching people in their own territory and interacting with them in their own language, on their own terms" (Kirk and Miller). Yet beneficiary assessment also includes direct observation, incorporating simple counting, and is expressed in quantitative terms. The ultimate goal of beneficiary assessment is to reveal the meaning people give to particular aspects of their lives so that development activities may better enhance peoples ability to improve their own living conditions, as they see fit. This demands close rapport between the practitioner of this approach, the beneficiary and the development manager. The beneficiary assessment approach is not intended to supplant the questionnaire survey but to provide reliable qualitative, indepth information on the socio-cultural conditions of a beneficiary population which is intended to be of immediate use to managers and policymakers responsible for improving peoples lives.

Rationale
The key assumption underlying beneficiary assessment is that the people for whom development is intended, the beneficiaries, too often lack a voice loud enough and clear enough to be heard by the managers of development activities: the corollary to this, and just as important, is that the managers, both in developing countries and donor institutions, do not listen to the beneficiaries as well as they might, for lack of training or inclination, or both. The need for intermediation is clear. Sound beneficiary assessment, by encouraging people to express their beliefs and values, leads to development which responds to, while it promotes, the fuller participation of people in their own development. In this way, beneficiary assessment may be seen as forging what is often a missing link in development work, introducing the socio-cultural dimension systematically so that programs and policies are designed and implemented for and by the people for whom they are intended and thus are sustained long after the last disbursement of funds is spent. The need to gain a practicable understanding of peoples values and behavior is essential to any effective development work. The way to proceed and the precise method to employ will vary according to the nature of the development

activity, the beneficiaries, the managers (both host-country and donor institution), and the phase of the activity. Nevertheless, there are general practices which should be followed in all beneficiary assessments and which set the course for this kind of learning. However, deviations from this course in the form of intensive and protracted stays at any point along the way will be made to accommodate the needs of each particular program.

Social Funds
The beneficiary assessment approach has shown its versatility by being effective in a variety of sectors, such as education, health, industry, agriculture and urban development. One particular area of Bank activity where beneficiary assessment has shown itself to be a useful tool is in the design, implementation and evaluation of Social Funds. Beneficiary assessment can ensure that SAPs are demand driven. The approach can be used to contribute to the implementation of a social action program, particularly as regards: Project Selection. The Social Fund has the responsibility to screen and select proposals for micro-projects. The unit staff should be trained in the beneficiary assessment approach in order to: i) assist the implementing agencies to ensure that it is used in proposal design. Consequently, the project should be responsive to community concerns and priorities. This facilitates the screening and selection and reduces the chances of project rejection;

Beneficiary Assessment Usage at the Bank

More than 300 Bank-supported projects in over 60 countries in all regions of the world where the Bank has lending operations have used the beneficiary assessment method. These projects have been in ten sectors. Agriculture Education Energy Environment Industry Multi-Sectoral Population/Health/Nutrition Social Action Program Transport Urban

ii) ensure proper evaluation and screening of a project proposal, particularly in terms of determining its relevance to community needs. Beneficiary assessments have been conducted on SAP's in over 50 countries.

While early assessments were largely for project design or one-shot evaluation purposes, most of the ongoing and planned beneficiary assessments are iterative learning processes undertaken periodically throughout the lifetime of a project. The beneficiary assessment approach has provided insights on the perspectives of intended beneficiaries which were found useful by managers in both the design and implementation of activities in all sectors and regions of Bank operations. In most cases, host-country managers expressed interest in continuing to use the approach. For a description of a broad-based beneficiary assessment conducted on education in Brazil, see Box 1.

Monitoring and Evaluation

Beneficiary assessment plays an important role in SAP's and other projects after the project selection phase. It enables the implementing agency to monitor and address any change in beneficiaries attitudes or unanticipated adjustments that may be necessary to effectively complete the project. It is useful in monitoring the degree of people's participation in the project. Regarding evaluation, the qualitative techniques used in this approach can be used to measure the success of the project in responding to the communities needs and in gauging their level of satisfaction with the project.

Box 1 BA in Education Sector: Lessons Learned from the Brazil Experience Context The experience of international development agencies applying beneficiary assessment to the education sector intensified in 1997. This is when the Found Foundation, the United Nations Childrens Fund, and the World Bank, in collaboration with local, state, and federal governments initiated a process of systematic listening to the key actors in education in Brazil. This systematic listening focused on the urban poor in the states of Pernambuco, Rio de Janeiro, and Sao Paulo. On the basis of this experience, the states of Bahia and Ceara followed with expanded beneficiary assessments of their own in municipal areas of varying sizes. Findings:

In Sao Paulo, middle class teachers had difficulties relating to the fatigue of students from poor families who spent most of their time away from schools working on part-time jobs to supplement their parents meager incomes. In Rio de Janeiro, the prevalence of violence in poor neighborhoods was seen to have an inhibiting influence on school attendance among the poor. This was not well understood by the school personnel living in the more secure communities.

In all regions:

The parents of poor children felt alienated from schools and related activities, as there was little meaningful interaction with school teachers and administrators. The importance of the school director and management in general is sometimes neglected. Yet, observation of more than 50 schools has demonstrated the strong influence of school management. The fact that many students in Brazil are made to repeat classes while not being told of the reason for repetition has resulted in their experiencing severe low self-esteem. They often despair and leave school altogether.

One of the broadest findings of the BA work on education in Brazil to date is also the most basic: children of primary school age see school mostly as a place of socializing while parents view it as a place of learning. Impact In the case of the Northeast Education Project in Brazil, the information gleaned from the pilots in Bahia and Ceara were usefully included in the policy-oriented document, A Call to Action.

2. Involvement of Managers and Related Authorities

While the focus of beneficiary assessment is clearly on the beneficiaries of a development policy or activity, the target is the decision making of the managers responsible for that activity. Recognizing the value of continuous interaction between managers and persons responsible for assessment, there are four times where the involvement of managers is crucial to the success of beneficiary assessment: During conceptualization, so that the manager feels "ownership" for the assessment; At the design stage so that issues in the interview guide, population covered, and quality of interviewers all meet with management approval; During progress review, midway through any particular phase of field work (or at periodic intervals if it is a running assessment) to provide feedback on relevance and utility of initial findings, assist with revision of interview guide and/ or sampling, and feel increased ownership for the assessment process; At the conclusion of the assessment. The final report should first be presented to management orally, subsequently in rough draft. The perspective of management should be incorporated in the final text.

While the target of beneficiary assessment is management decision making, the goal is the improvement of a project or policy. Thus, the persons conducting an assessment should be of service to the managers but not be subservient to them. Similarly, the focus on the beneficiaries should not become an over-identification with their immediate interests.

3. Assessment Design
Understanding the Context
A development program should be understood in light of its history, its architectsindividual and institutionaland the place and people for which it is intended. The first step in any beneficiary assessment is familiarization with the context in which the program of concern is situated. This entails three primary activities: Reading all pertinent documentation, including appraisal reports, supervision reports, evaluations, and consultants' reports. A review of broader country economic analyses would also be useful. Interviews with key persons involved with developing, implementing, or evaluating the program in donor, host-country government and other relevant (academic, consultant, NGO) agencies. Firsthand exploration of sites covered by the program being assessed. Despite the preliminary nature of this stage of the inquiry, the method to be employed follows the naturalistic one of beneficiary assessment itself. Disassociation from officials representing authority so as to avoid identification with them by beneficiaries is advisable. Unstructured conversations with randomly-selected beneficiaries, actual or intended, regarding planned or ongoing intervention will help in planning the more systematic assessment to follow. qualitative methodology employed in this approach, long conversational interviews, often complemented by participant observation, can provide a great deal of understanding from a relatively small number of beneficiaries. Aside from the key factor of the managers judgment, other parameters of the extent of coverage of the beneficiary population will be determined by the variability of the population of concern and the complexity of the program. Given the importance of the cultural variables of values and behavior, sampling should generally be stratified according to ethnicity, class or caste, income, and gender. Additional factorssuch as climate, topography, and land holding patternsmay be included as determinants of discrete population groups for the assessment depending on the nature of the program. While the one-on-one conversational interviews are meant to be quantifiedand hence must be numerous enough to be considered significant by decisionmakers focus-group interviews and participant observation with case studies are also conducted for triangulation and contextual understanding. While most of the subjects of discussion held during beneficiary assessment will be the poor persons for whom a development intervention is intended, attention must also be given to the perspective of the other key actors involved, the program managers, the field staff or extension workers, community leaders, and other relevant stakeholders. Where any of these groups is numerous (field staff such as teachers, health workers, or farm extensionists), representative sampling should be carried out along the same lines as for beneficiaries and stratified according to key variables.

Target Population and Sampling

The degree of coverage of the beneficiary population needs to be sufficiently large to serve as the basis of a management decision. While statistical sampling procedures may serve as a general guide, these will suggest sample sizes greater than those needed for beneficiary assessment. Because of the in-depth

Selection of the Assessors

In beneficiary assessment, more than in most other realms of social science, the human being is the major instrument of learning. The innate human faculties of intuition, perception, and empathy are all crucial to the successful execution of a beneficiary assessment. A capsule description of a good beneficiary assessor would be a practical communicator. More broadly, a person likely to conduct a good beneficiary assessment should be sensitive, respectful, unobtrusive, engaging, open, have good recall ability and writing skills, and, above all, be a good listener. Males and females should both be employed so as to best appreciate the gender perspective. In particular, it is important to remember that in most traditional cultures persons are most apt to address sensitive issues candidly with someone of the same gender. The best assessors are familiar with the particular culture in which the assessment is to take place; it is essential that the assessor have a sound conversational ability in the language of the intended beneficiaries. Given the difficulty of conducting a conversational interview with an unknown person often of modest means in such a way that their responses can be used to improve their conditions, and of recording this conversation in an orderly and intelligible manner, one may think it most difficult to find good beneficiary assessors. Experience demonstrates, however, that sufficient numbers of persons possessing the requisite skills for this work exist in all countries. The pool of applicants is broadened when considering that communication skills are more important than any particular academic discipline. While direction of an assessment should generally be in the hands of someone with advanced university education, interviewers with the proper mix of skills may be high school graduatesthough university level is normally where qualified interviewers are found. The person conducting participant observation,

involving total immersion in the community of beneficiaries, will require greater experience and academic training, generally at the graduate level, than the conversational interviewer.

Orientation for Assessors

To conduct interviews with beneficiaries for optimal results, the selected assessors will need specific orientation and guidance. The orientation should provide clear guidelines as to the goals of the interviews and the role of the assessor. Being a good communicator and listener will not be sufficient. Performing systematic qualitative observations requires additional orientation and preparation. Beneficiary assessors should be given an orientation to strengthen their ability to: write descriptively, and accurately; keep an organized record of field notes; separate useful detail from trivia; and avoid being influenced by any preconceived notions of what responses should be.

Gender-sensitive orientation is essential to gain an understanding of social, cultural and economic constraints faced by men and women. This should be a part of every orientation session even when the project is not dealing with issues directly related to gender. The particular objective is to promote fuller participation of women in all phases of projects by designing development interventions in response to women's felt needs. The orientation should sensitize assessors to the unique circumstances of the community they will be interviewing. Assessors should be well informed as to the general project goals and the context in which they must apply their social science skills.

Preparation of Final Report

Analysis
Assessors need to record the chronology of events, various settings, processes and key issues that arise. Some informal analysis should occur as interviewers are held and observations made. It is possible that the assessor will get ideas about possible analysis as he/she interviews people. These ideas should be recorded in the assessor's notes for reference during the preparation of the final report.

Reporting Findings
The final report should be utilization-focused. When the assessor sits down to write the report, the audience should be kept in mind. The audience will generally be the project manager who will use evaluation results for the project design, implementation or evaluation process. To make an assessment report as readable as possible, it helps to clearly address each major issue, either preconceived or which arose during the assessment. This means that the description, analysis and interpretation for each question is in one place in the report. To be readable and understandable, the report should avoid technical terminology to the degree possible. Assessors need to be very focused to sift through the cumbersome amount of data they have accumulated and select only relevant information. There should be numbers, descriptions and direct quotes to give precision, perspective and life to the information gathered. Assessors' reports should start with executive summaries that include findings and conclusions. The project manager or decision maker often has limited time to read documents; a short, concise report is apt to be more persuasive than a lengthy, detailed report.

4. Methodology
Using qualitative methods in research permits the researcher, or in this case the beneficiary assessor, to look at issues in the round. The assessor can look behind the structured responses obtained in a quantitative survey and gain some perspective with which to balance the numerical data and understand the reasons for the responses. Experience with projects in various countries has shown that qualitative research provides a true-to-life picture of project beneficiaries because the predetermined categories of quantitative methods are absent. This does not mean that there is no systematization for data collection. Qualitative methods used in beneficiary assessment must have a framework that allows for quantification of the data gathered. Given the complexity of this task, the social scientist involved in qualitative research has to be trained even more rigorously than quantitative counterparts. In quantitative research, the validity of the results hinges on accurate research instrument construction. In qualitative methods, validity hinges on the skill and competence of the researcher or inquirer. Any misinterpretation of information obtained through interviews or observations leads to erroneous or invalid data. Quantitative research enables the inquirer to measure the reactions of a large number of people to a finite set of questions, making it easier to aggregate a large number of statistical data. Qualitative research draws from openended interview topics to generate more detailed in-depth data, but often represents a smaller number of respondents. This approach is used in beneficiary assessment because it provides tailored information that can assist project managers to design and monitor development projects. The value of the assessment is in its usefulness in assisting decision makers to make effective interventions. The methods employed in beneficiary assessment are designed to reveal the true concerns of the population affected by a development process in such a way as to provoke managers to change that process for the better. The dominant research mode for this approach is qualitative, earmarked by a premium on listening to others on their own terms where they work and live; the dominant style is one of creativity and flexibility, adapting the particular method of inquiry to the needs and conditions of a given program and locale. But the predominance of the qualitative approach, which builds on empathy between interviewer and beneficiary, should not undermine the importance of representing findings in numerical terms. Beneficiary assessment strives to quantify qualitative information within a descriptive, explanatory context. This approach to developmental learning has three principal methods.

Direct Observation
This is the simplest of techniques. It involves counting, noting behavior and expression, and otherwise registering notable facets of a particular developmental situation. The potential for quantifying observed phenomena is major and immediate (see Box 2).

Conversational Interviews
These are also known as qualitative interviews, the basic tools of inquiry for the practitioner of beneficiary assessment. In well-guided, naturalistic interviewing people reveal their feelings, thoughts and beliefs about a particular issue. "The fundamental principle of qualitative interviewing is to provide a framework within which respondents can express their own understanding in their own terms" (Patton). The conversational interview is structured around a number of themes or topics selected by the managers for whom the assessment is intended

Box 2 Bolivia: A Case of Direct Observation In La Paz, Bolivia, it appeared that a slum upgrading project, which provided water, sewage, and street improvements, served as an incentive for home improvement. Roughly 70% of the inhabitants had borrowed money to improve their homes. To better understand the correlation between the project and the home improvements, three observers were sent to three other neighborhoods in the city which had roughly the same characteristics as the neighborhoods being improved. The observers spent 2-3 days counting the number of houses undergoing improvements in the previous year and found an average of 12 % in the three communities. This finding helped to buttress the case for ancillary development being brought about by infrastructure improvements.

timing of the interview, its duration and time of day, should be determined by what is most convenient and acceptable to the beneficiary. Generally, interviews should not exceed fortyfive minutes to one hour at any one time; to cover the interview guide adequately may require two or three conversations. These shorter interviews during repeat visits show respect for the beneficiarys time and allow increasing rapport to be established. It is important that each interview be conducted so as to elicit precise responses and, once sufficient rapport has been established, suggestions as to how the interviewee feels the situation discussed could be improved. Interviews should be conducted as naturally as possible, without the intrusion of any apparatus which could inhibit the respondent. Generally, the interviewer needs to recall what has been said during the interview and record it in writing shortly after each interview. In certain cultures, however, interviewers are given greater credibility and considered more serious and professional when they have pens and notebooks in hand; some researchers report communities where tape recorders are no obstacle to communication but may lead to greater rapport as the beneficiaries hear their own voices played back to them (much as the intrusion of an instant-developing camera may be overcome by the sight of ones own photograph). As with all aspects of beneficiary assessment, the key is adapting the means of inquiry to the culture of interest to elicit the most cogent message (see Annex 2). For one beneficiary assessment which relied exclusively on conversational interviews (see Box 4).

(see Annex 1); these are issues generally centering on the perspective of beneficiaries about which managers want additional information, or understanding. Interviews may be conducted either one-on-one or in groups (focus group interviews). The individual interview allows for freer expression of certain issues which might be suppressed or distorted in the presence of peers, and lends itself to quantification. One issue about which the value of discreetly, trustworthy communication is of paramount importance is HIV-AIDs prevention (see Box 3). Focus group interviews allow for a coverage of more persons in a given time period and may elicit certain responses from the incentive provided by the presence of peers which would otherwise not have been expressed. The difficulty of precise attribution of response to individual persons in a focus group interview may be a drawback for this method. The key to conducting a conversational interview is establishing good rapport between interviewer and respondents. Once trust is created in the interview setting, information elicited should be valid. Good rapport will generate unsolicited data which may be more important than direct responses to the predetermined topics in the interview guide. Respect for the intended beneficiary is basic to the establishment of rapport. The

Participant Observation
Most of the methodology of beneficiary assessment could be subsumed under participant observationwith the important caveat that it be directed toward managerial decision making in a development context. Participant observation as used here is the protracted residence of an outsider ina community of beneficiaries during which sufficient rapport is established and

Box 3 The Role of BA in Designing the Niger MAP Project: Rationale The purpose of the beneficiary assessment was to gain an understanding of the values and behavior of key population groups in Niger as related to HIV-AIDS so as to increase the effectiveness of the MAP project. Atrisk populations at strategic points throughout the country were targeted using individual and focus group interviews as well as participant observation techniques. The objective was to elicit candor and valid information that would be useful in fighting the disease. The topics addressed using BA methodology are as follows: Knowledge of AIDS, differentiation from HIV Perceived risks associated with sexual behavior Attitude towards persons infect with HIV/AIDS Methods of Prevention, what used and why Blood test for HIV- knowledge of and willingness to take Communications- reactions to messages; channels A total of 4,626 persons were interviewed for this BA during December 2002 and January 2002. The sample interviewed included youth, truck drivers, miners, farmers, barmaids, housemaids, street salesgirls and women-merchants, prostitutes, military, police, customs officials, prisoners, students, parent-teacher associations, persons with HIV, health workers. NGOs and community leaders. Approximately 43% of the persons interviewed were women and 57% were men. Findings Many of the persons interviewed were ignorant about key aspects of this disease. A third of them did not believe the disease existed and 11% were skeptical about its existence. There was confusion on the distinction between HIV and AIDS. There was low use of condoms. In all groups interviewed no more than 30% interviewed used condoms frequently. Among the prostitutes, street vendors, and maids interviewed, only about 8% used condoms. Major needs expressed by the sample were as follows: 91% requested more information about AIDS 83% wanted more information about the blood test 81% of respondents (particularly the youth) said they needed more access to condoms. They cited the poor quality of the condoms on the market. Improved content of HIV-AIDS prevention messages in the media Implications The use of qualitative methodology with quantification of findings, where possible, is particularly appropriate, even essential, for the kind of operational understanding for HIV/AIDS development work. For example, some of the findings of the BA related to fear of blood tests and lack of demand for condoms by prostitutes would most likely have been diluted had questionnaire interviews been used.

10

Box 4 Senegal: Interviewing Businessmen to Understand Private Sector Constraints The purpose of this assessment was to gain an understanding of the constraints which impede the growth of Senegalese business. The assessment was carried out by three staff members of a local graduate school of management, GESAG. A representative sample of businesses from food processing, textiles, chemicals, paper machines, and construction, among others, was used. Findings:

The businessmen felt that the Government impeded business transactions by creating bureaucratic
exigencies and onerous labor regulations. Businessmen resented what they considered to be unfair competition from the informal sector.

Liberalization measures imposed as part of the New Industrial Policy were seen by many businessmen

to have favored the informal over the formal sector. Generally, businessmen felt that those who formulate the policies and create and administer the regulations regarding industrial development had little understanding of and therefore were unresponsive to the needs and concerns of the private sector of Senegal. Impact: The assessment was able to meet its objective of conveying the concerns of a representative group of Senegalese businessmen. The findings were put to use by the Bank and the Government as they jointly acted to develop the private sector of Senegal.

involvement in their everyday activities occurs to allow for the kind of representation of their living conditions which will enhance the development potential of the community occupied. Unlike participant observation in social anthropology, where residence in a community being studied may last two or three years, the participant observer in a beneficiary assessment generally resides in a community of beneficiaries for one or two weeks to two or three months. The interest here is focused on the same topics which make up the interview guide and other topics which make up the interview guide and other topics considered by managers to be relevant to their concerns but too sensitive or too pervasive to lend themselves to the visiting interviwer format. The emphasis in participant observation, however, is not only on the topics perse, but on hwo these issues are affected by the socio-cultural and political context in whcih the beneficiary lives and works (see Box 5). To best illustrate this contextual dimension of the topics of interest, the participant observer generally conducts case studies on a small number (usually 5 to 10) of benefiiary households who are visited

many times over the course of residence in the community (for an elaboration of participation observation method, see Annex 3.).
Box 5 Ethiopia: Cultural Inhibitions to Prenatal Care Participant observers in the beneficiary assessment done in one major region of Ethiopia observed that pregnant women in this area almost never visited health centers. In-depth discussions held with men and women in the communities where they lived revealed that the major reason for this low visitation rate lay in the cultural belief that it was considered weak and improper for women to admit to any pain or discomfort. This information, which was new to the public health officials in Addis Ababa, was considered useful to help orient health education among the rural communities of this region of Ethiopia.

11

5. Monitoring and Evaluating the Assessments

Given the non-standardized, tailor-made nature of beneficiary assessment, and the sensitive nature of much of the information gleaned, it is necessary to monitor beneficiary assessments closely. Monitoring should be done by persons not directly associated with the program being assessed, yet familiar with its essentials who are well versed in the techniques of beneficiary assessment. Monitoring should always be done in conjunction with program management. The two key times for monitoring beneficiary assessments are roughly one-third through field work, when enough information has been collected to judge its utility and enough of the sample is still to be interviewed so that a revised assessment will be useful and, at the end, for the final report. A third possible monitoring point would be during the final phase of data tabulation, to see that recorded information is properly categorized and analyzed. Total times for monitoring should not exceed three or four weeks, assuming one or two brief visits during the execution of the assessment (See Box 6). Proper evaluation of the beneficiary assessment exercise is critial to determine the effectiveness of an intervention. Through conversational interviews and direct observation it will be possible to determine the response of the commuity to the completed projects. While beneficiary assessments are conducted to assist project management, their ultimate gola is to increase the effectiveness of activity directed at poverty reduction. Thus, the poor are both subject and object of beneficiary assessment; as such, it is important to share findings of the assessements with the members of the communities where these findings have been made. This will act as a check on their validity and, most
Box 6 Bolivia: An Example of the Importance of Monitoring The importance of the monitoring function can be illustrated in a Bank evaluation of an agricultural project in Bolivia which was using the beneficiary assessment approach. The introductory part of the assessment had been done through correspondence. Approximately one-third of the way through the assessment, it was discovered that a number of the interviewers had conducted very rich, lengthy interviews with farmers which went back to their childhood years prior to the revolution of 1952, but which had nothing to do with the objectives of the project. After an appropriate reorientation, the interviews were streamlined, brought up to date and directed to the evaluative goals of the assessment at hand.

important, inasmuch as it is the people's story that is being told, it is only right that they should be the first to critique it.

12

6. Impact of Beneficiary Assessments

Very often, new information and insights are generated in the process of implementing a beneficiary assessment. This was the case in one project assessed for which, from this new information, new regulations were made in project communities, benefiting previously untargeted poor persons. In La Paz, Bolivia, as a result of a slum upgrading project, infrastructural improvements were made so that the residents had access to water, sewage and other facilities. Through participant observation, it was learned that home owners who benefited from these infrastructural improvements often failed to extend connections to renters. Authorities promptly established an ordinance requiring all landowners to make connections for each and every household resident, regardless of tenancy status. In Malawi a beneficiary assessment has helped make the electric utility (ESCOM) more responsive to its consumers (see Box 7). For an amplified description of a beneficiary assessment on health in Lesotho and its impact (see Box 8).

Box 7 Malawi: Promoting Electricity Usage in Rural Communities The beneficiary assessment in Malawi had the objective of identifying socio-cultural factors as related to electric utility performance in rural settings. Three research teams, each comprised of a man and a woman, spent six weeks, one in each of three sites selected in collaboration with ESCOM (the local utility) on the basis of a) geographical location and b) success as a rural electrification scheme. Findings: Government officials homes (policemen and primary school teachers) are not connected to nearby electric lines. Even though they are willing and able to pay for ongoing changes, they are reluctant to pay the high installment costs, given the possibility of relocation. People are unaware of the real cost of electricity and overestimate it by as much as two to three times. Communication between ESCOM and its clients leaves much to be desired. While technical matters are commonly felt to be handled expeditiously, the delays in connecting houses, misunderstood readings and complicated electricity bills all give rise to distrust and alienation from ESCOM on part of clients. While no precise data on employment generation is available, it seems that electricity clearly was a boon to economic activity at the local level. Impact: The major impact of this assessment is that it helped ESCOM be more consumer oriented than it had previously been, resulting in attracting more customers who had the means to pay for the services but did not do do so due to some misconceptions about utility services. It also offered the government of Malawi the occasion to express its support of electricity as a development good which is becoming ever more important to the people of Malawi as a substitute for increasingly scarce fuel wood.

13

Box 8 Lesotho: The Impact of Beneficiary Assessment on Health Sector Some years ago, the Ministry of Health in Lesotho had the goal of expanding the provision of modern health services to the countrys population. The World Bank undertook to assist the Ministry towards the achievements of health for all by the year 2000. It was understood that to achieve this goal it was necessary to learn the attitudes, concerns and customs of the people to design an appropriate health care system. An in-depth qualitative analysis of individual and household behavior was conducted using beneficiary assessment techniques. This approach was considered to be particularly relevant, given the sensitive nature of the information needed, such as in family planning practices. Three communities were selected. The participant observer method was chosen as the main methodology. The participant observers were given two weeks of training and then were sent to live for approximately two months in the communities where they were to conduct interviews. Representative samples of close to 50% were interviewed in three communities. Supplementary interviews were carried out at clinics. The following fundings emered from this assessment that would not have surfaced through traditional questionnaire survey-techniques: The government village health worker (VHS) program failed because villagers did not use their services, which were largely directed towards preventative health care. The VHS were not given any curative remedies, even of a simple nature; this considerably lowered their status in the eyes of the villagers. Traditional healers, on the other hand, were sought out by villagers because they had curative remedies, herbs and such, for immediate use. Rural women often become pregnant because they did not have access to a steady supply of contraception and having to travel long distances for supplies deterred usage. Furthermore, the husbands were opposed to contraception, making it necessary for the women to hide their supplies. People were very dissatisfied with the quality of treatment and level of respect they received from doctors and other health professionals in hospitals. The poor were excluded by fees beyond their means and the well-to-do were getting services at what they considered to be low prices. Impact of Assessment (project design): The government village health workers were provided with aspirin and other simple remedies to facilitate interaction with the villagers; The traditional healers were brought into the national health system and given courses in basis health; Contraceptives were made more accessible to women; and The need to categorize and charge patients according to socio-economic status was recognized.

14

7. Timing and Cost

The duration and cost of beneficiary assessments will vary markedly according to the nature of the program being assessed. From experience with World Bank supported projects, beneficiary assessments have generally been conducted within a three to four month period, from design to the presentation of the final report. The breakdown of activities for the time period is generally as follows:
Activity Weeks

Pre-test; training Survey; participant observation Data tabulation Preparation of final report Total

1-2 6-8 2-2 3-4 12-16

Beneficiary assessments have cost an average of roughly US $40,000. Both duration and cost will increase when the assessment is done, as is more often the case at present, in iterative fashion, at periodic intervals throughout the life of the project.

15

8. Conclusion
By amplifying the voice of the people for whom development is intended, beneficiary assessment empowers these people to help themselves. By showing development managers the value of improved communication with beneficiaries, this encourages the formation of institutions more open and more responsive to their client populations. The ultimate test of the worth of beneficiary assessment is the creation and durability of bridges of understanding between managers and beneficiaries such that people subject to change ultimately become the artchitects of their own development.

16

17

Annex 1. Interview Themes

Conversational interviews are created around themes of importance to managers. Below are generic interview themes used in beneficiary assessments in the areas of health, education and energy. Education a. b. c. d. Expectations from education Schooling for girls Role of religion (obstacles to attendance) Quality of education received (i) degree of learning (ii) utility of what is learned e. Distance from home to school f. Physical aspects, maintenance, furniture, general appearance g. Language (for example, French, Arab, national) h. Economic situation: is school seen as aid or hindrance to productivity Health and Population a. b. Health problems affecting family Perception of health service provided by (i) government (ii) NGO's (iii) traditional healers c. Degree and reason for attendance at health center d. Problems of village (agricultural production, water, education, etc.) e. Family size f. Use of family planning methods and meaning of the same
18

g.

Suggestions for improving well being of family

Energy Topics for Discussion for Electricity Users: a. b. Process involved in obtaining utility service Cost of acquiring connection and how paid for c. Users of electricity in the household, business, institution and how often it is used d. Location where meters are installed and how often and accurately they are read e. Problems encountered paying utility bill; perceived fairness of bills f. Quality and efficiency of repair service g. Level of satisfaction with service h. Handling of complaints filed with the utility i. Attitude towards utility company; how service can be improved Topics For Discussion With Potential Users Interest in acquiring electricity service and how it would be used; perceived advantages and disadvantages b. Cost of being connected and how it compares with the cost of other energy sources c. Perceived problems with paying bills d. Existence of user cooperatives, for example, farmers clubs or womens clubspotential as vehicle for credit for electricity e. Occurrence of illegal connections and how users feel about it f. View of utility company and services it provides; suggestions for improvements a.

Annex 2. Conversational Interview Method

What has been referred to here as a conversational or qualitative interview is simply a way to broach a topic of interest in a natural manner, as part of a normal flow of discourse, such that the respondent will be at ease and induced to address the topic freely and with candor. Sometimes direct questions, especially on sensitive topics, put people on the defensive and lead to responses which are considered to be in the respondents best interests or what will please the interviewer. Indirect interview methods are meant to elicit more valid expressions of opinion, or of fact. In a beneficiary assessment of an urban development (upgrading) project there were a number of issues which lent themselves to conversational interview techniques. Three of these issues are presented with the kind of comments which the interviewer may make to encourage a freeflowing dialogue: Adaptation of the project to the needs and ties of the beneficiaries "If you were designing a project to improve this neighborhood what would you have done?" prioriof course they are technicians and do not live here as you do. Perhaps they missed something..." Degree to which people identify with the project "This is a big project. The municipality is making many improvements. Of course, a poor neighborhood like this will need many more improvements and maintenance once these works are finished. I wonder who will do this later work..."

"How is all of this infrastructure going to be maintained?" "These payments that you are being obliged to make (for improvements) what is the idea behind that?"

"Whose project is this, anyway?" (Too direct for early in conversation but may be a natural comment after rapport has been established.) Suggestions for project improvements "Just as a visitor here, Im impressed with all this project is doing to transform the neighborhood. I dont know how it could be any better?" "As in previous issues "Imagine you are in charge of this project, what would you do to improve it?" "So, youre happy with what the project is doing; couldnt be better..."

"Clearly, many major changes are being made here (as a result of the project), the streets and water (or public light ing, sewage system, schools, etc.) for instance. But then, I do not live here as you do, so its hard to tell what really matters most..." "Certainly the designers of this project thought they were responding to your needs, but

19

Once the respondent is provoked to give his or her opinions on a certain topic it is important that the interviewer not appear to be overly inquisitive or judgmental. The stance is one of being interested, supportive, respectful and somewhat innocent, or naive. Comments such as "really" or "thats most interesting" or "well, I had no idea" are the kind of supportive interjections which should lead to further elaboration of the issue being discussed.

This is the peoples story about a project that is changing the place where they spend the days and nights of their lives. The interviewer wants to hear this story and encourage its development. The respondent should be helped to understand that his or her opinions will be transmitted to the project management so that the project may more closely respond to the intended beneficiaries needs and values.

20

Annex 3. Participant Observation Residence in a Community

During the stay in a community, it is essential to get to know the key actors in the development program well: the beneficiaries in general; their leaders, formal and informal; and the key administrators of the implementing agency. Attempt to keep the interrelationship with each of these three major groups somewhat discrete. Ones manner of relating is decreasingly professional from program administrator to leader to regular beneficiary. While all know the reason for ones stay in the neighborhood, the official nature of that stay is to be kept most apparent to the program personnel, with whom a certain degree of personal distance must be maintained to avoid bias, real or alleged, intentional or unconscious. The leaders are also aware of the motive for ones presence, especially at the outset; less so over time. The people, however, while informed of ones reason for living in the community, should primarily get to view the participant observer as a neighbor and, to varying degrees, friend. At the level of the people, the relationship is far more personal than professional. The participant observer should cultivate a few close contacts from diverse, major segments of the population. These should represent various key income groups, political factions, owners and renters (where the latter are a significant part of population), etc. One should never be over-identified with any one group, but be open and accessible to all: diplomacy at the neighborhood level. Participant observers should participate in major organizations and activities of the community sufficiently to be appreciated and identified as a participant but not so much as to become overcommitted. The aim is to retain ones independence, yet demonstrate involvement.

21

22

References
Agar, M. 1986. Speaking of Ethnography. (Sage University Paper, Qualitative Research Methods Series, (Vol. 2,) Beverly Hills, CA: Sage. From the Natives Point of View: On the Nature of Anthropological Understanding. In C. Geertz (Ed), Local Knowledge: Further Essays in Interpretive Anthropology. N.Y.: Basic books, 1983. Chambers, R. 1988. Poverty in India: Concepts, Research and Reality. Discussion Paper 241, Institute of Development Studies, Sussex, England. Glaser, B., and A. Strauss. 1967. A. The Discovery of Grounded Theory. Chicago: Aldine. Guba, E.G.and Lincoln, Y.S. 1981. Effective Evaluation: Improving the Usefulness of Evaluation Results Through Responsive and Naturalistic Approaches. San Francisco: Jossey bass. Havens, Lester, L. 1983. Participant Observation. New York: Jason Aronson. Hirschman, A.O. 1985. Getting Ahead Collectively: Grassroots Experiences in Latin America. Pergamon, New York. Kirk, K. and M.L. Miller. 1989. Reliability and Validity in Qualitative Research, Sage University Series on Qualitative Research Methods, Vol. 1. Sage, Beverly Hills, California. Korten, K. and R. Klauss. 1984. People-centered Development. Kumarian, West Hartford, Connecticut. McCall, George J. and J.L. Simmons, eds. 1969 Issues in Participant Observation: A Text and Reader. Reading, Mass: Addison-Wesley. Merton, Robert K., Marjorie Fiske, and Patricia L. Kendall. 1956. The Focused Interview. Glencoe, IL: Free Press. Miles, M.B. Huberman, A.M. 1984 Qualitative Analysis: A Sourcebook of New Methods, Beverly Hills, CA: Sage. Operations Evaluation Department (OED). 1985. Sustainability of Projects: First Review of Experience. World Bank. Patton, Michael Quinn. 1980. Qualitative Evaluation Methods. Beverly Hills, California: Sage. Rondinelli, D. 1983. Development Projects as Policy Experiments: An Adaptive Approach to Development Administration. Methuen, New York and London. Salmen, Lawrence F. 1987. Listen to the People: Participant Observation Evaluation of Development Projects, New York: Oxford University Press. Salmen, Lawrence F. 1998. Toward a Listening Bank: A Review of Best Practices and the Efficacy of Beneficiary Assessment, Social Development Paper No. 23, World Bank. Salmen, Lawrence F. assisted by Misgana Amelga. 1998. Implementing Beneficiary Assessment in Education: A Guide for Practitioners (with examles from Brazil), Social Development Paper No. 25, World Bank.

23

Salmen, Lawrence F. 1999. Beneficiary Assessment Manual for Social Funds , Social Protection Discussion Paper No. 9930, World Bank. Salmen, Lawrence F. 2000. Beneficiary Assessment for Agricultural Extention: A Manual for Good Practitice, Rural Development , Agricultural Knowledge & Information Systems (AKIS) Good Practice Note, World Bank. _______. 1989. "Beneficiary Assessment; Improving the Design and Implementation of Development Projects." Evaluation Review. Vol. 13, No. 3. _______. 1991. "Reducing Poverty," Public Administration and Development, Vol. 11. Spradley, J. 1979. The Ethnographic Interview. New York: Holt, Rhinehart & Winston. Participatn Observation. New York: Holt, Rhinehart & Winston, 1980. Stone, Linda and J. Gabriel Campbell. 1984. "The Use and Misuse of Surveys in International Development: An Experiment from Nepal." Human Organization 43, No.1. Verspoor, A. 1989. Pathways to Change; Improving the Quality of Education in Developing Countries. World Bank, Washington, D.C. Whyte., William F. 1984. Learning from the Field: A Guide from Experience. Beverly Hills, CA: Sage.

24