Case study Example of an evaluation of the water and sanitation sector

Example This example is taken from the CE evaluation of the water and sanitation sector which has been conducted currently in seven countries in 2005. Situation This document describes the selection procedure of the countries studied in case studies and the use of data collection tools to identify and assemble information.

REASONS FOR CASE STUDY COUNTRY SELECTION

How and on what basis the 7 target countries were selected has been described in the Evaluation Terms of Reference (ToR). The main selection criteria that were applied in order of priority are as follows: 1. Countries being (in the present or in the past) among the major recipients of EC aid in the W&S sector; 2. Representative of each region - Bolivia is an Asian and Latin America (ALA) country; 3. Having W&S as a focal sector; and, 4. Not having been covered by the latest evaluations conducted by the Unit. The Team were given the opportunity to suggest alternative countries at the Inception Note phase, but after a study of the selection logic and process, this option was not considered appropriate. In the case of Bolivia there was an opportunity to compare differing development approaches (i.e. call for proposals, and sectoral investments support) as well as the influence EC policies had on national W&S strategies, programmes and projects As a guide for the field visits country portfolios were prepared for each of the 7 target countries and those for Bolivia are included in the Desk Phase Report (See Annexes 6.4 and 6.5 of this CN). One of the first tasks of the Team was to review the portfolio for Bolivia, and this exercise demonstrated that much of the data was accurate, and that the projects summary was a reliable description of the ECs involvement in the W&S sector.
DATA COLLECTION Methods used, availability, limits and potential constraints

The main data collection techniques applied during the field visit was comprised of literature reviews, briefings, debriefings, structured and unstructured interviews, group meetings and Focus Group discussions. A field visit to a representative EC partnered project allowed the range of data collection procedures to be expanded, and interviews with key stakeholders and beneficiaries conducted. The combination of these different data collection tools and methods allowed the collection of quantitative and qualitative information on the W&S sector. This was assimilated and used to address the 9 Evaluation Questions (See Section 4). Because of the scale of EC funding of the W&S sector in Bolivia, and the fact that sector support and call for proposals are being used concurrently, it was possible to obtain a comprehensive overview of these different implementation approaches. Being the only Spanish speaking country in the field visit programme was always envisaged as a challenge, and the combined skills of the Team were necessary to overcome the language constraints. On balance this was accomplished satisfactory and most of the key actors were English speaking, and some of the documents were in Spanish and English.
Meetings and briefings

The Delegation of the European Commission in Bolivia (EC Office) were advised by the Unit of the Teams arrival in advance, had been provided with a copy of the ToR, and were familiar with the aims of the field visit. The mission commenced with a briefing of the staff engaged in the sector on the objectives and structure of the Evaluation. Assistance with the collection of information on the principal stakeholders, programmes and projects was requested, and the activity schedule discussed and agreed. Preparatory arrangements had already been put in hand for a focus group meetings, which would assemble all the main actors and enable a joint review of EC funded W&S initiatives to be conducted. A number of key documents and references were identified and made available by the EC Office.

In addition to the actors attending the focus group meetings, interviews were held with Government of Bolivia (Government) officers, and personnel responsible for managing and implementing projects that had a pertinent W&S component or influence. Past, current, and future projects and initiatives were reviewed and information was acquired on their implementation modalities, relative strengths and weaknesses. Links to national government programmes and policies were explored, with the emphasis on EC funded initiatives in particular, but not exclusively. Using information on other donor and member state involvement contained in the Country Strategy Paper1 (CSP) as a guide (Ref Annexes 6 and 6 of CSP), the main W&S stakeholders were identified and met. The most relevant projects were examined and assessed for synergies, clashes, overlap or possible contradiction, in the light of EC sectoral investments. During the site visit numerous meetings were held with villagers, politicians, local administrative functionaries, and womens representatives benefiting from the EC funded Proyecto PRAS, in Santa Cruz. A detailed list of the Persons met during the field visit is included as Annex 6.3. At the end of the mission a debriefing took place at the EC Office and the Teams preliminary findings were presented. Details of the site visit and information on the persons and organisations visited during the mission were provided. Structured and unstructured interviews Interviewing the main stakeholders and beneficiaries formed an important component of the field visit data collection process, and was accomplished through numerous structured and unstructured meetings. These were aimed at gathering general information on the following topics: The W&S situation in the country past, current and projected constraints and challenges; The role of the EC policies, programmes and projects (past, current and future), and the performance of sector investments versus project implementation using the call for proposals procedures; Involvement of other donors and member states in the sector; Institutional and organisational relationships linkages, roles and responsibilities; and, The engagement and role of beneficiaries in W&S service delivery. The topics served as the basis for identifying, isolating, and gathering a range of information on specific W&S issues. The structured interviews were undertaken using the 9 key Evaluation questions, and were supported by unstructured interviews with stakeholders, beneficiaries and other actors engaged in the W&S sector. The latter interviews were used to test and verify information gathered from the former. Site visits After much deliberation and discussion with the Delegation over the project most likely to provide suitable evidence, the Team undertook a field visit to the Proyecto PRAS, in Santa Cruz. The primary aim of the visit was to review a representative EC funded project with a prominent W&S component. Additionally, the recent tendency of migration from the Altiplano to the major cities in Bolivia, and the resulting growth of peri urban populations were taken into consideration in the selection of the site visit. A number of other possible projects were considered but were either logistically inconvenient or completed some time previous. Projects in the latter category would have enabled the collection of information on relevance, impact and sustainability, but many of the project implementers have since dispersed, and
1

Bolivia, Country Strategy Paper, 17th May 2002

effectiveness, efficiency and sustainability would have proved difficult to assess. Details of the visit itinerary, persons met, groups interviewed, and places visited are included as Annex 6.3 The site visit was concerned with meeting office staff (national and international) and reviewing project performance in the light of the 5 key evaluation criteria (relevance, impact, effectiveness, efficiency and sustainability). Visits to representative project sites were conducted with supervision staff, and the appropriateness, quality, and management of works currently under construction were viewed. Meetings were held with the contractors, and managers of the cooperatives responsible for W&S service delivery, and a focus group was conducted comprised of beneficiaries, women and community government representatives. The meetings and interviews were based on the 9 Evaluation questions and the outcome has been assimilated into the General Findings (See Section 4). Focus group discussion Focus groups are used to augment and support other data collection tools and are a scientifically recognised, reliable, and valid method of social research. Its in the nature of focus group discussions that information, while perhaps at times emotive, and often viewed from a singular perspective, is qualitative by nature and rarely quantitative. There were 2 focus group discussions organised during this field mission. These were conducted with a range of actors involved in the W&S sector. The first focus group discussion was held at project level (PRAS, Santa Cruz), with 10 participants. They included 5 female and future beneficiaries of the project, 3 female communication trainers from ANESAPA plus their male coordinator, and the male president of the Neighbourhoods Union in the project area. The focus group was conducted in the Communication Centre of ANESAPA in Villa 1 de Mayo, Santa Cruz, and its primary objective was to carry out an open exchange of information, and gather qualitative and indepth data on project related W&S service delivery. As the project works in the area were not yet delivering services to beneficiaries, the main discussion was focused on acquiring an insight into the pre project negotiations (engagement, mobilisation, cost recovery plans, etc), and getting a feel for community expectations. The discussion was also used to gather data on the projects relevance, examine the socio-economic status of the beneficiaries, and see how gender aspects were being addressed. The focus group concentrated on evaluation questions 1, 2, 3 and 6 (See Section 4) but in the event only a limited amount of relevant information was gained, due to the following: It had been difficult to get hold of women beneficiaries so early in the evening because of their family duties; The project is not yet running and beneficiaries can only give limited view on future expectations, being therefore not able to compare an ex ante with a ex post project situation, and; The information level of the invited persons was limited, showing a clear gap in the projects objective, which is to deliver a continuous flow of information to the beneficiaries. The second focus group discussion was held at national level with 5 senior male representatives of the General Directorate for Basic Sanitation (DIGESBA), a key Government department. They deal with project management, national norms and technology, sustainable development, enterprise development and handle relations with foreign agencies. A moderators guide was prepared for the discussion (See Annex 6.7), which was conducted at the Vice-ministry of Basic Services in La Paz and chaired by the General Services Director, Ministry of Services and Public Works. The primary objective of this focus group was to promote an open exchange of information, and obtain the participants appreciation and views on the effectiveness of the ECs contribution to national

policies and objectives. Discussions addressed the implications of the 3Cs (consistency, coordination and complementarity) as regards W&S service delivery, inside and outside the ECs direct sphere of influence, to benchmark previously obtained quantitative and qualitative information. The 9 evaluation questions formed the discussion agenda (See Section 4), and after a brief introduction, the participants were asked to consider and reply to the questions in turn. They were also asked to give their views on the responses of other actors where applicable. The questions, the respective assessment criteria, and the primary reasons for their selection are briefly summarised below: Questions 1, 2 and 3 are designed to assess the impact and effectiveness of EC support to W&S, and one of these questions addresses the major MDGs; Questions 4 and 5 deal with IWRM, one focusing on improving water governance in accordance with IWRM, the other on the adoption of IWRM in programmes; Question 6 deals with gender, which is a major cross cutting issue associated with the water sector; Question 7 addresses the efficiency of W&S delivery programmes, and, Questions 8 and 9 deal with issues related to coherence, co-ordination complementarity and as such aim to address 2 of the 3 major purposes of this Evaluation. For the second focus group there was a free flow of information and the discussion was considered a success. The Team received a wide range of valid information on the W&S landscape, the ECs role (financial and managerial) in service delivery, the involvement of donors and member states, the institutional and organisational environment, and the engagement and role of beneficiaries in the implementation process. This has been used to frame the preliminary findings (See Section 4).