IDA13

Measuring Outputs and Outcomes in IDA Countries

International Development Association

February 2002
 Measuring Outputs and Outcomes in IDA Countries

Introduction......................................................................................................................................1

Establishing a Measurement System ...............................................................................................2

Establishing and Estimating Early Indicators of Country Output ...................................................4

Reviewing Progress..........................................................................................................................6

Annex 1: IDA Countries to Which Output Estimates Would Apply..............................................7

Annex 2: Primary Education Completion Rates for the Selected IDA Countries ..........................8

Annex 3: Measles Immunization Coverage ..................................................................................10

 Measuring Outputs and Outcomes in IDA Countries

Introduction

1. In the course of the IDA13 discussions, the importance of measuring the effectiveness of
IDA’s work in poverty reduction has been discussed in depth by donors. Also a theme of the
OED review of IDA10-12, it was felt that systematic and more comprehensive measurement of
IDA and country inputs and country outputs, would be an important element as IDA embraced
the PRSP process, and sought to assist countries in implementing, and measuring the
effectiveness of their poverty reduction strategies. The IDA13 report reflects several ways in
which such measurement -- especially as they relate to the Millennium Development Goals
(MDGs) -- will be implemented over the IDA13 period.

2. At the Montreux meeting of IDA13 Deputies, the US also raised the possibility of donors
providing additional contributions linked to specific milestones related to measuring and
monitoring inputs and outputs, and also suggested the possibility of estimating and targeting
selected outputs in advance. Many donors agreed that further exploration of this issue was
warranted. (Traditionally, donors review IDA’s performance and policies and country results
every three years in the course of replenishment discussions and base their replenishment
contributions on this assessment. The suggested approach would endeavor to shorten the time-
frame over which IDA reported on its performance.) Management agreed to do further work on
how a measurement and monitoring system could be established over the course of IDA13, and
milestones set that could be reviewed by donors (and possibly linked to additional donor
contributions) during the IDA13 period.

3. IDA provides its assistance to its borrowing countries through policies and programs,
which are IDA inputs to country-owned development strategies. Such inputs include credits,
country dialogue and analytical economic and sector work (ESW). Inputs from IDA (and other
development partners) influence reform programs undertaken by countries, which are country
inputs. Country inputs include, for example, programs and development investments undertaken
by countries to improve fiduciary management, initiatives to improve the quality of education,
policy reforms to improve the investment climate, etc. These reforms, or country inputs, in turn
lead to outputs, such as, for example, reduction of barriers to private investment, school
enrollment, and immunization. Ultimately, these outputs, over a period of time, lead to
outcomes -- which are overall country measures such as gains in literacy, increased life
expectancy, and lower infant mortality. To complete the circle, IDA’s strategic choice of its own
inputs is driven by an analysis of their likely impacts -- via country inputs, outputs, and
outcomes. Management looks to leading indicators that are correlated with final outcomes to
help ensure IDA’s strategy is on track, and to guide adjustments to that strategy as necessary.

4. IDA’s own inputs are obviously under the control of the institution, and are immediately
measurable. They represent the only set of instruments at IDA’s disposal to influence outputs,
and are for that reason the primary focus of donor deliberations and guidance during the
replenishment process. Country inputs are influenced by IDA inputs, and those of other
development partners, but they depend on borrower government commitment and actions and,
subject to the latter, tend to follow IDA inputs with short lags. Outputs, which result from this
process, are generated with lags of several years after IDA and country inputs are undertaken and
 -2-

are to an even lesser degree attributable solely to IDA inputs; they are dependent on many
exogenous factors over a longer period of time which sometimes mitigate or even reverse IDA’s
own efforts. Finally, the relationship between outputs and outcomes is multi-dimensional. For
instance, lowered infant mortality is at least as much a function of access to safe water and
education of women as it is of immunization.

5. IDA’s inputs are already subject to measurement through a number of reports that track the
quality of its interventions and the volume of lending, and the linkage of IDA lending to country
performance. In addition, the recent OED review of IDA also evaluated the extent to which IDA
complied with the policy guidance of its donors in implementing its programs. Outcomes are
measured by the World Bank and partner organizations on a periodic basis, and are reported
annually in well-known reference publications such as the World Development Indicators.
Within that broader context, the approach taken in this paper is to explore the possibility of
establishing (a) a phased approach to measurement of country inputs and outputs, (b) a basis for
estimating some key output indicators that could be expected to be achieved over the next two
years or so, and (c) a framework for reviewing progress.

Establishing a Measurement System

6. In order to focus the measurement system on key aspects influenced by IDA’s work, it
would be appropriate for the system to concentrate initially on indicators related to fiduciary
management, private sector development, health and education. These areas are of critical
importance to poverty reduction through good governance, pro-poor economic growth, and
improved social well-being. Within these broad areas, the indicators to be measured need to be
set up with due regard to issues such as attribution (the direct correlation to IDA’s actions), the
time lag between IDA and country inputs, and even more so the time lag between investments or
policy changes by IDA and IDA countries, and measurable outputs. The system will also need to
take account of difficulties in measuring and defining quality -- especially with regard to outputs,
and especially as data is aggregated across countries, regions, and globally.

7. Moreover, in measuring outputs, IDA needs to coordinate its efforts with individual
monitoring efforts by countries that are being established as they implement PRSPs, and in the
context of the MDGs which are being targeted by countries as they formulate their strategies.
Thus IDA’s own measurement system should be geared to measuring outputs which are related
to and support these objectives, and to tracking indicators that are broadly viewed as key to
development effectiveness.

8. With regard to country outputs, moreover, the availability of data on results will be to a
large extent driven by periodic and often infrequent country survey results carried out by IDA’s
partners. For example, data on education outputs is based on surveys conducted by UNESCO,
which are often 5-7 years apart in any given country. Similarly, the Joint Monitoring Program
(JMP) utilizes the combined resources of UNICEF and WHO to monitor water and sanitation
coverage in all UN member states. JMP has reported outcomes in 1990, 1992, 1993 and 1996,
and in addition in the year 2000 it published a comprehensive review of progress since 1990 in
coverage of safe drinking water and sanitation. The data is now available on a database
(although some of it remains contentious) and is updated on a triennial basis.
 -3-

9. The choice of the indicators to be measured will undoubtedly influence the way in which
the development process assigns priorities. The process will also, unless managed carefully,
detract from the importance of quality improvements of the inputs which are the basic instrument
at the disposal of IDA to influence country outputs. Thus a system measuring country inputs and
outputs should be looked at in the context of the broader development effort undertaken by IDA
through its policies and programs.

10. On the basis of the above, a measurement system that has as a first phase the measurement
of country inputs, and is extended gradually in the second phase to country outputs could be
established. The first phase, for selected country inputs, could be put in place during the course
of 2002 and the second phase, to gradually extend the system on the basis of further work to
include selected country outputs, could be put in place by end-2003.

Phasing of a Measurement System

Phase 1 Establish a Measurement System that would

report on country inputs in Fiduciary Management,
To be established by end-2002. PSD, Health and Education.

Phase 2 Extend Measurement System to report on country

outputs in Fiduciary Management, PSD, Health
To be established by end-2003. and Education.

11. Specifically, the measurement system will be set up to begin measuring the following
country inputs in IDA countries with active programs by the end of 2002.

Fiduciary Management: number of public expenditure, financial management and

procurement systems improvement programs being implemented;

Private Sector Development (PSD): number of programs designed to improve invest

climates;

Education: number of education initiatives underway; and

Health: number of health sector programs underway.

12. The measurement system would be extended to begin measuring country outputs in
countries with active IDA programs by the end of 2003. Indicators to be measured could be, for
example, as follows:

Fiduciary : effective use of and accountability for public funds through appropriate
public expenditure, procurement and financial management systems. Specific measures,
such as budget comprehensiveness, financial input and control systems and internal and
external transparency will need to be defined building on, but not necessarily limited to,
the indicators being established for the HIPC tracking system;

Private Sector Development (PSD): policy reform indicators such as the extent of
procedural impediments and time/cost measures on business startups and collateral
 -4-

registration; and growth indicators, such as productivity and share of private credit in
GDP.

Education: completion rates for primary school, gross and net enrollments, secondary
enrollment, secondary completion rates, measures of learning, ratio improvement --
student-teacher, pupil-textbook, pupil classroom;

Health: immunization rates, indicators such as children sleeping under insecticide

treated bednets, children under age 5 who are underweight, proportion of births with
skilled attendant, share of population with access to improved water sources.

13. Over time, it would be expected that the output indicators being defined would be further
refined and revised to include other outputs that are viewed as central to economic and social
development in IDA countries. Moreover, the CPIA process which assesses country inputs (both
policy and institutional) should continually be assessed to ensure a tight linkage to the outputs
being viewed as important, and thus the annual update of the CPIA system should take into
account the country output indicators being tracked.

Establishing and Estimating Early Indicators of Country Output

14. The system proposed above would provide a basis to begin progressively measuring
changes in country output indicators over time, and would be closely integrated into a broader
World Bank program, which management is planning, to strengthen and improve the
measurement of the Bank’s development performance. To gain experience in making more
timely performance assessments based on development outputs, Bank staff have been working to
develop a limited set of early indicators of output performance which have good linkages,
ultimately, to outcome objectives expressed in the MDGs. The objective is not to substitute a
few headline numbers for the careful tracking of development performance. Rather, it is to
identify a limited set of measurements which have, in the professional judgment of specialists,
two characteristics: that they are reasonably reliable early indicators or proxies for important
trends in a given sector or area of development effort; and that they are tracked, enumerated and
reported with relatively good consistency, frequency and accessibility. The approach would be
to focus on a sub-set of output indicators selected from the broader set of education, health and
private sector development measures discussed earlier, and then try to push to develop early
tracking results for this limited sub-set.

15. These quantitative estimates should not be construed as targets, per se. As noted earlier,
achieving outputs will depend on several factors, and hence it is important to weigh progress
toward the desired results in a country and global context. Moreover, country outputs will reflect
considerable lags vis-à-vis both IDA and country inputs. Thus a short-term measure of outputs is
a poor indicator of IDA’s recent efforts, but will, more likely, reflect the result of resources
provided many years earlier. Development outputs, such as those outlined below, which could
be measurable in IDA13 during 2002 and 2003, for example, are already “set” and largely
impervious to being affected by action now and in the immediate future: such results wo uld
need to be related to IDA funding and programs from replenishments prior to IDA12, as well as
to the other factors affecting them in these earlier years.
 -5-

16. Since IDA allocates its resources on the basis of performance, the linkage between IDA
inputs and outputs would be strengthened by focusing the estimates for outcome measures on the
group of countries that constitute the top three quintiles in the 2001 CPIA exercise. 1 Moreover,
quantitative estimates for this purpose should be based on currently measurable outputs (using
data for benchmarking as recent as possible to December 2001) that are central to developmental
impact, and should be set at a level that is both challenging and realistic. As the first step in this
exercise, the following output indicators could be targeted in Education, Health and Private
Sector Development.

Education: Achieve improvements in the primary education completion rate. This output
indicator is increasingly being used by the World Bank and other agencies as a core indicator of
education system performance. It is calculated as the total number of students successfully
completing (or graduating from) the last year of primary school in a given year, divided by the
total number of children of official graduation age in the population. 2

The average annual growth rate of the primary completion rate (population weighted)
would be improved. The annual average growth rate, using currently available data is -
0.23%. Target improvement : the average annual growth rate would be increased to 0.5%
using data available in December 2003. This would imply a turnaround of 0.73
percentage points.

The number of countries (that do not have universal primary education) where average
annual growth exceeds 1%. On the basis of current data there are 10 countries in this
group. Target improvement : at least a 50% increase -- to 15 countries based on latest
available data in December 2003.

Health: Achieve improvements in measles coverage. Measles immunization is one of the core
intermediate determinants for achieving the MDG of child mortality reduction.3

The overall coverage rate (population weighted) of measles immunization would be

improved. The coverage rate (based on currently available data) is slightly below 60%.
Target improvement: an increase to 62% on the basis of the latest available data in
December 2003.

The number of countries where coverage has reached 80% -- a threshold beyond which
measles can be contained. Currently there are 22 countries in this group. Target
Improvement: increase the number of countries to 26 on the basis of the latest available
data in December 2003.

PSD: Achieve improvements in two measures that are key to investment climate reform and
which will be evaluated in country investment climate assessments:

1
The countries are listed in Annex 1. Moreover, the outputs are calculated subject to availability of relevant data
for these countries. As noted earlier, output data are not universally available in these countries.
2
Annex 2 provides the available data in the selected IDA countries to calculate these statistics, and the current
average annual growth rate.
3 Annex 3 provides data on immunization rates for the selected IDA countries.
 -6-

Time required for business start-up (registration and licensing procedures): Measures the
time required to obtain legal status to operate a firm, in business days. Time delays in
registration are a major impediment to formal entry of private firms. Delay deters
participation in the formal economy and increases the potential for corruption (“speed
payments”). Further, time delays in business entry are associated with lower income
growth. Baseline measure: 81 days in the specified IDA countrie s (see Annex 1). The
corresponding measure for all OECD countries is 32 days. Target Improvement: 10%
reduction in the number of business days in the specified IDA countries (weighted by
population) from December 2001 to December 2003 based on most recent available data
(generally a 4-6 week lag).

The formal cost of business start-up (registration and licensing procedures): Measures
the total official cost of obtaining legal status to operate a business, as a percentage of
GDP per capita. The cost of registration is a major barrier to entry in poor countries,
deterring formal participation in the economy. Further it is negatively associated with
income growth. Baseline measure: 73% in the specified IDA countries (see Annex 1).
The corresponding measure for all OECD countries is 17%. Target Improvement: 10%
reduction in percentage cost in the specified IDA countries (weighted by population)
from December 2001 to December 2003, based on most recent available data (generally a
4-6 week lag).

Reviewi ng Progress

17. Timing : A report could be provided to IDA’s Executive Directors by December 2003 on
the establishment of the measurement system, and progress toward meeting specific estimates for
selected output indicators. Also, a donor review could be achieved by scheduling the IDA13
mid-term review to facilitate such a discussion in December 2003 or early 2004. This would
however, mean delaying the mid-term review considerably -- and it would take place shortly
before the IDA14 discussions would be expected to begin (Spring 2004). Alternatively, an
earlier date could to be chosen, but this would obviously reduce the scope of the measurement
system that could be implemented, and the estimates that could be reached.

18. Learning Process: This approach to setting up a system and the initial indicators to be
measured has been arrived at on the basis of both pragmatic considerations -- i.e., what is
available, and can be tracked fairly accurately, as well as selecting indicators that would be best
measures of successful outcomes. As experience is gained with this approach, and more data
becomes available as countries implement monitoring systems under their PRSPs, it will be
important to periodically reassess the components of the measurement system. Thus, in the
course of the review it would be important to assess again the indicators being measured, and to
revise and add to them as appropriate in order to continue to strengthen the link between IDA’s
efforts and country outputs and outcomes.
 -7-

Annex 1

IDA Countries to Which Output Estimates Would Apply

(Constitutes IDA countries in top three CPIA 2001 quintiles)

Albania Madagascar
Armenia Malawi
Azerbaijan Maldives 1/
Bangladesh Mauritania 1/
Benin 1/ Moldova
Bhutan 1/ Mongolia
Bolivia Mozambique
Bosnia & Herzegovina Nepal
Burkina Faso Nicaragua
Cape Verde Pakistan
Dominica 1/ Rwanda 1/
Eritrea 1/ Samoa 1/
Ethiopia Senegal
Georgia Sri Lanka
Ghana St. Lucia
Grenada 1/ St. Vincent 1/
Guyana Tanzania
Honduras The Gambia 1/
India Uganda
Indonesia Vietnam
Kenya Yugoslavia FR
Kyrgyz Republic Zambia
Lesotho 1/

Note:
1/ The following IDA countries from the top 3 CPIA quintiles are not included in the
benchmarks for the PSD measure due to missing data and thus will not be included in the
targeted improvement: Benin, Bhutan, Dominica, Eritrea, Grenada, Lesotho, Maldives,
Mauritania, Rwanda, Samoa, St. Vincent, and The Gambia.
 -8-

Annex 2

Primary Education Completion Rates for Selected IDA Countries

Average
Total Annual Change*
Population Year (percentage
First Quintile (year 2000) 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 points)
Bhutan 805000 7 23 2.67
Uganda 22063000 39 40 0.20
Maldives** 276000
Sri Lanka** 19360000
Mauritania 2669000 34 38 45 1.22
Samoa 169200
Tanzania 33696000 46 59 1.63
India 1015923008 75 73 -0.40
Senegal 9530000 45 48 44 -0.10
Albania 3411000 97 89 -1.60
Second Quintile 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
Armenia 3827000 82
Honduras 6485000 66 70 1.33
Vietnam 78522704
Malawi 11042000 30 50 3.33
Nepal 23920000 49 51 0.50
Benin 6284000 23 37 38 1.67
Burkina Faso 11274000 19 24 24 0.56
Rwanda 8508000 34 28 -0.67
Bosnia-Herzegovina 3923000
Zambia 10089000 97 80 -2.43
Pakistan 138080000 44
Ghana 19200000 63
Lesotho 2154000 64 69 0.83
Madagascar 15523000 34 30 27 -0.78
Third Quintile 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
Mozambique 17620000 30 21 21 -1.00
Moldova 4264000 66 95 5.80
Bolivia 8328700 71
Indonesia 210420992 92 91 -0.17
Bangladesh 129754000 50
Guyana 863000 92 79 -2.17
Eritrea 4097000 19 36 35 2.00
Georgia 5460000 83
Kenya 30057020 63 58 -1.00
Kyrgz Republic** 4932700
Azerbaijan** 8052000
Yugo. (Serb./Mont.) 10616000 72 70 -0.33
Gambia, The 1286000 40 55 3.75
Nicaragua 5044000 45 58 1.86
Ethiopia 64298000 22 17 23 0.11

Population weighted average annual change = -0.23

 -9-
Annex 2

Source: The primary completion rate is calculated by Bank staff using education data from UIS and World Bank/United Nations
population data.

Notes: The primary education completion rate is calculated as the total number of students successfully completing (or graduating
from) the last year of primary school in a given year, divided by the total number of children of official graduation age in the
population.
Where countries do not report the number of students successfully completing primary school each year, a proxy
primary education completion rate is used. This is calculated as the total number of pupils in the last grade of primary school,
minus the number of students that repeat the grade in a typical year, divided by the total number of children of official
graduation age in the population.

* Average annual change is computed as a weighted average.

** These countries have achieved universal primary school completion.
 - 10 -

Annex 3

Measles Immunization Coverage

(percent)

Country Name 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
Bhutan 93 89 86 84 81 85 86 84 71 76
Uganda 52 54 56 57 59 57 55 54 53 53
Maldives 89 89 89 86 85 85 85 86 88 86
Sri Lanka 80 79 82 86 84 87 89 94 94 95
Cape Verde
Mauritania 38 32 43 49 53 67 66 64 62 62
Samoa 82 77 90 94 96 96 99 100 91 93
Tanzania 80 81 81 77 81 81 78 73 78 72
India 56 43 51 59 67 72 66 55 51 50
Senegal 51 54 57 58 59 80 70 65 62 60
Albania 88 80 87 76 90 91 92 95 89 85
Armenia .. .. 93 95 95 96 89 92 94 91
Honduras 90 86 89 94 94 90 91 99 99 98
Malawi 80 85 91 87 83 90 90 87 90 83
Vietnam 85 88 90 93 96 96 96 96 96 93
Benin 79 60 70 67 77 65 60 66 66 79
Nepal 57 57 58 58 58 56 65 73 73 73
Burkina Faso 79 69 60 50 44 43 40 41 46 53
Rwanda 83 89 82 74 25 84 76 66 78 87
Bosnia and H 52 48 57 53 70 86 84 83
Zambia 90 80 85 91 96 86 87 89 90 90
Pakistan 50 51 52 52 53 53 54 54 54 54
Dominica 91 98 99 99 92 96 99 99 98 99
Ghana 61 63 64 66 68 70 71 73 73 73
Lesotho 80 80 80 81 81 82 82 82 82 77
Madagascar 47 54 54 54 63 55 46 46 46 55
Mozambique 59 50 52 55 56 61 57 57 57 57
Moldova 94 93 92 92 95 99 98 99 99 99
Bolivia 53 54 57 57 64 58 61 51 50 79
Indonesia 58 59 61 62 62 63 71 71 71 71
Bangladesh 65 68 69 74 78 79 69 72 73 71
Eritrea .. .. 18 34 51 58 66 73 81 88
Guyana 77 81 73 80 83 77 91 82 93 87
Georgia 99 81 16 61 63 61 65 69 73 80
Kenya 78 81 84 84 84 83 81 79 79 79
Kyrgyz Republic .. 94 94 94 88 97 98 98 98 97
Azerbaijan .. .. 66 28 91 97 98 97 98 98
Yugoslavia 83 75 82 85 81 86 90 92 88 84
Gambia, The 86 87 83 87 89 91 94 92 92 88
Nicaragua 82 54 73 83 73 81 91 94 99 99
Ethiopia 38 17 12 22 54 38 54 49 46 27