IE4D

IMPACT
EVALUATION
FOR
DEVELOPMENT

Principles for Action

This paper was written by the IE4D Group

June 2010
WHO WE ARE

THE IE4D GROUP

The authors of this paper come from a variety of perspectives. As scholars, practitioners, and commis-
sioners of evaluation in development, research and philanthropy, our thematic interests, disciplines,
geographic locale, and experiences may differ but we share a fundamental belief that evaluative
knowledge has the potential to contribute to positive social change.

We know that the full potential of evaluation is not always (or even often) realized in international de-
velopment and philanthropy. There are many reasons for this – some to do with a lack of capacity, some
methodological, some due to power imbalances, and some the result of prevailing incentive structures.
Evaluation, like development, needs to be an open and dynamic enterprise. Some of the current trends
in evaluation, especially in impact evaluation in international development, limit unnecessarily the
range of approaches to assessing the impact of development initiatives.

We believe that impact evaluation needs to draw from a diverse range of approaches if it is to be useful
in a wide range of development contexts, rigorous, feasible, credible, and ethical.

Developed with support from the Rockefeller Foundation this article is a contribution to ongoing global
and regional discussions about ways of realizing the potential of impact evaluation to improve develop-
ment and strengthening our commitment to work towards it.

Patricia Rogers is Professor of Public Sector Evaluation at the Royal Melbourne Institute
of Technology, Australia. Her work focuses on credible and useful evaluation methods,
approaches and systems for complicated and complex programs and policies.

Sanjeev Khagram is a professor of public affairs and international studies at

the University of Washington as well as the co-founder and executive director of
Innovations for Scaling Impact (iScale).

David Bonbright is founder and Chief Executive of Keystone (U.K. and South Africa),
which helps organizations develop new ways of planning, measuring and reporting
social change. He has also worked for the Aga Khan Foundation, Ford Foundation and
Ashoka.

Sarah Earl is Senior Program Specialist in the Evaluation Unit at the International
Development Research Centre (Canada). Her interest is ensuring that evaluation and
research realize their full potential to contribute to positive social change.

Fred Carden is Director of Evaluation at the International Development Research Centre

(Canada). His particular expertise is in the development and adaptation of evaluation
methodology for the evaluation of development research.

Zenda Ofir is an international evaluation specialist, past President of the African

Evaluation Association (AfrEA), former board member of the American Evaluation
Association and the NONIE Steering Committee, and evaluation advisor to a variety of
international organizations.

Nancy MacPherson is the Managing Director for Evaluation at the Rockefeller

Foundation based in New York. The Foundation’s Evaluation Office aims to strengthen
evaluative practice in philanthropy and development by supporting rigorous,
innovative and context appropriate approaches to evaluation and learning.
 PRINCIPLES
OF IE4D

mpact evaluation can make a difference to development. Accurate, feasible and useful evaluation that is

I aligned with the type of development initiative it is meant to assess can help make development better. It
can guide improvements in policies, programs, and practices, identify and explain successes (so they can
be emulated) and failures (so they can be avoided), shape investment decisions, and encourage funders
and partners to maintain their support. There is, however, another side to impact evaluation. Low quality
or misaligned impact evaluation can waste scarce resources and reinforce inequalities by supporting ineffective
or inappropriate policies or practices or by failing to demonstrate the benefits of effective programs.

The promise of well-defined and properly implemented evaluation to improve development, as well as the
threat of poorly defined and improperly implemented evaluation to derail it, have brought us together. In
this paper we set out an agenda for action to ensure impact evaluation fulfills its promise. We argue for a
decisive move away from impact evaluation of development to impact evaluation for development – impact
evaluation that not only assesses development but consciously and demonstrably contributes to improving
development. This paper sets out a seven-point agenda of rethinking, reshaping and reforming impact
evaluation for improving development. We propose actions that can be taken by all those involved in impact
evaluation of development - practitioners, thought leaders, agenda-setters, philanthropists, and managers and
commissioners of evaluation.

We propose that impact evaluation should:

Contribute to sustainable improvements in Draw on the full range of methods and

1 development. Impact evaluation must ultimately
be judged by its contribution to actually improving
3 designs for systematic and rigorous empirical
investigation, including ethnographic, case study,
development. This means that technical merit is statistical and experimental/quasi-experimental
necessary but not sufficient. Impact evaluation approaches and methods, as well as those from such
must be judged in terms of its timeliness, relevance, fields as complexity science, participatory research,
and usefulness to the different actors involved in and action learning. No one approach is inherently
making development work. Development should more rigorous than another. Rigor depends upon
also be evaluated on whether it is likely to build the both appropriate method choice – selecting methods
capacities of local people, institutions and systems based on evaluative purposes and contexts – and
over the longer term. successful implementation that meets accepted
standards of quality.
Suit the nature of development. Impact evaluation
2 methods and processes should be appropriate
to the context of development and aligned with the 4 Produce a comprehensive analysis of impacts
and outcomes including intended, unintended,
type of development initiative under consideration. positive and negative effects, as well as the
Impact evaluation should not start with a decision distribution of results, costs and benefits.
about a particular approach, but remain open to an
appropriate mix of methods to suit the nature and
context of the specific development initiative being
evaluated.

1
 Explain how and why impacts occur. A good
5 understanding of how results have been achieved RETHINKING
(what works, under what conditions, by whom, how, IMPACT
why and for how long) is essential for learning from EVALUATION
successes and failures. How local actions are affected
by national and international systems, strategies
and policies and vice versa should also be assessed. 1. MAKE A DIFFERENCE
Rigorous examination of causal mechanisms and
he value of impact evaluation must ultimately
contextual factors is required when transferring
initiatives to new settings and wider application.
T be judged by its usefulness in helping to
improve development. In order to do this, impact
evaluation needs to respond to the challenges
Be an integral part of robust systems of
6 monitoring, assessment and learning. Isolated,
discrete impact evaluations are of less value and
produced by the changing nature of development.

Development clearly involves more than donors

use than impact evaluations that are integrated
and national governments. Yet, evaluation practice
into robust systems of monitoring, evaluation and
often tends to consider them the primary audience
learning. The latter can create synergies between
of evaluations in general and impact evaluation
adaptation during a project or program and
specifically. Citizens, civil society, philanthropic
improvements made after periodic evaluation of
organizations, the private sector, regional and
results. Impact evaluation is one element of the
global organizations, partnerships and networks
process of understanding what works, under what
all contribute to development. In particular, there
conditions, by whom, how and why and for how
is increasing recognition of the role of community
long. Integrating impact evaluation within a sound
members as primary agents of development, not as
monitoring and learning system strengthens its
passive recipients of aid. Impact evaluation therefore
contribution to improving development.
needs to find practical, yet meaningful, ways to
Involve fundamental rethinking, reshaping and engage the range of actors in the process and
7 reform of existing practice. Immediate steps
can be taken to practice appropriate evaluation to
address their specific information needs.

Impact evaluation should increase and improve

improve development. The distribution of power, the
accountability not only to funders and decision
pressure for tangible immediate results and impact,
makers, but also to the primary constituents and
and prevailing incentives in development often push
other key stakeholders. Mutual accountability
agencies, foundations and practitioners toward
provides deeper legitimacy and improved
inappropriate approaches to evaluation. A deliberate
effectiveness. Narrow legitimacy is achieved through
reform effort is essential to ensure that impact
accountability towards funders and other partners.
evaluation contributes to improved development.
A more profound legitimacy is achieved when those
who are meant to benefit from development are able
to hold to account those who promote development.

Too many development efforts fail because of

an insufficient focus on creating conditions for
long-term success. Development initiatives are not
only about producing results in the short-term, but
also about developing capacities in individuals,
institutions and systems for ongoing results,
thus enhancing the chance that development will
be sustained. Impact evaluation needs to both
inform short-term decisions about policies and
programs and help to build capacity for improved
planning, implementation and learning to produce
long-term results. Rethinking the role of impact

2
evaluation includes considering how it can support • Contextual factors including social, cultural,
or undermine self-determination, constructive economic, geopolitical factors are understood
negotiation, and improved results in the long term. as critical conditions that shape the nature of
development.
2. WAG THE TAIL • This broader array of initiatives often involves
any of the methods and approaches used today many interacting influences and non-linear,
M in impact evaluation have been developed
for discrete, standardized interventions, like
recursive causality.
• Effective development often requires
drugs or seed varieties. These are development experimentation, repeated practice, learning and
initiatives where the intended results, and how to adaptation during a project or program.
measure them, are clear and agreed, and controlled
comparisons of results with and without drugs The increasingly prevalent emphasis on using
or seeds are both possible and meaningful. For evaluative evidence to inform development
development interventions like those above, these practice and make funding decisions has particular
methods can work implications for the
well when they have practice of impact
adequate resources and evaluation. If the only
Starting from current practice, two principles
are well implemented. evidence that is seen as
about impact evaluation rise above others:
it must contribute to improved development, credible is evidence from
But many development discrete, standardized
initiatives are not like and it must suit the nature of development.
interventions, there is a
this. They are broadly risk that multi-faceted or
defined approaches systems-based initiatives
or strategies, often will not be supported – even if they are effective. This
implemented in different ways in different situations, will have dire consequences for development.
and highly influenced by a wide range of contextual
factors. Simple with/without comparisons are So what are we to do? Avoiding these realities is
often not the most meaningful way of assessing not an option. Failing to undertake rigorous impact
impacts because the initiatives (and their underlying evaluation of initiatives with complicated and
change logics) have complicated aspects (multiple complex aspects, and relying on their face validity,
components that vary in different settings) or or good intentions, or selective descriptions of
complex aspects (dynamic and emergent) or both. direct beneficiaries, is not sufficient. But neither
is it appropriate to mold these interventions into
Development has changed significantly in the 21st discrete, standardized programs to make them easier
century due, in part, to processes of globalization. to evaluate – that would be like the tail wagging
In particular, development initiatives are more often the dog. Impact evaluation for development has to
complicated and complex in the following ways: be able to evaluate the impact of complicated and
• Diverse initiatives are being implemented more complex initiatives in a rigorous manner, and provide
often than discrete, standardized projects. useful advice to support translating those findings
• Multiple development goals associated with to other situations. And it can do so, by following
a wide range of issues are being promoted the principles for reshaping and reforming impact
simultaneously. evaluation below.

• Development effectiveness involves a wider range

of dimensions.
• A broader range of legitimate stakeholders
(beyond aid agencies and national governments)
are often actively engaged, and diverse sources
and types of capital are used to support develop-
ment initiatives. Primary constituents are often
recognized as central agents of development.

3
 A range of designs can be used to discover and
RESHAPING describe different impacts. It is possible for experi-
IMPACT mental and quasi-experimental methods of impact
EVALUATION evaluation to pay attention to differing impacts if
relevant variables can be identified and measured
in advance. However, it is frequently difficult, if not
3. BE MORE SCIENTIFIC impossible, to anticipate (and account for) many
e applaud the increased emphasis on more relevant factors. Impact evaluation approaches that
W scientific impact evaluation. However, some efforts
to improve the technical merit of impact evaluation have
draw on a wider range of credible evidence, especially
multiple kinds of comparison and triangulation, make
ended up with an unnecessarily narrowly defined set of it possible to include variables that may not have
methods and inappropriate hierarchy of which methods been identified or anticipated
are best, driven by a conceptualization of science that is
inaccurate and outdated. There is an ethical imperative to identify and analyze
unintended consequences, especially negative
Scientific impact evaluation draws on insights ones. Impact evaluation should deliver a balanced
and techniques from natural, social, systems and assessment that includes intended and unintended,
complexity sciences. It emphasizes multiple kinds of positive and negative impacts. It should assess the
valid comparisons, including the use of counterfactuals distribution of benefits and costs of initiatives, address
when appropriate, triangulation across methods and effectiveness, legitimacy, efficiency and sustainability.
types of quantitative and qualitative data to improve
measures and analysis. Sophisticated diagnostic tools 5. EXPLAIN HOW AND WHY AND RECOGNIZE
for appropriate (and often mixed) methods selection DIFFERENCES
and measurement are increasingly available and mprovement in impact evaluation means building
should be utilized. Appropriate norms, guidelines
and criteria should also be used to provide assurance
I
knowledge about what works, under what
conditions, by whom, how and why. It means
that any method or set of methods meet technical and assembling knowledge about when and in what ways
ethical standards. initiatives should be transferred to other settings
and how to generate and
Scientific rigor should use evidence to inform
be judged in terms To enact these cardinal principles, the practice policy and practice.
of appropriate data, of impact evaluation should be more scientific, When impact evaluation
method selection and avoid a narrow focus, recognize differences and does not recognize local
implementation. This close the loop. knowledge, cultures or
kind of rigor identifies situational differences
and addresses threats in its results, then it
to internal and external leads to ‘one-size-fits-all’ policies to the detriment of
validity that are present in any measure, method or improving development.
research design. Validity also has an inherently cultural
dimension – and all data, methods and analysis must Data about the average intended effects of a project
address the challenge and opportunity of multiple are critical but often not sufficient to inform good
cultural understandings of reality. policy or practice. Information is needed about
the results for different groups (particularly for
4. AVOID A NARROW FOCUS the most disadvantaged) to inform judgments
evelopment has many dimensions and many about the value, transferability and scalability of a
D development initiatives in the 21st century have
multiple goals. While it is not possible for every impact
development initiative. Information is also needed
about the contexts in which the project is successful
evaluation to cover all aspects of a development to inform conclusions about whether the findings
initiative, impact evaluation should produce an can be generalized. Few development initiatives are
adequately comprehensive assessment of results. expected to work the same way in all cultures and
settings at all times. The task is to help understand

4
how development may be affected by local and global
contexts. Understanding how and why impacts differ is REFORMING
at the heart of good impact evaluation. IMPACT
EVALUATION
For some initiatives it will be possible to identify in
advance how planned activities and outputs could
produce clearly defined results. For many develop- 7. CHANGE THE SYSTEM
ment initiatives, however, no single actor, factor or
hile we recognize the importance of power
causal mechanism working independently will be
sufficient to produce results. Rather, impact could be
W and politics in improving impact evaluation for
effective development, we also argue that everyone
achieved through the combined efforts of multiple
involved in evaluation of development initiatives can
actors, with overlapping and interacting causal mech-
take action and contribute to positive change. We
anisms, acting in particular contexts. Solutions to
have identified below an initial list of steps to reform
development problems will involve changing beliefs,
the system.
attitudes, relationships, capabilities, conditions
and behaviors and often need to be worked out over
time with constituents as well as with partners. For What should evaluation practitioners do?
these types of initiatives, an iterative and adaptive Evaluation practitioners are critical actors in this
approach is essential and impact evaluation will need system. They are the implementers of impact
to document and support this. evaluation policies and approaches on the ground.
Practitioners, particularly those who work in their
6. CLOSE THE LOOP home countries, have power, knowledge and
access to the projects, programs and policies where
mpact evaluation is more effective when it is part
I of a robust learning and improvement system
that integrates cycles of planning, implementation,
development action is implemented. While they do
not normally have a mandate to make institutional
change, independently there are many steps
monitoring, assessing and reporting. Such learning
evaluators can take to move us all towards a vibrant
and improvement-oriented systems translate credible
field of IE4D.
evidence of an effective development process or
initiative into actionable guidance, especially for Challenge yourself; challenge your clients.
policy makers and citizens. They do so in ways n Be a model of good practice; maintain
that are appropriate for effective and immediate
high quality using appropriate evaluation
utilization. Embedded impact evaluation creates
standards; ensure that you follow impact
synergies between immediate improvements and
evaluation practices across disciplinary
longer-term assessment of results.
boundaries.
n Recognize that your clients may not be
The effectiveness of development initiatives will
improve when those who fund and manage them are clear about what they need, what degree of
held accountable by those most affected - positively certainty evaluation can provide and what can
and negatively. This implies that public reporting of realistically be achieved within their time and
impacts should reflect the ways in which a develop- resource constraints. Help them to identify
ment initiative responds to and balances the needs their needs and make decisions that will make
and perspectives of its various constituents. Impact impact evaluation useful.
evaluation design needs to explicitly consider: (a) how n Promote impact evaluation from a systems
the initiative translates its understanding of what is perspective. Understand your assignment in
happening into action; (b) how it adapts and improves; the wider development context not solely in the
and (c) how different constituents of an initiative under- narrow parameters of the existing initiative.
stand and act on evidence about the results reported.
Promote appropriately rigorous methods.
Impact evaluation is only one type of evaluation n Be a learner as well as an innovator: do not
that can contribute to improved development, and become complacent in your practice. Be aware
resources for it should not detract from other types of changes in development practice, and the
of evaluation. broader external environment. Follow and
5
 participate in debates, seek out and contribute transparent about their values and practices and
to new tools and methods and contribute brave enough to invite constructive criticism and to
to standards for good impact evaluation listen to evaluators who will ‘speak truth to power’.
practices. Evaluation commissioners and managers must be
n Recognize that your values and practices are committed to learning from success, but must also
shaped by your history, as are those of others. be prepared to face the failures of development and
Doing impact evaluations in other cultural shatter ill-founded dogmas, question self-serving
contexts is therefore extremely sensitive and assumptions and challenge complacency among
requires active dialogue with, and participation stakeholders. While this may be a tall order in many
of, key stakeholders. political environments and organizations where
‘safe spaces’ for evaluation and learning are limited,
Build vibrant evaluation networks. there are practical steps that all commissioners and
n
managers can take to create a supportive environment
Seek out and work with others through
and a set of incentives to work towards IE4D:
associations and communities of practice.
These provide not only professional support Be clear how impact evaluation is intended to
but are also key mechanisms for reforming contribute to improving development
the policies and agencies that shape impact
n Conceptualize and frame evaluation around
evaluation practice. Do not settle for passive
clearly defined theories and assumptions
engagement
of what contributes to
in your field
effective development
and passive
acceptance of
Fundamental reform requires each of us to ask, and improvements
“How can we do better?” and “What do we in the lives of people
popular trends in
need to do differently?” to implement these and the systems upon
practice without
changes. which they depend.
carefully thinking
(Use explicit definitions
through the
of development and
implications.
hypotheses about the
role that rights, choices, freedom, economic
Contribute to improved reporting and knowledge
growth, sustainability, accountability, etc. play
sharing.
in development processes).
n Tailor reporting and communication of impact
n Focus on outcomes and impacts - not only
evaluation results, approaches, methods, and
outputs - to the extent feasible and appropriate.
lessons to reach a broader range of audiences
across disciplines, practice, policy and
Reflect standards and guidelines
research.
n Set, promote, use and support the

improvement of standards and guidelines

What should those who commission and for development evaluation that reflect
manage evaluation do? international good practice and regionally
Those who commission and manage impact evalu- adapted standards (such as the African
ations are in an extraordinary position of privilege. Evaluation Guidelines).
Whether they recognize it or not, they have the power
and authority to change the way impact evaluation for Extend Boundaries
development is conceived, commissioned, managed, n Commission evaluations that push the

reported and disseminated. If they fail to recognize, boundaries of existing approaches to impact
or act on, the prevailing inequities and biases of evaluation and that evaluate beyond the
much current development evaluation practice they interests of those in charge of specific
perpetuate inappropriate practices. Those who com- development initiatives.
mission and manage evaluations must be strong,
independent thinkers, well grounded in development
and evaluation theory and practice. They should be

6
Broaden the focus What should thought leaders, agenda
n Join forces with other organizations and setters, educators, and mentors do?
agencies to broaden the focus of what is Those who create impact evaluation knowledge and
evaluated in order to capture what matters in teach, mentor or influence those who commission
development. and practice impact evaluation can also contribute to
n Evaluate the the drivers and root causes positive change. As the field of evaluation - including
of unsustainable development and the impact evaluation - becomes more reflective, it calls
effects of developed country policies on for champions who can communicate its value, who
developing countries in areas such as foreign can be influencers among decision-makers and
investment, trade, globalization, migration, who can motivate and engage the broader public.
pollution, and intellectual property. Include They need to translate theory into practice and
evaluating capacity development at individual, improve theory based on good practice, as well as
institutional and systems levels as a critical disseminate new ideas, develop new concepts and
step towards achieving development outcomes challenge conventional wisdom.
and impact.
s development is increasingly recognized as
Embed evaluation within the management and
leadership of organizations.
A trans-disciplinary, so too should training and
capacity building in impact evaluation bring together
n Make sure evaluations consider the way an the fields of social, natural and emerging sciences,
organization or initiative plans, monitors, management and cultural disciplines to provide
assesses, and reports back to its constituents much needed insights into complex questions of
and the wider public. behavioral, institutional and structural change.
Thought leaders, agenda setters, trainers and
Untie funds for impact evaluation for development mentors can support IE4D through the following
n Manage the procurement process to prioritize strategies:
the use and development of high quality Broaden the collaboration
regional, national and local evaluation skills.
n Provide ‘safe spaces’ for evaluative learning,
n Set up pooled national funds at arm’s length analysis and reflection within academic and
from power-holders to enable less empowered research environments, and in civil society and
and marginalized groups and organizations government.
to choose their own evaluators to assess
n Bring together citizens, civil society, business
development initiatives.
and public sector to examine new ways to
Invest in evaluation infrastructure collaborate on impact evaluation.
n Invest in capacity and field building in impact Plan for different types of accountability and learning
evaluation at multiple levels – developing
n Incorporate accountability frameworks,
individual skills, organizational infrastructure,
transparency requirements, inclusive
networks of practitioners and professional
methodologies, and citizen engagement in new
evaluation associations.
ways of practicing development evaluation.
Improve reporting n Promote improved reporting and knowledge
n Commission and support improved reporting sharing of evaluation results, methods, tools,
and communication of evaluation results, advice, and examples to reach a broader range
methods, and lessons, in order to reach a of audiences across disciplines, practice,
broader range of audiences across disciplines, policy, and research.
practice, policy, and research. This could n Ensure training and professional development
include authoritative, user-friendly briefs on incorporates multiple approaches and methods
progressive approaches to, and examples of, in impact evaluation, drawing from different
impact evaluation to extend and balance the disciplines, perspectives and worldviews.
existing body of information.

7
FINAL
THOUGHTS

e are privileged to work in an expanding field

W where our findings and processes can change
lives for the better. Evaluation approaches are
currently in the spotlight among politicians, policy
makers and philanthropists who seek to maximize
the impact of their resources. It is therefore timely
to work towards the fundamental changes both
necessary and possible to make impact evaluation
more relevant, credible and useful for development.

We extend an invitation to those who want to make a

difference, to those who want to bring about change.
There is much to do – let’s get to work.

June 2010
Edited by Leslie Cole
Layout by Cynthia Spence