MONITORING

EVALUATION AND
LEARNING
Concepts, principles and tools
Publications in this series

CDAIS manuals and guidelines

• Capacity Needs Assessments – A trainers’ manual (2nd edition)
• Innovation Niche Partnerships – A guide to the coaching process
• Organisational Strengthening – A guide to the coaching process
• Organising a Marketplace – A practical guide
• Organising a Policy Dialogue – A practical guide
• Monitoring, Evaluation and Learning – Concepts, principles and tools

CDAIS stories and conversations

• Building Competence and Confidence in Agricultural Innovation – Stories of Change
• Catalysing Innovation in Agriculture – Conversations of Change

The manuals are intended as working documents. The project supported the development of the Common
Framework on Capacity Development for Agricultural Innovation Systems of the Tropical Agriculture Platform,
and tested it in eight pilot countries. One key finding was that the framework requires adaptation in each
country situation, and as such the manuals are intended as general guides only.

https://cdais.net
www.tapipedia.org
 MONITORING
EVALUATION AND
LEARNING
Concepts, principles and tools

Published by
Agrinatura, and the Food and Agriculture Organization of the United Nations

Paris, 2019
Disclaimer

The CDAIS project is funded by the European Union. This document has been produced with the financial assistance
of the European Union. The views expressed herein can in no way be taken to reflect the official opinion of the
European Union.

The designations employed and the presentation of material in this information product do not imply the expression of
any opinion whatsoever on the part of Agrinatura or the Food and Agriculture Organization of the United Nations (FAO)
concerning the legal or development status of any country, territory, city or area or of its authorities, or concerning the
delimitation of its frontiers or boundaries. The mention of specific companies or products of manufacturers, whether
or not these have been patented, does not imply that these have been endorsed or recommended by Agrinatura
or FAO in preference to others of a similar nature that are not mentioned.

The views expressed in this information product are those of the author(s) and do not necessarily reflect
the views or policies of Agrinatura or FAO.

Some rights reserved. This work is made available under the Creative Commons Attribution-
NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO)

www.creativecommons.org/licenses/by-nc-sa/3.0/igo/legalcode.

Under the terms of this licence, this work may be copied, redistributed and adapted for non-commercial purposes,
provided that the work is appropriately cited. In any use of this work, there should be no suggestion that Agrinatura
or FAO endorses any specific organisation, products or services. The use of Agrinatura and FAO logos is not permitted.
If the work is adapted, then it must be licensed under the same or equivalent Creative Commons license. If a translation
of this work is created, it must include the following disclaimer along with the required citation: This translation was not
created by Agrinatura and FAO. Agrinatura and FAO are not responsible for the content or accuracy of this translation.
The original language edition shall be the authoritative edition.

Any mediation relating to disputes arising under the license shall be conducted in accordance with the Arbitration
Rules of the United Nations Commission on International Trade Law (UNCITRAL) as at present in force.

This publication may be freely quoted and reproduced provided the source is acknowledged.
No use of this publication may be made for resale or other commercial purposes.
All photographs are CDAIS unless credited otherwise.

Citation

Agrinatura and FAO, 2019. Monitoring, Evaluation and Learning – Concepts, principles and tools.
Agrinatura, Paris and FAO, Rome. 24 pp.

ISBN: 978-2-35709-006-4 (Agrinatura)

ISBN: 978-92-5-131494-4 (FAO)

© Agrinatura and FAO, 2019

Agrinatura FAO
The European Alliance on Agricultural Food and Agriculture Organization
Knowledge for Development of the United Nations
European Economic Interest Grouping Via delle Terme di Caracalla
42 rue Scheffer 00153 Rome, Italy
75116 Paris, France [email protected]
[email protected] www.fao.org
www.agrinatura-eu.eu

2 CDAIS Monitoring, Evaluation and Learning

Contents
Introduction  4

1. Background  4
1.1. CDAIS and the ‘common framework’ on capacity development  4
1.2. Why monitoring, evaluation and learning in CDAIS?  5

2. Objectives and principles of the MEL system  5

2.1. Support project implementation  5
2.2. Provide evidence of changes  6
2.3. Key principles for the design of the MEL system  7

3. The MEL framework  8

3.1. MEL at the two levels and three dimensions of interventions  8
3.2. MEL at agricultural innovation system level: identifying the impact pathway  9
3.2.1. Key evaluation questions  9
3.2.2. Realist evaluation  9
3.2.3. CDAIS theory of change and ex-ante impact pathway  9
3.2.4. Actors of change and threshold of irreversibility  12
3.3. MEL at innovation niche partnership level: Supporting and assessing capacity-development processes  14
3.3.1. Key evaluation questions  14
3.3.2. Participatory outcome mapping  14
3.3.3. Progress markers  14
3.3.4. Embedding MEL and experiential learning cycles  16

4. MEL times and tools  17

4.1. MEL phases and times  17
4.2. MEL tools at agricultural innovation system level  18
4.3. MEL tools at innovation niche partnership level  19

5. MEL in practice  20
5.1. Coordination of the MEL system  20
5.2. Perceptions of value and benefit  20
5.3. Challenges in gathering, analysing and using information  20
5.3.1. Moving towards more consistent and better-quality data  20
5.3.2. Ensuring the right people participate  21

References  21

Acknowledgements  22

CDAIS Monitoring, Evaluation and Learning 3

Introduction
T
his document presents the framework into which The CDAIS project was jointly developed and implemented
the monitoring, evaluation and learning (MEL) by Agrinatura-EEIG (European Economic Interest Group) and
component of the Capacity Development for the Food and Agriculture Organization of the United Nations
Agricultural Innovation Systems (CDAIS) project (FAO). It was conceived to support the implementation of
was conceived, the objectives it pursued, and how the Tropical Agriculture Platform (TAP) action plan, a G20
it was implemented. Initiative on improving the global coherence of capacity
development for agricultural innovation, based on a
It highlights the concepts, principles and tools that have preliminary diagnosis that many countries are not fully
been used by MEL teams in each of the eight pilot countries. exploiting their innovation potential (TAP, 2016). As TAP
The worksheets that have been developed in parallel to partners, and in line with their visions, Agrinatura and FAO
guide the CDAIS country teams on how to apply and use collaborate towards a coherent approach to strengthening
MEL are available at https://cdais.net/publications. agricultural innovation systems, guided by the TAP Common
Framework on Capacity Development for Agricultural
Innovation Systems (TAP, 2016), referred to herein as the
‘Common Framework’. The Common Framework was tested
1. Background in the eight pilot countries between 2015 and 2019.

The main assumption of the common framework is that the

1.1. CDAIS and the ‘Common Framework’ functional capacities of individuals and organisations must
on capacity development be strengthened to enable them to innovate and, at the same
time, reinforce the agricultural innovation system that in turn
The overall objective of the CDAIS project was to promote creates an enabling environment.
agricultural innovation systems that are efficient and
sustainable in meeting the demands of farmers, agri- The common framework is grounded in three theoretical
businesses and consumers while facing environmental perspectives that were taken into account during
and socioeconomic challenges. Its specific objective was implementation in all countries.
to establish a global partnership on capacity development
in agricultural innovation systems on a sustainable footing, • A systemic perspective. Innovation systems are
with needs assessed and approaches validated in eight pilot more effective if what affects the relationships between
countries – Angola, Bangladesh, Burkina Faso, Ethiopia, components of the system (i.e. functional capacities)
Guatemala, Honduras, Lao PDR and Rwanda. is strengthened. The functional capacities identified in
the common framework, also referred to as the ‘4+1
capacities’, are the capacity to (i) navigate complexity,
(ii) collaborate, (iii) reflect and learn, and (iv) engage in
Agricultural innovation system strategic and political processes, which together result
“a network of actors or organisations, and individuals, in (v) the capacity to adapt and respond to realise the
together with supporting institutions and policies in the potential of innovation.
agricultural and related sectors that brings existing or • A strategic innovation niche partnership
new products, processes, and forms of organisation management perspective. This involves a dual-pathway
into social and economic use.” approach of capacity development for agricultural
innovation systems at the innovation niche partnership
Capacity level and at the agricultural innovation system level.
“the ability of people, organisations and society as • A learning perspective. Agricultural innovation system
a whole to manage their affairs successfully.” stakeholders – both individuals and organisations – have
to go through a five-stage supervised learning process
Capacity development to develop their capacities to innovate: galvanising
“the process whereby people, organisations and society commitment, visioning, capacity needs assessment,
as a whole unleash, strengthen, create, adapt and developing a capacity-development strategy, and
maintain capacity over time”. implementation.
Source: TAP (2016).

4 CDAIS Monitoring, Evaluation and Learning

 BACKGROUND

An innovation niche partnership is where several To test the validity of the common framework, a comparative
actors interact to solve specific problems and then cross-country analysis was needed to check the theoretical
develop their collective capacities to innovate. It is a framework against empirical data. Those data were gathered
locus of learning and experimentation and micro-level and collected through the MEL system to build these
transformation. comparisons.

At a minimum, it is a network of individuals and

In short, the MEL system was seeking the ‘difference’ that
organisations that are contributing to the same
the CDAIS project was making.
innovation goals. They might not all have the same
interests and level of engagement, but they do have
common objectives and interactions. At best, it is a
community of individuals and organisations that share
a common language, vision and strategy, that know
2. Objectives and principles
how to work together, and that are mutually engaged of the MEL system
to achieve innovation.

The MEL system in CDAIS had two main objectives: (i) to

support project implementation, and (ii) to provide evidence
1.2. Why monitoring, evaluation of the changes that the project made. To achieve these
and learning in CDAIS? objectives, it was designed following nine key principles.

Based on a diagnosis of country needs to develop or

reinforce national agricultural innovation systems, the 2.1. Support project implementation
CDAIS project sought to address a wide range of issues.
Those issues were broadly related to the lack of efficiency, The first main objective of the MEL system was to provide a
responsiveness and efficacy of agricultural innovation continuous assessment and learning process to help country
systems, as evidenced by insufficient demand-driven teams adapt capacity-development interventions in response
innovations, a lack of coordination of capacity-development to specific needs, and thus move towards greater impacts.
interventions, inadequate capacity-development interventions
and too many small-scale interventions with narrow scope The CDAIS project proposes a new way to ‘do development’,
mostly focused on individual technical capacities. which is to train and support agricultural innovation system
actors so that they become able to autonomously organise
In the common framework, assumptions about change themselves and work in a more efficient manner to achieve
mechanisms were based on a large corpus of literature, innovations faster. Doing development this way requires
combining concepts from agricultural innovation systems project teams to contribute actively and explicitly to project
and systems thinking, capacity development and innovation learning, for instance by providing feedback and reflection in
management literature. In effect, the CDAIS project meetings or after-action reviews. This aims at informing and
implemented several pilots at the same time. To capture the feeding into project decision making processes at all levels
diversity of changes, mechanisms of changes and impacts (national and global). However, project teams in countries
achieved by the CDAIS project in the different countries, the do not necessarily have the necessary skills to do this – or
global coordination team decided to develop a monitoring, they do not see it as a part of their day-to-day work, and
evaluation and learning (MEL) system at the initial stage may consequently not make time for it. The MEL system is
of the project. therefore particularly important to help structure this type of
process and to provide the quick feedback loops needed
The MEL system is derived from the principle that continuous to inform adaptive programming such as that underlying
evaluation and learning enables adaptive management of the CDAIS project. In consequence, the CDAIS MEL
transformational projects and, hence, helps improve their system was designed and developed as an internal project
performance. Monitoring is used as an evidence-based process that encouraged a culture of learning at all levels,
approach to detect and support changes in knowledge, i.e. project teams and boundary partners. In turn, it helped
attitudes and practices within partners of the project to project teams and boundary partners to collectively identify
achieve systemic changes and greater impacts. capacity-development interventions that might be the most
impactful.

CDAIS Monitoring, Evaluation and Learning 5

 Objectives and principles of the MEL system

For this, the MEL system provides tools and methodologies learning by using participatory monitoring and evaluation
to: (i) assess changes in functional capacities and their approaches.
effects on agricultural innovation systems, (ii) support
the adaptation and refinement of capacity strengthening Table 1 lists the benefits of MEL to project stakeholders,
to achieve greater impacts, and (iii) stimulate continuous implementers and partners.

Table 1. Benefits of the Monitoring, Evaluation and Learning (MEL) system to project stakeholders

Project stakeholders Benefits of MEL

CDAIS direct partners MEL helps partners answer the following questions (among others):
and boundary partners • Is what they are doing contributing to the long-term change the organisation/ innovation

(individuals, organisations) niche partnership is trying to create?

• Are their project activities on track?
• Are their partners’ activities on track?
• Are their activities pursuing the achievement of demand-driven innovation?
• Are they experiencing anticipated changes as a result of the project?
• Are there any challenges to address? If so, what are they?
• What is working well, what is not, and why? What could be done differently?

CDAIS implementing MEL helps implementing organisations answer the following questions (among others):
organisations • Are their activities on track?
• Are their activities reaching target communities?
• Are communities experiencing anticipated changes as a result of the project?
• Are there any challenges to address? if so, what are they?
• What is working well, what is not, and why? What could be done differently?

CDAIS project partners and MEL helps provide insights into (i) lessons learnt during the project and conditions
funders for replicability, and (ii) other possible interventions to develop capacities to innovate,
not covered by the project

2.2. Provide evidence of changes The MEL system seeks to provide documented evidence –
and better understanding – of the leveraging role of the five
The second objective was to provide analytical tools and pre-identified functional capacities but also of other emerging
methods to measure and compare the performance and capacities as they are developed in different cases and
relevance of the common framework when applied in eight contexts. The documented evidence provides partners and
different country contexts. funders solid feedback on outcomes and lessons learnt.

The central assumption of the CDAIS project was that For this, the MEL system provides tools and methodologies
functional capacity development is key for a more efficient to ensure: (i) a standard and minimal data collection
and responsive agricultural innovation system that will enable process, and (ii) standardised analyses and reporting on the
more smallholder-centred innovations, making agriculture contribution of the CDAIS project to capacity development
more productive and sustainable. Other capacities might and the advancement of innovation.
be needed, depending on the individuals and organisations
that are involved, their activities and purposes. The types of
needed capacities might also depend on the nature of the
innovation process (incremental or radical) and the type of
innovations sought (technological, services or organisational).

6 CDAIS Monitoring, Evaluation and Learning

 Objectives and principles of the MEL system

Table 2. CDAIS project Monitoring, Evaluation and Learning (MEL) system components and purposes

MEL component Purposes

Tracking and monitoring • ‘Tracking’ refers to a periodic process of identifying changes in stakeholder capacities
to innovate, using progress markers
• It supports the monitoring process: ‘monitoring’ refers to the monitoring of stakeholder
objectives, activities and capacities to achieve innovation, and analysis of the information
to guide the design of capacity-development interventions, in light of project resource
constraints

Internal evaluation • ‘Internal evaluation’ refers to the assessment and analysis of capacity needs,
contributions of the CDAIS project to capacity development and possible impacts
• It is conducted at the start and the end of project implementation by the project
implementers
• Capacity assessments made at the beginning of the project are used as a baseline
to be compared with the final capacity assessments at the end of the project

Learning • ‘Learning’ is the process through which information generated from tracking, monitoring
and evaluation is reflected upon and intentionally used to continuously improve the
capacities of project partners and the ability of the project to achieve impacts

2.3. Key principles for the design of the MEL system 3. About the data
• Focus the MEL approach on the capture of observable
The MEL system was developed, implemented and adapted changes during the project, from inputs to outcomes.
throughout the CDAIS project (2015–2019). It was designed Some tools could be proposed to national partners for
to track, monitor and evaluate outcomes in a participatory future impact assessment.
manner that enabled, on the one hand, learning and capacity • Combine methods to generate both quantitative and

development of project partners and, on the other hand, qualitative data – this leads to more comprehensive
learning and adaptation of the implementation strategy for understanding and comparability among innovation niche
project implementers. partnerships and countries.
• Ensure comparability of key data across countries – the

The key principles for the design of the MEL system were use of MEL tools, approach and results will help to build
the following. comparison between innovation niche partnerships and
between countries.
1. About the participants • Ensure credibility and trustworthiness of data for all parties
• Adopt a participatory approach to identify and evaluate (innovation niche partnership and national stakeholders,
outcomes and thus enhance ownership of the process. donors, TAP partners).
• Work with all key actors of the innovation niche

partnerships and agricultural innovation system to 4. About time and cost

establish dialogue in developing and using the MEL. • By adopting this iterative and continual coaching
• Acknowledge that different stakeholders may have approach, ensure MEL is relatively low-cost and not too
different understandings of ‘capacity development’ time consuming for both those collecting data and those
and of the purpose of the MEL. responding.

2. About the process

• Use iterative, continual, reflective feedback approaches to
determine what is happening in the capacity-development
process and why it is happening.

CDAIS Monitoring, Evaluation and Learning 7

 The mel framework

3. The MEL framework

3.1. MEL at the two levels and three dimensions • ‘Systemic dimension’ refers to the capacities of a
of interventions network of actors that are engaged in joint actions for
reinforcing the agricultural innovation system by bridging
The MEL system tracks changes at two levels (innovation its four elements: research and education; bridging
niche partnership level and national agricultural innovation institutions such as agriculture extension and innovation
system level) in three dimensions: individual, organisational support service providers; business and enterprise; and
and systemic. the enabling environment (policies and institutions).

• ‘Individual dimension’ refers to capacity development Processes and interventions to be monitored and evaluated
of individuals such as national innovation facilitators at each level are shown in Table 3.
(NIFs), innovation niche partnership actors, policy makers
or organisations’ employees who are involved in an
innovation process in one way or another.

• ‘Organisational dimension’ refers to capacity

development of organisations involved in the activities of
innovation niche partnerships, either as innovation support
service providers or as innovation project promoters.

Table 3. Processes and events assessed within the MEL system

Level Processes to be monitored and Related capacity-development

assessed interventions to be assessed

Agricultural • Improvement of the institutional and • National Innovation Facilitator training

innovation policy context for innovation niche and mentoring
system partnerships, through: • Marketplace event
- the capacity development of National • Policy dialogue
Innovation Facilitators; • Setting up of a national innovation
- the emergence and/or strengthening platform
of innovation support service providers; • Capacity-development activities for
Country - the development or improvement innovation support service providers
of policies promoting agricultural
innovation

Innovation niche • Innovation niche partnership • Capacity-development activities for

partnership stakeholders’ capacity development innovation niche partnerships and their
• Contribution of functional capacities to relevant member organisations
advance innovation

8 CDAIS Monitoring, Evaluation and Learning

 The mel framework

3.2. MEL at agricultural innovation system level: Moreover, different stakeholders will have different
identifying the impact pathway information and understanding about how the project is
supposed to work and will therefore implement it differently.
3.2.1. Key evaluation questions
• Taking into account the way it has been designed and Data collection processes (interviews, focus groups,
implemented, how did the CDAIS project contribute to questionnaires, etc.) should partly be constructed to identify
improvements in the effectiveness of the agricultural the particular information that specific stakeholder groups will
innovation system in a given country? have, and thereby allow theories about how and for whom
• Was the CDAIS approach relevant to the intended users? the project ‘works’ to be refuted or refined.
In other words, did it suit the priorities and policies of the
target groups, recipients and development partners? Did it 3.2.3. CDAIS theory of change and ex-ante
in fact engage target populations and promote continuous impact pathway
learning? What factors influence the sustainability and The theory of change of the CDAIS project was initially
replicability of CDAIS approach at global level? developed at the beginning of the project and refined
during the mid-term evaluation. The main assumptions are
3.2.2. Realist evaluation explained in the common framework, i.e. the development
By applying the common framework in eight different of 4+1 functional capacities in three dimensions (individual,
contexts, CDAIS was testing a theory about what might organisational and systemic) using a dual-pathway approach
cause changes within an agricultural innovation system. (at innovation niche partnership and national system levels)
It was therefore assumed that in different contexts changes should help to reinforce agricultural innovation systems and
will be reached through different mechanisms, such that make them more effective in a diversity of contexts.
the project cannot simply be replicated from one context
to another and automatically achieve the same outcomes. An ex-ante impact pathway scheme maps the inputs,
Theory-based understanding about ‘what works for whom, outputs, and expected outcomes and impacts that might
in what contexts, and how’ is, however, transferable. be produced within a project (see Figure 1).

A realist evaluation approach has particular implications for While inputs and outputs are usually fairly easy to identify, it
the design of an evaluation and the roles of participants. For is harder to determine the nature of outcomes and impacts,
example, rather than comparing changes for participants their actual place and role in the impact pathway, and the
who have undertaken a project with a group of people contributions of capacity-development interventions in
who have not (as is performed in randomised controlled or their production. Outcomes are strongly linked to capacity
quasi-experimental designs), a realist evaluation compares strengthening at individual, organisational and systemic
‘context–mechanism–outcomes’ configurations within the levels. These are the necessary changes in capacities
project. It may ask, for example, whether a project works (that enable stakeholders to join and amplify the innovation
more or less well, and/or through different mechanisms, in process) that will eventually lead to actual impacts. In some
different localities (and if so, how and why); or for different cases, these changes can happen fast and be observable
population groups (e.g. men and women, or groups with during the lifetime of the project. In other cases, it could take
differing socioeconomic status). longer, according to the opportunities that stakeholders will
have to implement new activities. Assessing impacts was
considered beyond the reach of the CDAIS project.

CDAIS Monitoring, Evaluation and Learning 9

 The mel framework

Figure 1. Ex-ante impact pathway of the CDAIS project

10 CDAIS Monitoring, Evaluation and Learning

The mel framework

Inputs: -- Partnerships, negotiation or advocacy issues

Inputs are the resources used by the implementing to address technical, organisational or financial
organisations: human resources (Agrinatura Focal challenges with external stakeholders (policy makers,
Persons [AFPs], FAO Country Project Managers [CPMs], donors, industries, input suppliers, banks, etc.)
National Innovation Facilitators [NIFs], National Project • At the organisation level, outcomes correspond
Coordinators [NPCs] from relevant national institutions, field to new initiatives to better support partnership
staff, capacity-development experts or subcontractors), stakeholders: new strategies, new vision, new support
knowledge, skills, material and financial resources. services or funding opportunities
• At the individual level, outcomes correspond to
Outputs: changes in the behaviour, knowledge or activities of
Outputs are the immediate results of project inputs – partnership stakeholders. For example, members of the
the processes, goods and services that CDAIS staff national innovation platform or policy makers, following
produced. Examples include: training of National national workshops or events (marketplace) and policy
Innovation Facilitators, inception workshops, needs round-tables.
assessments, policy dialogues, capacity-development
activities, classroom-based training, training manuals, and Expanded outcomes:
research and assessment reports or strategy documents Expanded outcomes are changes in the functioning of
(capacity-development plans). These immediate results the agricultural innovation system – i.e. becoming more
are directly controlled by the implementing organisation. effective, efficient, responsive and/or sustainable. They
They include all the facilitation activities, capacity building arise from individual and organisational capacities to
and mentoring with innovation niche partnership and innovate and correspond to the overarching capacity to
agricultural innovation system stakeholders. They do not innovate at the national level.
include what the individual group does (or does not do) They can be revealed by the scaling up of innovation
with the new knowledge, skills or attitudes emerging from support processes, allowing innovation niche partnerships
these activities. to enter the mainstreams of agricultural development. They
also imply that changes become observable at the level
Primary outcomes: of the innovation processes themselves: the innovation
Primary outcomes are observable changes in the processes are shaped, accelerated and/or modified in
behaviour, attitude, practice and/or mindset of direct an intentionally manner.
project ‘beneficiaries’. Expanded outcomes make the probability of impacts
Outcomes result from the appropriation and use of outputs greater or closer.
by these first project beneficiaries (NIFs, partnership The CDAIS project considered that expanded outcomes
stakeholders, national organisations, national innovation made the strengthening of the agricultural innovation
platform and CDAIS country partners). system irreversible, in contrast to primary outcomes that
A project can thus only influence outcomes; they are not may produce no effect in the absence of incentives.
under control of the implementing organisation.
Impacts:
• At the partnership level, outcomes correspond to joint Impacts refer to the long-term, sustainable changes in
actions, i.e. activities that are collaboratively decided the livelihoods of farmers, the state of the environment
and designed (with the support of NIFs) and executed and the conditions of rural poor, resulting from the spread
by partnership stakeholders and aimed at increasing or adoption of the innovations.
collective efficiency and thus collective capacity to Due to the long-term horizon and the increasing influence
innovate. Joint actions can address two different areas of a wide range of contextual factors over time, functional
related to internal and external issues from partnership capacity-development interventions can only contribute
stakeholders’ perspective, respectively: (partially and indirectly) to these enduring results in society
-- Management issues related to the innovation process or the environment.
itself, at the innovation niche partnership level (new
governance tools, new strategy, etc.)

CDAIS Monitoring, Evaluation and Learning 11

 The mel framework

3.2.4. Actors of change and threshold of irreversibility Not all outputs, outcomes and impacts in the theory of
change will be relevant in all countries. Thus, the exact
The different types of stakeholders in a capacity-development pathways of change will also vary by country depending
project are distinguished by their level of involvement: actors on the different capacity needs, resources and contexts.
from the direct project sphere, boundary partners and A capacity-development intervention may lead to very
beneficiaries. different results in different partnerships, in different
organisations, and in different innovation systems. Equally,
there may be outputs, outcomes and impacts that are
Boundary partners “are those individuals, groups, and
not represented in the ex-ante impact pathway. Only the
organisations with whom a program interacts directly
most common changes that are expected to occur can be
and with whom the program anticipates opportunities
described before interventions start. The ex-ante impact
for influence.” These actors are called boundary partners
pathway also gives information about the most important
because, even though the programme will work with
preconditions for achieving the project goals. The views of
them to effect change, it does not control them. The
stakeholders are particularly important for the MEL process,
power to influence development rests with the partners.
especially for identifying the possible mechanisms of change
The programme is on the boundary of their world. The
from their perspective as boundary partners in the project.
programme tries to facilitate the process by providing
access to new resources, ideas or opportunities for
There are also different options of pathways of change –
a certain period of time. A single boundary partner may
towards developing functional capacities and strengthening
include multiple individuals, groups or organisations if
the agricultural innovation system – depending on the
a similar change is being sought in all.
national context, the positioning and influence of the project
and other capacity-development situations or interventions
Actors from the direct project sphere are those,
in relation to agricultural innovation support that may happen
belonging to a boundary partner that are directly
in parallel to the project’s own activities. In countries where
working in the project; this is the case, for instance,
the concepts of agricultural innovation systems and capacity
of the members of an innovation niche partnership.
development for agricultural innovation systems are relatively
new, the focus is likely to be on raising awareness and
Beneficiaries are the wider range of actors and the
training in the CDAIS approaches. In more mature innovation
communities that, within or outside of the project sphere,
systems or in countries with stronger or more defined
benefit from the progress of direct actors and boundary
agricultural innovation policies, there is a greater possibility
partners, and, in the case of capacity development
that opportunities exist for engaging quickly in more
for agricultural innovation systems will benefit of the
collaborative work to bring about systemic change.
intended innovation.
Source: IDRC (2001).

12 CDAIS Monitoring, Evaluation and Learning

 The mel framework

Figure 2. Actors of change and threshold of irreversibility of the CDAIS project

Outcomes might be reversible if the boundary partners However, the theory of change assumes that the influence of
(individuals or organisations) do not have an incentive – or the project will decrease as a growing number of agricultural
overall favourable conditions in their environment – to put innovation system actors – including those at institutional
their functional capacities into use. In the early stages of and policy making levels – take ownership and control of the
a project, the project team plays this incentive role that is CDAIS approach itself. We assume that at a certain point
supposed to decrease as the agricultural innovation system (that may vary from a country to another) a threshold of
is strengthened (providing favourable enabling conditions). irreversibility should be crossed, ensuring the realisation
Capacity-development interventions implemented by the of the innovation while contributing to the strengthening
project should aim at creating the conditions for learning and of the agricultural innovation system as a whole.
acting in a transformative way, i.e. to innovate in a collective
manner, responding to farmers needs and ensuring impacts.

CDAIS Monitoring, Evaluation and Learning 13

 The mel framework

3.3. MEL at innovation niche partnership level: 3.3.3. Progress markers

Supporting and assessing capacity-development Progress markers describe the gradual or milestone changes
processes in a boundary partner as it progresses from its current
situation to full achievement of the innovation project, or the
3.3.1. Key evaluation questions improvement of its situation, from the very first steps right
• How did the coaching process of innovation niche through to deep transformative changes.
partnerships contribute to develop functional capacities
of partnership actors (individuals and organisations)? Identifying realistic and observable progress markers is
• How did the functional capacities of innovation niche essential to the success of outcome mapping. Progress
partnership actors contribute to accelerate or achieve markers provide a framework for observing changes in
innovation processes? boundary partners’ actions, interactions, relationships,
procedures or policies over time, and can measure the
direction of those changes in relation to the agreed intention.
Coaching process – The ‘coaching’ of innovation niche
partnership aims at supporting the collaborative work to
Within the MEL system, progress markers help reveal
achieve innovation. In the CDAIS project, the focus of the
patterns of changes during progress towards the capacities
coaching was to develop their functional capacities and
to be developed at the partnership level (see Table 4). They
achieve innovation.
are used by different groups of actors in different ways:

• by National Innovation Facilitators to fine-tune their

3.3.2. Participatory outcome mapping coaching process according to the achievements made
The technique of outcome mapping is used to capture the during learning cycles at t1 and t2 (see 4.1 The MEL
understanding and expectation of the different groups of phases and times);
people of an innovation niche partnership: direct partners, • by innovation niche partnership actors and other direct
boundary partners and potential beneficiaries. Outcome beneficiaries as milestones of their innovation project;
mapping helps innovation facilitators learn about the • by the country team as a tool for assessing outcomes
influence or progress of change among direct partners in the at the end of the project;
innovation niche partnership. It therefore helps them to think • by the global team as a tool for transversal (cross-country)
systematically and practically about what they are doing and analysis.
to adaptively manage variations in strategies to bring about
desired outcomes.

Outcome mapping starts from the view that development

comes as a result of complex interactions between different
actors, forces and trends. As it is difficult to attribute
development impact to interventions directly, outcome
mapping focuses on contributions of interventions to
developmental results. In doing so, it focuses on people.
Outcome mapping shifts the focus of development from
bringing ‘changes in states’, to ‘changes in behaviour,
relationships, activities or actions’ among those in innovation
niche partnerships. Progress markers are used as indicators
of those changes. They are statements of desired overall
behavioural changes that the boundary partners would like
to exhibit by the end of the capacity-development activities.
They are identified in a participatory manner at the beginning
of the project intervention and are regularly evaluated and
refined throughout the coaching process of innovation niche
partnership stakeholders.

14 CDAIS Monitoring, Evaluation and Learning

 The mel framework

Table 4. Examples of progress markers

Innovation Priority Strategy Capacities to be Expected changes: progress

niche objectives developed at the markers from stakeholders’
partnership innovation niche perspective
partnership level (extracts)
Drip irrigation Improve Develop a local Capacity to Expect to see:
systems for technology manufacturing collaborate • A leader of development organisations
small family and make it base for making is identified who acknowledges the
farms accessible to and repairing Capacity to reflect challenges to be overcome
(Burkina Faso) smallholder the equipment and learn from past • The leader has the requisite skills for
farmers all over needed experiences driving the agenda
the country • The leader supports the collaborative
Raise awareness development of a strategic road map
and knowledge to scale the technology
among
smallholders on Like to see:
the economic and • Public research is willing to implement
environmental a collaborative research and
advantages of the development project integrating local
technology artisans to co-develop technologies

Love to see:
• The Ministry of Agriculture agrees
to give priority to private sector
development in the scaling out of the
technology
• The Ministry of Agriculture agrees
to integrate the technology into its
advisory system

CDAIS Monitoring, Evaluation and Learning 15

 The mel framework

3.3.4. Embedding MEL and experiential learning cycles 3. Abstract conceptualisation – reflection gives rise to a new
At partnership level, the MEL system is embedded into the idea, or a modification of an existing abstract concept the
learning cycles of partnership stakeholders and innovation person has learnt from experience.
facilitators. Learning is the process by which knowledge is 4. Active experimentation – the learner applies the idea(s)
created through the transformation of experience, with the to the world around to see what happens.
experiential learning cycle comprising of four stages.
The MEL data are used as inputs for learning cycles.
1. Concrete experience – a new experience or situation is
encountered, or an existing experience is reinterpreted.
2. Reflective observation of the new experience – of particular
importance are any inconsistencies between experience
and understanding.

Figure 3. MEL embedded into learning cycles at the partnership level

16 CDAIS Monitoring, Evaluation and Learning

 MEL times and tools

4. MEL times and tools

4.1. MEL phases and times Phase three (t3): after completion of capacity-
development activities
At the country level, a three-phases approach has been • At both levels, outcomes are assessed, and a country-

adopted to implement the MEL system. wide analysis is performed, as is a cross-country analysis.
• MEL provides tools and methodologies to collect and

Phase one (t 0): before capacity-development compile end results at both levels, and combine them into
activities start ex-post outcome pathways.
• At partnership level: capacity needs are assessed,

innovation projects identified and explained, a coaching To have reference points to measure changes achieved by
plan is designed to develop the capacities of actors the project, various timings were proposed to set a tentative
towards realising their innovations. coordinated pace to the project implementation across the
• At national agricultural innovation system level: key eight countries:
organisations are identified, innovation facilitators are
• t0 is before capacity-development capacities are launched;
recruited, and the existing enabling innovation and
• t1 corresponds to the end of the first learning cycle,
agricultural policies are reviewed.
approximately 5–6 months after starting capacity-
• MEL provides tools to form a baseline, ex-ante outcome
development activities;
pathways and set progress markers.
• t2 corresponds to the end of the second learning cycle,
approximately 5–6 months after t1;
Phase two (t1, t 2): during capacity-development activities
• t3 is at the end of capacity-development activities.
• At partnership level: capacity-development activities are

implemented. Synchronisation of MEL and capacity-development activities

• At agricultural innovation system level: policy-dialogue in the three phases is shown in Figure 4.
activities are implemented.
• MEL provides tools to track and monitor ongoing changes

in the three dimensions (individual, organisational, systemic).

Figure 4. Times and phases of MEL in the CDAIS project

AIS: agricultural innovation system; CD: capacity development.

CDAIS Monitoring, Evaluation and Learning 17

 MEL times and tools

4.2. MEL tools at agricultural innovation system level

Table 5: Minimum common tools for MEL at the agricultural innovation system level in the eight pilot countries

Time Tool Short description Learning events

Scoping study A study based on interviews with key informants National validation workshop
is used to map agricultural innovation system Presented the results of the capacity
Baseline (t 0)

stakeholders, identify innovation political agendas, and needs assessment to agricultural

assess strengths and weaknesses of the agricultural innovation system stakeholders
innovation system and jointly validated pre-identified
possible impactful capacity-
NIF radar NIF radar is a self-assessment tool for National
Innovation Facilitators (NIFs), focused on progress development interventions
made in their skills, knowledge and attitudes

Stories of Stories of change is a tool for communicating progress Meetings of the technical project
change and the value of the CDAIS project in an accessible committee
way. Stories were regularly collected and written by The country team regularly met to
Monitoring

country teams adapt the implementation strategy

(t1, t 2)

on the basis of feedback from

Event log Event log is an online system that captures
(i) information on the organisation of events (when, participants and the identified
what, why, how), and (ii) participants’ evaluation and capacity-development needs
learning after each workshop or event

Self- NIFs assessed their progress for key skills for the Final assessment workshop
assessment facilitation of innovation processes and national CDAIS forum
radar for NIFs Validated the ex-post impact
pathway and designed an exit
Enabling A study based on interviews with key informants,
Assessing (t3)

strategy using inputs from the results

environment used to assess changes in the institutional and policy
of the MEL system
questionnaire context that enable innovation in the country

Ex-post Draws the causal relationships between inputs,

impact outputs, outcomes and potential impact, thus
pathway documenting how change was generated in the
agricultural innovation system through intervention at
all levels (individuals, innovation niche partnerships and
organisations) and the policy-dialogue activities

18 CDAIS Monitoring, Evaluation and Learning

 MEL times and tools

4.3. MEL tools at innovation niche partnership level

Table 6. Minimum common tools for MEL at partnership level in the eight pilot countries

Time Tool Short description Learning

Innovation The timeline is a method for joint reflection on a network Capacity needs assessment
timeline process. It helps participants to share perceptions on workshops and outcome
what is going on. It uncovers the history of the network,
pivotal moments and the next steps
mapping
With the help of the facilitator,
Network Network analysis takes stock of who forms the network of translating the theory of change into
innovation actors and the nature of interactions between actions
analysis Designing a coaching plan, including
them (provision of services, information and knowledge,
vision, capacity needs, capacity-
Baseline (t 0)

funding, etc.) and whether they are actually supportive to

the innovation project development strategies and progress
markers
Capacity This is an individual evaluation of the functional and
assessment technical capacities of the group that forms the innovation
niche partnership. A scoring tool and a coxcomb graph
questionnaire are used to represent the level of capacities
and scoring
tool
Progress Identify progress markers according to the capacity-
marker development strategy
identification
Progress Assess whether progress markers have been met, Reflection and refinement
marker exceeded, or whether there has been a deviation from the (R&R) workshops
initial progress markers that were identified
evaluation and Based on the monitoring results,
Monitoring (t1, t 2)

refinement reflecting on the coaching plan and

refining it if needed
Enriched Collect stories of small victories, learning situations,
innovation and failures with partnerships, organisations, and other
stakeholders including direct beneficiaries
timeline Help stakeholders to select the stories that drove the
most significant changes in the innovation process and in
their innovative capacities
Report them on the timeline drawn during the capacity
needs assessment. Write the associated stories

Capacity Assess the progresses made for each functional Evaluation workshop
assessment capacity in the course of the project, based on individual Based on the assessment of
perceptions progresses made, identifying with
questionnaire
innovation niche partnership actors
and scoring how they can continue by themselves
tool without external support
Assessing (t3)

Updated Assess the current actors in the network: who has

NetMap dropped out and who has joined?
Assess the nature of relationships between actors
(interpersonal, influenced by market stakes or
organisational stakes)
Assess whether these relationships have evolved during
the project

Contribution Contribution diagram links the project’s outputs to

analysis changes in capacities and related outcomes. It seeks to
describe and weight the contribution of the former to the
diagram latter

CDAIS Monitoring, Evaluation and Learning 19

 mel in practice

5. MEL in practice

5.1. Coordination of the MEL system The learning curve, however, proved arduous and was
often questioned by the teams themselves. The facilitators
The project established MEL teams at the global and would see it as an extra burden and control, so tools meant
country levels. The global MEL team helped to take informed for back-office management (e.g. the coaching plan) were
decisions, validate evidence of progress, and reinforce the presented to partnership actors and used directly for data
project goals and approach. It also provided methodological collection, adding more complexity to the complexity.
inputs, guidelines, backup and coaching of MEL country
teams. Progress markers generated a great deal of confusion at first,
as they are not part of standard monitoring and evaluation
MEL country teams were largely embedded in the respective (M&E) procedures. It was hard for actors to differentiate
country teams and worked in close collaboration with the progress markers from indicators, so they questioned why
team of innovation facilitators (i.e. the coaching teams) that CDAIS did not simply use indicators. The main turning point
were responsible for designing and implementing capacity- in the adoption of the MEL tools and concepts by the teams
development activities. The collaboration was fostered by was the first Reflection and Refinement (R&R) workshop,
creating spaces for reflection and sharing both before and organised after the first learning cycle: there the MEL tools
after capacity-development intervention cycles. (coaching plan, progress markers, innovation enriched
timeline, output timelines, analytical reports) started to
MEL country teams also had direct roles in helping the make sense because they allowed the teams to pause the
coaching teams prepare workshops, register the capacity- capacity-development process, take a step back to analyse
development processes and provide support to the it and, together with other actors, jointly measure progress
innovation facilitators through on-the-spot monitoring and made and make informed decisions about the next steps.
recording of participants’ reflections. In addition, they Having completed this learning process, the now skilled
provided analytical reports that helped the country team experts are an asset for their countries.
further analyse the activities of the project and interact
with the various actors. In several countries, the innovation
facilitators also took a share of MEL activities by participating 5.3. Challenges in gathering, analysing
in the recording and analysis of activities and participants’ and using information
actions and reactions.
5.3.1. Moving towards more consistent and
better-quality data
5.2. Perceptions of value and benefit As teams would not immediately perceive the relevance and
usefulness of the MEL tools and they lacked practice in their
On the one hand, the MEL system is an internal coordination use, the initial use of tools was erratic, and the data collected
and management tool, on the other hand it is a feedback was not as accurate as it could have been.
mechanism to those who designed the TAP Common
Framework and to the funders of the CDAIS project. Both Field support and joint analysis brought by the global MEL
dimensions were rapidly put into practice, but were not team gradually helped refine the collection and analysis of
immediately valued as such by the country teams, which data. A feedback loop was also created among the country
were dealing with actors in search of support for activities teams by which the more they performed MEL-based
that would serve their own purposes. There was therefore analysis and used MEL as a management tool – thanks to
a progressive shift from the global coordination team to the which they could not only show their results but also adjust
country teams in terms of perception of value, purposes their performance – the more they perceived its usefulness
and quality of the MEL system. This was largely due to a and relevance, and the more they harvested and provided
learning process of concepts that were unequally known quality data. Substantial progress was made in the last six
and mastered, but progressively started to make sense as months of activities due to a conjunction of the following
activities were implemented, monitored and evaluated and factors:
the proposed tools and concepts were actually applied. The
ability to measure the effects of the project was enhanced • one or more full learning cycles had already been
over time, as activities provided more ‘results’ to be implemented;
evaluated and analysed. Alongside this, the usefulness of the • the process for R&R workshops was detailed and
MEL system in enabling the required analysis also increased. workshops were organised in all countries;

20 CDAIS Monitoring, Evaluation and Learning

 mel in practice

• guidelines had been developed and agreed, concepts was shorter as support was provided on a regular basis in
were explained and in-the-field support was brought terms of both explaining the concepts and showing how and
to all countries; where to apply them concretely. However, in each country,
• the need to show results became more pressing as the a dedicated MEL focal person at the field level was hired and
end of the project neared. trained to drive the whole process.
The interest by all teams in understanding and measuring
the impact they had produced grew as a reasonable amount Equally, in all circumstances, MEL was rightly regarded
of data had been accrued from activities and a sufficient as a tool to support implementation, and its full adoption
length of time allowed had passed to enable retrospective required both the country team and the AFP to adopt it and
analysis and measurement of their own progress in delivering promote its use. With field support, it became clear that MEL
the project. is a common tool that can and has to be used by all team
members, particularly during workshops. All team members
5.3.2. Ensuring the right people participate then had the opportunity to try using MEL tools, and which
It is worth noting that when Agrinatura Focal Points were proved useful in the preparation of the R&R workshops and
permanently posted in the countries, the learning process other moments of analysis.

References

Astbury, B. and Leeuw, F. (2010) Unpacking black boxes: Mechanisms and theory building in evaluation.
American Journal of Evaluation, 31(3): 363–381.

Barret, D., Blundo-Canto, G., Dabat, M.H., Devaux-Spatarakis, A., Faure, G., Hainzelin, E., Mathé,
S., Temple, L., Toillier, A., Triomphe, B., Vall, E. (2018) ImpresS Methodological Guide. Methodological
Guide to Ex Post Impact Evaluation of Agricultural Research in Developing Countries. Centre de coopération
internationale en recherche agronomique pour le développement, Montpellier, France. 96pp.

Douthwaite, B., Kuby, T., van de Fliert, E. and Schultz, S. (2003) Impact pathway evaluation:
An approach for achieving and attributing impact in complex systems. Agricultural Systems, 78: 243–265.

Grovermann, C. (2017) Assessment of Innovation Capacities – A Scoring Tool. FAO, Rome, Italy, 33pp.

Earl, S., Carden, F. and Smutylo, T. (2011) Outcome Mapping: Building Learning and Reflection into
Development Programs. International Development Research Centre, Ottawa, Canada, 139pp.

Mayne, J. (2001) Addressing attribution through contribution analysis: Using performance measures sensibly.
The Canadian Journal of Program Evaluation, 16(1): 1–24.

Outcome Mapping Learning Community. Online discussions. https://www.outcomemapping.ca

Pawson, R. and Manzano-Santaella, A. (2012) A realist diagnostic workshop. Evaluation, 18: 176–191.

Pawson, R. and Tilley, N. (2001) Realistic evaluation bloodlines. American Journal of Evaluation, 22:
317–324.

Tropical Agriculture Platform (TAP) (2016) Common Framework on Capacity Development for Agricultural
Innovation Systems: Conceptual Background. CAB International, Wallingford, UK.

CDAIS Monitoring, Evaluation and Learning 21

Acknowledgements
This page contains the names of individuals and institutions who led the experimental work
and/or who contributed to the tests and improvements of the methodologies.

This document captures the framework into which the monitoring, evaluation and
learning (MEL) system of the Capacity Development for Agricultural Innovation
Systems (CDAIS) project was developed, funded by the European Union.

It was designed through a working group led by CIRAD (Aurélie Toillier, Agrinatura
Focal Person for Burkina Faso), in close collaboration with FAO (Christian
Grovermann and Manuela Bucciarelli), iCRA (Renaud Guillonnet, Hanneke
Vermeulen, Myra Wopereis-Pura) and the MEL country teams, especially MEL focal
persons Lampheuy Kaensombath (Laos), Elias Zerfu (Ethiopia), Aristide Sempore
(Burkina Faso), Rozana Wahab (Bangladesh), Oliveira Paulo (Angola), Jair Escobar
(Guatemala), Roduel Rodriguez (Honduras) and Straton Habumigisha (Rwanda).

Concepts and approaches were progressively tested and refined by all MEL country
teams before being consolidated in MEL training sessions and global workshops.

The use and adaptation of the MEL system was facilitated by the Agrinatura Focal
Person in each country: Claire Coote (NRI), Agrinatura Focal Person for Bangladesh;
Patrick d’Aquino (CIRAD), Agrinatura Focal Person for Laos; Stefano Del Debbio
(AICS), Agrinatura Focal Person for Honduras; Hans Dobson (NRI), Agrinatura Focal
Person for Rwanda; Nury Furlan (AICS), Agrinatura Focal Person for Guatemala;
Madalena Teles (ISA), Agrinatura Focal Person for Angola; Aurélie Toillier (CIRAD),
Agrinatura Focal Person for Burkina Faso, and Hanneke Vermeulen (iCRA),
Agrinatura Focal Person for Ethiopia.

22 CDAIS Monitoring, Evaluation and Learning

FINANCIAL SUPPORT GLOBAL PARTNERSHIP

This project was

implemented with the
financial assistance
of the European Union

IMPLEMENTING PARTNERS

Angola
• Instituto Superior de Agronomia (ISA), Universidade de Lisboa
• Instituto de Investigação Agronómica (IIA)

Bangladesh
• Natural Resources Institute (NRI), University of Greenwich
• Bangladesh Agricultural Research Council (BARC)

Burkina Faso
• Centre International de Recherche Agronomique
pour le Développement (CIRAD)
• Ministère de l’Enseignement Supérieur, de la Recherche
Scientifique et de l’Innovation (MESRSI)

Ethiopia
• iCRA
• Ethiopian Institute of Agricultural Research (EIAR)

Guatemala
• Agenzia Italiana per la Cooperazione allo Sviluppo (AICS)
• Ministerio de Agricultura, Ganadería y Alimentación (MAGA)

Honduras
• Agenzia Italiana per la Cooperazione allo Sviluppo (AICS)
• Secretaría de Agricultura y Ganadería (SAG)

Laos
• Centre International de Recherche Agronomique
pour le Développement (CIRAD)
• National Agriculture and Forestry Research Institute (NAFRI)

Rwanda
• Natural Resources Institute (NRI), University of Greenwich
• Ministry of Agriculture and Animal Resources (MinAgri)

CDAIS Monitoring, Evaluation and Learning 23

 COLOPHON

This publication was produced under the Agrinatura and FAO publication agreement (FAO
Ref.: AGDR/2-19/28 under license CC BY-NC-SA 3.0 IGO). This page is dedicated to individuals
and organisations who were the main designers of the work and writers of this publication.

Authors
Aurélie Toillier (CIRAD/Agrinatura), Renaud Guillonnet (iCRA/Agrinatura), Manuela Bucciarelli (FAO),
Hanneke Vermeulen (iCRA/Agrinatura), Myra Wopereis-Pura (iCRA/Agrinatura)

Copy-editing
Guy Manners, Green Ink

Design
Studio Teekens

Photographs
CDAIS unless otherwise stated

Printing
Drukkerij Tesink
This project was
implemented with the
financial assistance
of the European Union