Noted at 2/2021 CPR Meeting on 9 February 2021

Annex 5: The Programming Manual Toolbox

This annex provides practical tools to support project proposal design, development,
implementation, monitoring, and completion. Consider this a short course on project
proposal development and project work plan development for ASEAN cooperation
projects. Several tools are included, covering every stage of a project’s lifecycle (see
Figure 1).

Completion & Evaluation: Design & Development:

ASEAN goals
- Tool 12: Results-Based Emerging - Tool 1: Stakeholder
Reporting Issues Assessments
- Tool 13: Sustainable Projects - Tool 2: Problem-Tree Analysis
- Tool 14: Lessons Learned 1. Project - Tool 3: Objective-Tree Analysis
Design and
Proposal - Tool 4: Intervention Logic
Templates: Project Completion Development - Tool 5: Risk Analysis
Report, Financial Report. - Tool 6: Developing Indicators
4. 2. Appraisal
Completion and Templates: Project Proposal
Implementation & Monitoring: and Approval
Evaluation Project Budget, Work Plan.
- Tool 7: Monitoring Ch 5
Framework
- Tool 8: Survey Design 3. Implemen-
tation and
- Tool 9: Interview Techniques Monitoring
- Tool 10: Focus Group
Discussions Appraisal and Approval:
- Tool 11: Most Significant
Change Templates: Project Compliance
Review and Pre-Appraisal
Template: Monitoring and Sheet, Clarification Note,
Evaluation Framework. Disbursement Letter.
Figure 1. Overview of The Toolbox (Annex 5).

Detailed descriptions have been provided on how to use each tool, as well as
recommendations on what to keep in mind while implementing, or facilitating, a
project. When relevant, descriptions include why the tool has been recommended,
as well as how it can be used in the context of ASEAN cooperation project proposal
development and project implementation.

1
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 1: Stakeholder Assessments

Introduction

Project proposals are neither developed nor implemented in a vacuum. Many actors
and stakeholders play different roles, or might influence a project and its results, at
every stage of a project’s lifecycle. Those people planning a project must have a
broad understanding of its context, to aid development of a comprehensive proposal
and a work plan. Conducting a stakeholder assessment, as early as possible in the
project design stage, is recommended.

Stakeholder assessments are critical, since a key element of results-based

management is looking at results from the perspective of a project’s main targets
and beneficiaries. Results-based management answers the question what do project
results mean for beneficiaries.

Assessments should include input from key stakeholders, such as representatives of

the target group and the endorsing Sectoral Body, among other people. The project
team will need to engage with some or all of these stakeholders at various stages of
a project’s lifecycle. The more complex a project is, the more stakeholders will be
generally involved.

Using the Tool

Potential Stakeholder Checklist:
A stakeholder assessment is • Government departments, ministries
conducted to learn how different • Private sector companies
stakeholders might potentially • NGOs
influence a project and how a project • International Non-Governmental
Organisations (INGOs)
can best engage with them. Two
• Universities and Research
variables are accounted for in this
Organisations
assessment methodology:
• Civil Society Organisations
• Interests. To what extent • Target Groups, Beneficiaries
will stakeholder needs and • Community Organisations
interests be impacted–
whether positively or negatively–by the planning, implementation, or
outcomes of the proposed project?
• Influence. What power do stakeholders have over planning and
implementation, e.g., over decisions to be made, or over project

2
 Noted at 2/2021 CPR Meeting on 9 February 2021

implementation.

Follow these two steps when conducting a stakeholder assessment:

First, start brainstorming. List all the potential stakeholders in your project’s context.
It helps to do this exercise with staff members who will be involved in project design
and implementation.

Write each stakeholder’s name on a Post-It note, one stakeholder per note. While
there will likely be many stakeholders, it is best to be inclusive and have too many
notes, rather than omit a stakeholder who might later turn out to be important.

Second, determine interests and influence. Discuss and decide for each stakeholder
what their interests are and how influential might they be during project planning
and implementation.

As the discussion proceeds, attach the Post-Its to an interest/influence matrix, as per

the example in Figure 2. Make this matrix on a large sheet of paper, so that the notes
can be moved from one quadrant to the next, depending on the discussion. As seen
in Figure 2, the notes can be placed anywhere in the matrix. Even a note’s relative
positioning in a quadrant might be meaningful.

1. High Interest/Low Influence 2. High Interest/High Influence

School
Environmental Principals
Science Faculty Environmental
Social Media
Organisations
Students Environment Influencers
Ministry
Environment

Education
Ministry
Teachers

3. Low Interest/Low Influence 4. Low Interest/High Influence

Figure 2. Sample interest/influence matrix for a project aiming to increase green economy awareness and
engagement among schoolchildren.

3
 Noted at 2/2021 CPR Meeting on 9 February 2021

Third, formulate engagement strategies, based on the deeper explanation of the

interest/influence matrix in Figure 3.

1. 2.
High Interest/Low Influence High Interest/High Influence
Beneficiaries: Empower, make Potential allies: Partner, leverage,
capable. funding sources.

3. 4.
Low interest/Low Influence Low Interest/High Influence
Inactive stakeholders: Raise Advocacy targets: Lobby, raise
awareness, inform, or ignore. awareness, advocacy.
Figure 3. Description of interest/influence matrix.

In Figure 3, Quadrant 1, stakeholders have high interest and low influence. These
stakeholders might see the need for a project, and stand to benefit from it, but they
do not (yet) have much power to influence it. Often these are project beneficiaries,
or those close to beneficiaries. These people are best engaged by building their
capacities, strengthening their connections, and otherwise empowering them.
In Figure 3, Quadrant 2, stakeholders have high interest and high influence. These
are stakeholders who might be potential allies, or people with who an Implementing
Agency may wish to partner. They might include organisations with similar ideas and
aims. A good engagement strategy is to get to know these stakeholders better, learn
how to work with them to avoid repetitive effort, and discover potential synergies. It
may be possible to engage with them to raise additional project funding.
In Figure 3, Quadrant 3, stakeholders have low interest and low influence. While
these stakeholders might seem unimportant, they may become critical to a project
as work progresses. The goal should be to raise their awareness of the project’s
importance. In future, such stakeholders may become a necessary link to others.
Using the previous example, a journalist might not be interested in a project aiming
to influence youngsters on the green economy. However, a journalist who becomes
enthusiastic about the green economy might start influencing the Education Ministry
more constructively.
In Figure 3, Quadrant 4, stakeholders have low interest and high influence. These
are powerful people who are not (as yet) interested in a proposal, or who might
potentially oppose it. Positive engagement with these stakeholders requires energy
and effort, meaning it may be best to delay making immediate contact. A better

4
 Noted at 2/2021 CPR Meeting on 9 February 2021

strategy would be to start work with stakeholders from Quadrants 1 and 2 and
engage with stakeholders from Quadrant 4 at a later point, to ensure policy support
and strengthen long-term project sustainability.

Stakeholder Assessment Tips

• Be specific when writing the stakeholders names on Post-Its,

e.g., instead of writing ‘government’, write the specific
department or ministry. It might be helpful to write the
names of specific people in these departments.

• Invite a wide group of relevant team members to join the

exercise and encourage free brainstorming and discussions. It
may be useful to invite members of close partner
organisations. Discussion is as important as placement of the
Post-Its.

• Remember: The stakeholder assessment is a living document.

Regularly review and update the matrix during project
implementation, especially when faced with challenges
involving complex stakeholder relations.

5
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 2: Problem-Tree Analysis

Introduction

A thorough knowledge of a project’s context is needed to achieve meaningful and

sustainable project results. However, before deciding on a project’s design or
direction, there must be an understanding of the nature of the problem to be solved.
An analysis of the existing situation or the context surrounding the problem is
needed.

As with the stakeholder assessment, a problem-tree analysis exercise is best done

with key stakeholders and potential project partners. The stakeholder assessment
may also offer clues as to which stakeholders to invite to the problem-tree exercise.

Using the Tool

Problem-tree analysis is an exercise that identifies major issues related to specific

problems and constraints associated with the problem’s context. The analysis yields
a visualization of cause and effect relationships (see Figure 4) that aids the
identification of the main problem. After brainstorming and inventorying all the
issues, problems, and constraints that affect the project environment, an
understanding will emerge on the connections between smaller problems and the
main problem.

Effects

Main
problem

Causes

Figure 4. Problem-tree matrix.

6
 Noted at 2/2021 CPR Meeting on 9 February 2021

The problem-tree exercise includes the following steps:

• Brainstorming and inventory of all issues, problems, and constraints that group
members think are linked to the project context. Write all issues and sub-
problems on cards, one per card. Limit discussion and criticism at this point and
be inclusive. It’s best to write too much when brainstorming than to omit
anything that might later prove important.
• Once most issues have been identified and written down–this can take between
30 minutes to several hours–the group may start ordering cards logically,
placing causes below and effects above. Participants should ask which card
comes first, and which card comes after? Which cards are causes and which
cards are consequences? This is not always a question of true or false; there
may be different opinions. Allow this.
• Cluster cards in logical groupings, such as everything to do with government
policy, or teacher capacity, or student awareness. Analyse what comes first and
what comes next. What causes what?
• Fix the cards to a large sheet of paper draw arrows to indicate the logic. Figure 5
depicts a sample completed problem-tree matrix.

Increased public Decreased Decreased Decreasing

health spending productivity economic activity community
in the district income levels

Increasing health District less attractive for

Effects hazard in communities tourism and clean investors

High levels of waste and

air pollution in
communities

Low environmental Government doesn’t allocate

awareness among funds for green economy
teenagers policies
Causes

Communities do not Teachers aren’t Polluting Government

know about green interested in green industry has doesn’t prioritize
economy economy lot of power green economy

Figure 5. Example completed problem-tree analysis.

7
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tips for Problem-Tree Analysis

• This exercise is best conducted with the participation of a group of relevant

people, such as project team members and close stakeholders.
• Think out of the box, not just 'on your own street'. Actively invite
stakeholders outside the project team to contribute on their issues so that
tunnel vision from the perspective of the project team is avoided.
• Work with cards and place those on a large sheet of paper. Do not write
directly on the paper sheet, or fix the cards to the sheet too quickly, as you
will not be able to move the problem statements relative to each other
anymore. Encourage group members to move the cards in line with the
discussion, without fixing them to the paper.
• The discussion is as important as the final results of the exercise.
• Use sentences with verbs for each problem. People will have a tendency to
write single or just a few words on cards to represent problems, like
‘communications’ or ‘lack of awareness’. Ask them to be specific and write
the problem in a full sentence so that meaning becomes clear. Make sure
they indicate a verb, and an actor (e.g. ‘There is little communication
between Department X and Department Y’).

8
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 3: Objective-Tree Analysis

Introduction

This tool is directly derived from the problem-tree analysis. This flow ensures that
projects are designed with a problem’s context in mind. The people and stakeholders
from the problem-tree exercise should join the objective-tree analysis.

Using the Tool

Now that we know what the problems are that surround the project context and
how these problems are interlinked with the main problem and with each other, it
becomes easier to start designing the project. This can be done by reformulating the
problems from the problem tree into an objective tree. Each sub-problem must be
reformulated into a positive and realistic objective. See example of an objective tree
from Figure 6, directly derived from the problem tree-analysis.

The objective tree can be seen as the start of project design. It gives a good idea of
which short-term results, mid-term results and long-term results to aim for.
Tips for Conducting an Objective-Tree Analysis:

• This exercise is best conducted with the participation of relevant people,

such as project team members and close stakeholders.
• Think out of the box, not just 'on your own street'. Actively invite
stakeholders outside the project team to contribute on their issues so that
tunnel vision from the perspective of the project team is avoided.
• Work with cards and place those on a large sheet of paper. Do not write
directly on the paper sheet, or fix the cards to the sheet too quickly, as you
will not be able to move the problem statements relative to each other
anymore. Encourage group members to move the cards in line with the
discussion, without fixing them to the paper.
• Use sentences with verbs for each problem. People will have a tendency to
write single or just a few words on cards to represent problems, like
‘communications’ or ‘lack of awareness’. Ask them to be specific and write
the problem in a full sentence so that meaning becomes clear. Make sure
they indicate a verb, and an actor (e.g. ‘There is little communication
between Department X and Department Y’).

9
 Noted at 2/2021 CPR Meeting on 9 February 2021

However, the project may have neither the capacity nor the mandate to cover all
these objectives. For instance, it may want to focus particularly on interventions with
schools and students, but not with governments. This is fine as long–as it is clear that
higher-level objectives might not be fully achieved without engaging stakeholders
from governments, civil society, or the private sector. It will at least be necessary for
the project team to find out if other organisations cover these areas, and if possible,
to network or partner with them. This could also be an area for follow up after the
first project phase has been completed.

Steps in conducting the objective-tree analysis:

• The objective tree is directly derived from the problem tree. For each of
the cards in the problem tree, discuss what would be a realistic, positive
result if the problem would no longer exist. Write this on new cards.
• It may not make sense to directly rewrite the problem card into an
objective card, e.g., it is not realistic to change ‘polluting industry has a lot
of power’ into ‘polluting industry does not have a lot of power’. A more
realistic objective would be countering industry power with advocacy
power from civil society, so to formulate this objective statement as ‘civil
society advocates for green economy’.
• Place the cards on a new sheet of paper and review whether the order of
the problem tree still makes logical sense (see Figure 6).

10
 Noted at 2/2021 CPR Meeting on 9 February 2021

Decreased public Increased Increased economic Increased

health spending productivity activity in the community
district income

Long-term Decreasing health hazard Increased attractiveness of the district

results in communities for tourism and clean investment

Decreasing levels of
waste and air pollution
in communities

Increased environmental Government allocates funds for

awareness and engagement green economy policies and
Short-term amongst teenagers inclusion in school curricula
results

Communities Teachers are Civil society Government

understand the concept enthusiastic to teach advocates for prioritizes
of green economy about green economy green economy green economy

Figure 6. Example of an initial objective-tree analysis, before a proposal focus is chosen.

11
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 4: Intervention Logic

Introduction

A completed objective-tree analysis offers an indication of the potential direction

and goals for a proposed project. However, a broad analysis of a problem typically
yields objectives that are too broad for a single project to address. Thus, establishing
a project’s intervention logic can guide the identification of the relevant parts of the
objective-tree analysis that might realistically be incorporated into a project’s design.

The intervention-logic exercise, which determines an organisation’s mandate for

developing a problem intervention, may be conducted with a smaller, primarily
internal group than the objective-tree or problem-tree exercises.

Using the Tool

Intervention logic requires that an organisation look at the objective-tree analysis of

a problem and decide which parts of the analysis should be included in a proposed
project design. It answers the following questions:

• What fits with the organisation’s mandate?

• What fits with the priorities of targeted cooperation partners?
• What fits with the organisation’s skills, knowledge, and expertise?
• What hasn’t been addressed sufficiently by other organisations or project
interventions?
• Where can the organisation make the greatest difference?

12
 Noted at 2/2021 CPR Meeting on 9 February 2021

Decreased public Increased Increased Increased

health spending productivity economic activity community
in the district income levels

Long-term Decreasing health hazard Increased attractiveness of the

in communities district for tourism and clean
results
investment

Decreasing levels of
waste and air pollution
in communities

Government allocates funds

Increased environmental for green economy policies
awareness and engagement and inclusion in school
Short-term amongst teenagers curricula
results

Communities Teachers are Civil society Government

understand concept of enthusiastic to teach advocates for green prioritizes
green economy about green economy economy green economy

Figure 7. Example of developing an intervention logic, based on an objective-tree exercise, with the area of
focus for the proposal in red.

In the green economy education programme example, the organisation decided to

work with schools and communities, and not with governments or civil society
organisations (see Figure 7). However, the problem-tree and objective-tree analyses
indicated that that civil society and government engagement was important for
influencing policy and allocating funding to solve the problem. Therefore, it is
important to know if other entities are working to influence the same areas, so that
the organisation’s project results may deliver sufficient results. The organisation
might network or partner with these other organisations, to align approaches and
achieve project synergy.

Start developing the intervention logic by determining the level of control that an
organisation has over the individual elements of the objective tree. Organisations
control elements in three ways. First, there are factors that an organisation fully
controls. Second, there are factors that an organisation influences, where it can

13
 Noted at 2/2021 CPR Meeting on 9 February 2021

boost its control through implementation of high-quality projects. Third, there are
factors beyond its control, where an organisation has little influence and is
interested in making a contributing in the future.

Outputs are results that an organisation fully controls, and comprise the direct
results or deliverables of a project’s activities or interventions. Results often include
services or goods delivered or accessed by target groups, knowledge, skills, attitudes,
or awareness raised.

Outcomes are results that an organisation can influence. Outcomes stem from
outputs, and comprise changes to a target group’s behaviour or practices.

Impacts are long-term results to which an organisation may contribute.

Organisations can influence, as opposed to control, impacts, since many other actors
and factors influence such long-term results.

14
 Noted at 2/2021 CPR Meeting on 9 February 2021

Intervention Logic How to Formulate

Impact Statement Needs a clear link to ASEAN’s strategic
Ultimate benefits for target objectives, and regional social, economic,
population, long-term results. environmental, and political conditions.
Formulated in perfect tense, e.g., ‘improved’,
‘strengthened’, ‘increased’ or ‘reduced’.
Outcome Statement Typically, one or two outcome statements
Short-to-medium term changes per project, in ‘perfect tense’:
in project context. Not what a • Increase volume of regional trade
project does or delivers, but how • Reduced level of domestic violence
target groups utilise its outputs. • Increased number of students
Outputs Typically, two or more outputs will generate
Delivered or provided products one outcome, formulated in ‘perfect tense’:
and services, whether tangible or • Study completed/published
intangible, or access created. • Journalist trained/skilled in
• Report produced/presented
• Skills built/improved
• Access to finance created
• Draft legislation prepared
Activities Typically, two or more activities will produce
Tasks undertaken to produce one output, formulated as things done, in
outputs. ‘present tense’:
• Provide technical assistance
• Develop training
• Organise workshops
• Publish newsletters on…
• Procure equipment for…
• Engage consultants to…

Figure 8. Examples of how to develop intervention logic.

15
 Noted at 2/2021 CPR Meeting on 9 February 2021

Impact Level

Outcome Level

(Intermediate)
Outcome Level

Output Level

Figure 9. Examples of intervention logic with levels of control.

Based on the colour codings in Figure 10, a project-results hierarchy can be

developed (Fig. 9). This is the intervention logic, which is the foundation of a
project’s logical framework.

Figure 10. Colour-coded hierarchy for developing an intervention logic.

The other elements of the logical framework will be discussed in the next section.

16
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 5: Risk Analysis

Introduction

Often projects fail or do not achieve intended results because potential risks were
not accounted for initially. If
risks are analysed during the PESTLE Examples
Category
project design stage, a project
P Political • Government Policy
plan can make plans to
• Political Stability
mitigate risks before or during • Corruption
implementation. Risk analysis • Foreign Trade Policy
involves predicting or • Tax Policy
anticipating what might • Labour Law
happen during project • Trade Restrictions
E Economic • Economic Growth
implementation, as well as
• Exchange Rates
determining factors that might • Interest Rates
interfere with achieving a • Inflation Rates
project’s results or which • Disposable Income
might reduce the uptake of • Unemployment Rates
deliverables by target groups. S Socio-Cultural • Population Growth Rate
• Age Distribution
This tool takes you through • Career Attitudes
conducting a simple risk • Health Consciousness
• Lifestyle Attitudes
analysis and provides a guide
• Cultural Barriers
for formulating mitigation T Technological • Technology Incentives
strategies. • Level of Innovation
• Automation
Using the tool • R&D Activity
• Technological Awareness
Risks are external factors that And Change
might negatively affect the L Legal • Discrimination Laws
achievement of results or the • Employment Laws
successful implementation of • Consumer Protection
a project. While designing a Laws
• Copyright and Patent
project, it is important to Laws
anticipate potential risks so • Health And Safety Laws
E Environmental • Weather
Figure 11. PESTLE categories. • Climate
• Environmental Policies
• Climate Change
17
 Noted at 2/2021 CPR Meeting on 9 February 2021

that (i) a project’s design can be adapted, e.g., by developing additional activities and
outputs to reduce the chance of risks occurring, and (ii) a project may include
contingency plans to mitigate the effects of risks that emerge during
implementation.

There are three steps for conducting a risk analysis:

1. Look at the project context to determine where risks might occur. Let
discussions be guided by the PESTLE checklist (Figure 11), noting that some
categories may not apply, depending on the project.
2. Brainstorm and inventory potential risks. Write risks on individual Post-Its. Be
specific and avoid overly simple statements, such as ‘corruption’ or ‘natural
disaster’. Brief descriptions are too broad and will cause confusion when
assessing risks.
3. Assess risks in terms of:
• Probability. How likely is it (high/medium/low) that this risk will
happen in the project context?
• Potential Harm. If the risk occurs, how much harm
(high/medium/low) will be done to project implementation and
results?
4. Copy the risk matrix (Figure 12) below on a large sheet of paper. Place Post-its
according to the group’s assessment of the risk.:
Probability of Occurrence

Potential Harm
Figure 12. Risk matrix.

18
 Noted at 2/2021 CPR Meeting on 9 February 2021

5. Consider the risk colour-coding schema in Figure 13 and the risk management
strategies in Figure 14:

High
Medium
Low

Low Medium High

Figure 13. Colour-coded risk matrix.

Risk Category Risk Management Strategy

Negligible Risk Ignore, no measures necessary if risk does not become
more serious.
Low Risk Monitor risk to ensure it does not become more serious,
add its non-occurrence as an assumption in the logframe,
or consider adding mitigating activities to the project plan.
Moderate Risk Mitigate risk: Make adaptations to the project plan by
adding activities, and add assumptions to the logframe.
High Risk Mitigate risk: Redesign project plan by adapting or adding
outputs, create a contingency plan to deal with the risk
should it occur.
Killer Risk Do not continue before rethinking project, including
whether it is wise to start work at this stage.
Figure 14. Risk-management strategies.

19
 Noted at 2/2021 CPR Meeting on 9 February 2021

Using the previous green economy project example might result in the risk analysis
in Figure 15.

Risk Probability Potential Risk

Harm Category
1. School principals do not
prioritise the green economy Medium High High
curriculum.
2. High-polluting industry starts Low High Moderate
near the village.
3. Students not interested in
organising green economy Low High Moderate
events.
4. Political unrest in the country
prevents economic and tourism Low Medium Small
growth.
Figure 15. Example of a completed risk analysis.

Activities can be added to the intervention logic to decrease risks. For example, in
the example for Risk 1, school principals may not prioritise the green economy
curriculum, is a high-risk category, which has been assigned a high potential harm
and a medium probability of occurrence. Accordingly, a project team might add
outputs to mitigate the risk, decreasing it from a medium probability to a low
probability, such as an activity to build awareness among the principals.

While Risk 2 (appearance of a new industrial site) and Risk 4 (political unrest) are
almost impossible to predict or influence, risk management dictates that an
organisation continue to monitor these risks during programme planning and
implementation, refining the project work plan and adding outputs and activities, as
needed.

20
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 6: Developing Indicators

Introduction

While the results statements in Tool 4 (intervention Logic) tell us what we want to
achieve through projects, indicators are the first step in operationalising how these
results can be monitored. Results often cannot give a clear indication of how
progress or achievement can be measured, which is why indicators must be
developed.

This step is generally done by the project team, with support from monitoring and
evaluation (M&E) specialists, if possible. It may be helpful to involve target groups in
this step to formulate realistic indicators, which would also enhance their
commitment and ownership.

How to Develop Indicators

Indicators can be developed by answering the following questions:

• What will increase or decrease due to the project intervention?

• How will the project ascertain differences or changes before and after the
intervention? How can this be measured?

Traits of Well-defined, Objectively Verifiable Indicators

Variable Element measured (what?) No. of articles on circular
economy
Quantity Actual situation vs. situation to be Increased by 25%
reached (how much?)
Target People affected by the project (who?) No. of articles published by
Group mainstream media journalists,
trained by the project
Place Location concerned (where?) In a specific province
Period Timeframe. When should target be In 2021
reached (when?)

Figure 16. Traits of well-defined, objectively verifiable indicators.

21
 Noted at 2/2021 CPR Meeting on 9 February 2021

Examples of complete indicators can be formulated as follows:

‘Number of articles (or percent increase in articles, compared to the baseline)

published on the circular economy in the mainstream media by journalists
trained in our workshops in English and in City A in the year 2020.’

‘Number of awareness raising events on the circular economy organised

independently by the student clubs in City B by 2020.’

Once potential indicators are

Means of Verification (Primary Data Collection)
identified, establish where
• Ways of collecting evidence to support indicator
information on the indicators be measurement.
found: • Examples: Surveys, interviews, interviews,
observations, photo evidence.
• Does an indicator comprise
secondary information that Sources of Verification (Secondary Data
can be obtained from other Collection)
sources or organisations? • Location of evidence to support indicator
measurement.
• Does an indicator comprise
• Examples: Project reports, meeting minutes,
primary information that
national census, statistical data, etc.
must be collected by the
organisation itself? Where
and how will the information be collected?
• Is measuring the indicator feasible? Does the organisation have the right tools,
equipment, and resources (time, staff, finances)? Can data be collected in a
culturally appropriate and sensitive manner?

During implementation and monitoring, indicators are used to measure project

progress against the approved work plan to see if a project is on track or needs
redirection. The project team may conclude, based on monitoring indicators, that it
will not be possible to achieve project results as intended. In that case, the work plan
must be updated and the Proponent or Implementing Agency must inform key
project stakeholders, including the ASEAN Secretariat, relevant Desk Officer, or
Sectoral Body.

At the completion stage, project performance is assessed by comparing achieved

indicators in the project completion report against planned indicators in the
approved project proposal. Please the table of Project Result Achievements below.

Results Indicators Reasons for Deviations

22
 Noted at 2/2021 CPR Meeting on 9 February 2021

Planned Achieved

Objective:

Output:

Output:

Output:

23
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 7: Monitoring Framework

Introduction

After developing a project’s intervention logic; finalising indicators, and sources and
means of verification; and completing the risk analysis; it is time to develop a
project’s results framework, or logframe. The results framework is the basic tool of
project management, and its elements has been integrated into the ASEAN
Cooperation Project Proposal Template, in Section 3: Project Results.

However, the results framework does not provide sufficient guidance on project
monitoring. It does not, for instance, indicate the baseline levels for indicators at a
project’s start. Lacking baseline information, measuring indicators during
implementation will not capture needed information. A results framework also does
not detail any data collection methodologies, nor does it specify who will measure
indicators, or the frequency of measurement. This level of detail is often not
available during project proposal development.

However, once a project proposal is approved, this information must be included in

the monitoring framework. An organisation’s monitoring and evaluation (M&E)
Officer or M&E team, if present, will develop the monitoring framework. In their
absence, developing a monitoring framework is the job of project management.

The monitoring framework (see Figure 17) has nine columns and must include the
following information from the results framework.

1. Results Logic. List all planned project results, as in Section 3 of the ASEAN
Cooperation Project Proposal Template. If the results framework is adapted or
further detailed during the inception of a project, update the monitoring
framework as well. Results logic is sometimes called intervention logic (Tool 4).
Example: ‘Increased environmental engagement among teenagers’.
2. Indicator. Tool 6 explained how to develop indicators, which are the
measurements that show if project is achieving its intended results, outcomes,
or outputs. Indicators are the backbone of the monitoring framework and guide
monitoring work.
Example 1: ‘Increase in number of awareness-raising events organised by youth
groups in project areas between January 2020 and December 2021.’

24
 Noted at 2/2021 CPR Meeting on 9 February 2021

Example 2: ‘Percent increase in blogs on the circular economy published on

social media by youth influencers trained in project workshops in a specific
language in 2020.’

3. Baseline. How can an indicator be measured accurately at the start of a project?

Sometimes this is easy, if the indicator is something that starts with the project,
as in the example. In this case, the baseline measurement would be zero. If we
are measuring something that already exists, or is already happening and is
hoped to increase due to the project, like the number of articles published on
the circular economy, we need to measure this indicator at the start of the
project. Without measurement, it will be difficult to know the actual increase
during project implementation.
Example 1: Baseline 0.
Example 2: Baseline average two blogs per targeted youth influencer per month.

4. Target. How much change is targeted during the duration of the project? A
multi-year project might target change on a longer timeline, measuring results
annually or quarterly.
Example 1: Target for two events in Year 1, eight events in Year 2.
Example 2: Target 10 blogs per trained youth influencer per month.

5. Data Sources. Where can information be found? It is important to ask this

question when developing indicators. All too often, interesting indicators are
formulated during the project design phase, but turn out to be difficult, costly,
or unreliable to collect in practice. Take, for example, an awareness-raising
event run by youth groups. Is the organisation in contact with these groups? Is
the organisation involved in the events? Does the organisation also need to do a
survey of the blogs run by the youth groups? Can the organisation find another
way if this is too complicated?
Example 1: Youth group project progress reports.
Example 2: Facebook and Instagram metrics.

6. Methodology. Data sourcing and methodology are closely connected. Some

information is readily available and can be reviewed or consulted with ease.
However, sometimes project managers will need to develop a methodology to
extract data from a given source.
Example 1: Reviewing youth group project progress reports.
Example 2: Facebook and Instagram matrix study or an online survey among
trained youth influencers.

25
 Noted at 2/2021 CPR Meeting on 9 February 2021

7. Frequency. How often will data be collected? Monitoring of data collection

should be done more frequently at the output level (e.g., quarterly) than at the
outcome level (e.g., annually or at project completion). Impact indicators are
generally not monitored during a project, and are instead evaluated after
project completion.
Example 1: Annually.
Example 2: Twice a year.

8. Responsible. Who is responsible for collecting data? While this might be a

dedicated M&E Officer or project manager, often the project team has
responsibility for collecting data, as they are present in the field and have easy
access to target groups.
Example 1: Project Officer.
Example 2: M&E Officer.

9. Reporting. Where and how is monitoring data reported? This is an important

question, as the right people must receive the right information to make
appropriate decisions at their level, without the burden of information
overload. For example, a project manager primarily needs monitoring
information on the output level, to assess if activities led to the intended direct
results, or whether changes need to be made in the implementation of
activities to improve results. While information on outcome-level indicators is
also valuable to project management, it is even more valuable to higher-level
management, who must assess if a project’s design is still relevant, or if design
modifications can still be made during project implementation or for a next-
phase project.
Example 1: Mid-Term and Closing Reports.

An example monitoring framework is presented in Figure 17.

26
 Noted at 2/2021 CPR Meeting on 9 February 2021

Figure 17. The Monitoring Framework

1. Results Logic 2. Indicator 3. Baseline 4. Target 5. Data Source 6. Methodology 7. Frequency 8. Responsible 9. Reporting
Overall Objective
(Impact) Youth Group
… Project Annually
Project Objective 0 Progress
(Outcome) Report
…
Immediate
Outcome 1
…
Output 1.1
Increase in
…
Awareness-
Output 1.2 Raising Events
… Organised by Project
Youth Groups in Officer
2020: 4
Immediate Project Areas
Outcome 2 2021: 8 Review Of Mid-Term
from Jan 2020 To
… Dec. 2021 Youth Group and
Increased
Output 2.1 Progress Completion
Environmental
… Report Report
Engagement
among
Output 2.2
… Teenagers

27
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 8: Data Collection Methodology-Survey Design

Introduction
In project design, indicators are used to establish what will be measured for
monitoring, and to establish means and sources of verification for measurement.
Data for indicators may be available in an organisation, outside the organisation, or
from project documentation, like training implementation reports. However, project
managers often must do additional work to collect data to monitor indicators, such
as by designing a survey for workshop participants. Accordingly, this tool provides
practical guidelines for survey design.
Surveys done for monitoring may be conducted by project or field staff or
outsourced to external consultants or enumerators. Surveys are often compiled by
monitoring and evaluation (M&E) officers or external consultants.

Using the Tool

A survey is a set of questions administered to a group of respondents, who are taken
as a sample from a population, to get data about a number of questions or
indicators. In the context of ASEAN cooperation projects, survey answers are used by
project managers to monitor and evaluate progress. Therefore, a survey’s
questionnaire must capture the needed information to measure progress in
achieving indicators and also be relevant to the survey’s objectives.

Follow these steps when creating a questionnaire:

1. Write a list of objectives for the survey, based on the indicators from the
logical framework or monitoring framework. These objectives must illustrate
the big picture for data collection and will be the basis for selecting questions.
2. Determine the type of questions to be used. There are two main types of
questions.
• Structured questions, with closed or fixed responses. Respondents are
given selection of answers from which to choose. This type of question is
used when there is a certainty about expected responses, definitions, or
categories, and if it is not essential to record an original answer.
Examples of structured question formats include multiple choice,
rankings, yes/no, and rating scales.

28
 Noted at 2/2021 CPR Meeting on 9 February 2021

• Open-ended or unstructured questions. These questions are useful for

gathering new or original answers, or individual answers from
respondents. Results are difficult to organise, group, or analyse
systematically. Any question that does not limit the scope of answers is
an open question.
3. Determine a survey method, e.g., direct interview, telephone interview, web-
based survey or application, or a written questionnaire.
4. Ensure that questions are:
• Comprehensible. Use simple language and keep questions clear and
concise. Be sure to ask who, what, where, when, and why, as needed, so
respondents have every option when providing answers.
• Answerable. Respondents must be able to answer accurately, without
having to do research. For example, people might not be able to answer
exactly how much money they spent dining outside the home last year,
but they might be able to estimate how much they spent last week.
• Focused. Avoid double-barreled questions. Make sure that each survey
question has only one query to answer. For example, instead of asking if
a respondent likes to exercise in the afternoon or after eating, break the
question into two: One question related to afternoon exercise and
another about exercising after eating.
• Not overlapping. Ensure that answer choices do not overlap. For
example, multiple choice questions should not use the same number for
choices related to two answer ranges, for example between 10 and 30
or between 20 and 40.
5. Arrange survey questions in a logical sequence.
• If your questionnaire covers diverse topics, group questions by subject
or theme.
• Equate sensitive questions by grouping them with neutral questions, and
putting them at the end of the survey, after a sense of closeness or
rapport is obtained.
6. Design the questionnaire so that it is easy to read or easy to scroll through.
• Use large font sizes and clear fonts.
• Leave a gap between questions, so it's easy to see where one question
ends and the next begins. Make sure there is enough space to write
longer answers, if you use open-ended questions.

29
 Noted at 2/2021 CPR Meeting on 9 February 2021

• The space for answers must be placed immediately after or under the
question. Avoid putting questions on one page and asking the
respondent to turn the page to answer.
• Use page numbers.
7. Give respondents information that is clear and sufficient so surveys can be
completed according to the rules. To ensure superior results, a questionnaire
should prepare respondents to answer by adhering to the following precepts.
• Explain the survey’s purpose. When respondents understand the
reasons behind a question, they are more likely to provide accurate
personal information.
• Give clear guidance for completing questions. Explain the format of the
question (e.g., multiple choice, rating scale, etc...). Give examples of how
to answer the question correctly. Offer clear instructions, such as to read
all questions before answering, or to try guessing an answer rather than
leaving a question blank.
• Tell respondents how many questions are in the questionnaire, and
provide an estimated completion time.
8. Make improvements to the questionnaire as needed. Whenever a survey is
administered, analyse its results with an eye toward making changes that will
increase its effectiveness in the future.
9. Test questionnaires on co-workers or target groups before use. If certain
questions are consistently skipped, those questions may need to be revised
to make them clearer
• If the respondent cannot give a full answer due to space restraints, you
can change the layout.
• If a simple yes/no answer does not give you the range of data you want,
then you might want to change to a multiple-choice format.

Surveys are especially useful in capturing information related to trends, and can
offer quantifiable information on qualitative indicators. However, surveys
generally cannot ascertain why trends are occurring. That is something more
easily done through interviews, focus group discussions and analyzing the most
significant change–the subject of the next sections of this Annex.

30
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 9: Data Collection Methodology-Interviewing Techniques

Introduction
Interviews are a commonly used methodology to collect the data and information
needed to measure the indicators devised for a project’s logframe. The main
advantage of interviews, as compared to other methodologies, is the collection of
qualitative information on why certain trends or behavioral changes have emerged.
In a monitoring context, interviews can be done by project or field staff, or by
monitoring and evaluation (M&E) officers, to create distance between those who
implement a project and those who measure its progress and results. Interviews can
also be outsourced to trained interviewers.
Interviews can be used to collect data on qualitative and quantitative indicators. A
well-prepared and well-done interview delivers valuable information. There are
simple techniques can make any interview more effective: active listening, proper
question techniques, and monitoring content, structure and process. These are
briefly discussed below.

Using the Tool

An interview is a specific situation where an interviewer obtains relevant and reliable

information from an interviewee. The interviewer has no intention to share
information: their only task is to get the person interviewed to provide needed
information.

Interviewers must focus their attention on several aspects of the interview at the
same time, namely:

• the contents of information delivered by the interviewee,

• the structure of the interview and directing the interviewee, and
• on motivating the interviewee to provide information by ensuring that the
process goes well.

The interviewer must constantly ask themselves during an interview if this is the
information that is really needed. Do I understand what is being said? Did I get
sufficient details? In this way, the interviewer judges answer as they are provided.
Interviewers must also ask themselves additional questions: What is the value of
what is has been said? How does this information relate to what is already known?
Can this answer be true?

31
 Noted at 2/2021 CPR Meeting on 9 February 2021

Structuring the interview, e.g., determining how questions are asked or answered, is
the explicit task of the interviewer. When an interviewee starts talking and keeps
jumping between subjects, the interviewer must bring the subject back on the track.
An interviewee will only feel motivated to provide the right information if they feel it
is the right thing to do, if the atmosphere is conducive, and if the interviewer is
trusted.

Interviewers must ask themselves the following questions during an interview:

• How do I introduce the subject?

• How can I get more than superficial answers?
• How do I stop the interviewee from talking too much?
• How do I stimulate an interviewee to talk more?
• Is the interviewee committed to the interview?
• Is the interviewee withholding information?
• How can I create trust with the interviewee?
• How does the interviewer/interviewee relationship feel?
• How is the relationship affecting the content and procedure of the
interview?

Active Listening
In an interview context, active listening is a skill that helps interviewees answer
questions themselves. Efficient interviewers, as active listeners, understand both the
content of what an interviewee says and the feelings that go with it. Active and
attentive listening means that the interviewer empties their mind to make room for
the interviewee’s perspective, and is receptive to the ideas and experience of others.

Active listening involves the following steps:

• Listening attentively. Listening is one of the best ways to communicate,

improve relationships, and develop understanding. Listen with an open
mind and an open heart. This will take practice if it does not already come
naturally.
• Checking your understanding. Repeat or summarise an interviewee’s
answers, so they know that they have been heard and give them the
opportunity to correct any misunderstandings.
• Proceeding with the interview. Let the other person continue to answer or
ask another question.

32
 Noted at 2/2021 CPR Meeting on 9 February 2021

During an interview, active listening works on several levels. Content, or answers,

are verified with the interviewee to ensure their message has been understood and
that the interviewer and interviewee share a similar understanding of the answers.

Structurally, active listening allows an interviewer to lead the interviewee back to

the point if they lose track, stopping them politely if answers are too long and
summarising their main points. At the level of process, active listening makes the
interviewee feel that the interviewer is seriously engaged with them and is aware of
their point of view. This has an enormous positive effect.

Asking Questions
The only way to get information during an interview is to ask questions. However,
the way questions are phrased influences the answers received. Interview questions
are either open-ended or closed.

• Open-ended questions (e.g., ‘Tell me about your experience.’) invite an

interviewee to answer in their own words, as briefly or as long as desired.
• Closed questions are structured to limit responses to yes, no, or simple
data. (e.g., ‘Do you run your own business?’ ‘When were you born?’).

Open questions are so-called accelerators, stimulating a response. Closed questions

are so-called brakemen, discouraging lengthy replies. An open question can provide
an interviewer with information on something that they do not know much about,
while a closed question offers specific information on something already known or
something to be checked.

Structurally, using closed questions makes it easier for an interviewer to control a

conversation. Closed questions also need less time to be answered. On the level of
process, use both open and closed questions. Asking too many closed questions
irritates an interviewee and offers no chance for them to discuss information that
they deem important. On the other hand, too many open questions might confuse
the interviewee, making them suspicious: Why does the interviewer want to know
this? What do they want from me?

A good balance between open and closed questions is important. Striking that
balance depends on the time available, purpose of the interview, and the
information needed.

33
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 10: Data Collection Methodology: Focus Group Discussions

Introduction
The focus group discussion (FGD) is a qualitative research method that collects data
and is used to gain a level of understanding that may not sufficiently be obtained
from a survey. It is used to validate outcomes–such as changes in behaviour, actions,
relationships, or policies–that have been reported by project implementers, and to
understand why certain trends in results have been observed. FGDs are a powerful
tool for engaging with target groups and stakeholders, so as to enhance their
commitment to and ownership of project results.

FGDs are implemented by trained facilitators, who typically have some distance from
the project team. Facilitators might be monitoring and evaluation (M&E) officers, or
outsourced consultants or facilitators, to ensure a professional and objective
approach.

What Is an FGD?
A focus group is a semi-structured interview of small groups of 6 to 10, maximum 12,
led through an open discussion by a skilled facilitator. They run from 45 to 90
minutes. The moderator nurtures disclosure in an open and spontaneous format to
generate a maximum amount of discussion and opinion sharing. Typically, an
assistant moderator or facilitator will take notes, observe the process and the body
language of the participants (especially for sensitive topics), and ensure that the
atmosphere of the discussion is comfortable for everyone. An assistant moderator
from the community who understands the local culture and issues is invaluable.

While free flowing, a FGD should have a maximum of 10 questions. To obtain valid
results, more than one focus group is needed to discuss a topic. Typically, three or
four are required. Sufficient FGDs have been held when it is found that new sessions
do not generate new information.

Participants should be roughly homogenous in terms of age, gender, and power. This
will allow participants to speak freely, without domination by one or a few
participants. In some cases, it may be best if participants do not know each other, in
order to reduce inhibitions when discussing sensitive topics. Participants can be
selected randomly or by nomination.

The role of the moderator or facilitator is important, and that person must be
carefully selected. Ideally, an FGD moderator has adequate knowledge of the topic,

34
 Noted at 2/2021 CPR Meeting on 9 February 2021

is able to stay neutral, and refrains from volunteering their opinion. Likewise, a
moderator should be able to listen and think at the same time, and able to listen
attentively, with sensitivity and empathy. Moderators must believe that every
participant has something valuable to say about the topic of discussion and will be
given the opportunity to offer their input, regardless of education, experience, or
background.

There are three types of FGD questions:

• Engagement questions. Designed to introduce participants, make them
comfortable with the topic of discussion, and explain the use of
information collected during the FGD.
• Exploratory questions. The main questions that lead participants to the
core of the discussion.
• Exit questions. Reiterating and verifying the points generated in the
discussion, while checking to see if anything was missed.

The typical flow of an FGD is:

• Welcome and introduction of the facilitation team.
• Explanation of the process and the main reason for the FGD.
• Ground rules.
• Consent for recording and clarification.
• Introduction of participants.
• Discussions (use the type of questions above).
• Summarising findings and identify what is missing.
• Closing: Thanking the participants.

Compile and Analyse

Notes of the results of the FGD should be processed immediately. They can be
compiled in many ways, such as in Figure 18.

CODING OF
FGD CODE PARTICIPANT ID PARTICIPANT TYPES RESPONSES (one idea for each row)
OUTCOME

Figure 18. Sample table for processing FGD results.

35
 Noted at 2/2021 CPR Meeting on 9 February 2021

The team can process this information and draw conclusions as to whether there is
enough credible evidence to validate an outcome, and give weight or importance to
the intervention’s contributions to observed changes.

36
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 11: Data Collection Methodology-Most Significant Change

Introduction

The data collection tools presented so far in Annex 5 have focused on collecting data
to measure indicators for the results framework. It is essential for project managers
to have a vision of their intended achievements before implementation begins, so
results can be measured.

However, project activities, whether direct or indirect, can yield unintended changes
that are also worth analysing. The Most Significant Change (MSC) Technique was
designed to measure intended and unintended changes, as it is less dependent on
pre-formulated results and indicators. MSC is generally facilitated externally, while
engaging a wide range of project staff and stakeholders to ensure project learning.

Origins

The Most Significant Change Technique was developed by Rick Davies in 1996 for
monitoring and evaluating complex participatory rural development programmes in
Bangladesh. The technique was refined by Jessica Dart in Australia, and she and
Davies developed a clear and useful guide to MSC that is available for free download:
www.mande.co.uk/docs/MSCGuide.htm

Summary

The MSC Technique is a form of participatory monitoring and evaluation that does
not use indicators. Instead, the MSC Technique collects stories of significant change
(SC) at the field level. Panels of stakeholders or staff then systematically select the
most significant stories. This method, also known as the ‘story approach’ or
‘evolutionary monitoring’, is flexible and has several advantages over indicator-
based monitoring:

• Suitable for identifying unexpected changes.

• Identifies prevailing values.
• Easy to use, does not require special professional skills.
• Encourages analysis and data collection.
• Useful for conceptualising impact.
• Delivers a rich picture of what is happening.

37
 Noted at 2/2021 CPR Meeting on 9 February 2021

• Monitors and evaluates bottom-up initiatives that were not foreseen in the
programme or project design.

Implementing MSC is a ten-step process:

1. Generate Interest. Using the MSC Technique is recommended when the

significance of outcomes is under discussion or there is a need to identify
unexpected outcomes.

2. Define Domains of Change. Defining a domain (area) to be considered will focus

the search for specific change stories. However, this may also exclude other relevant,
but completely unexpected, changes.

3. Define the Reporting Period. Do this at the pace of monitoring, e.g., monthly,
quarterly, etc…

4. Collect SC Stories. Most important. This can be done in many different ways. For
example, fieldworkers may collect unsolicited stories that they have heard,
systematic interviews with structured note taking may be conducted, group
discussions may be organised, or beneficiaries may write down their stories directly,
among other things.

Stories must be documented, including information on who collected the story,

when the event occurred, a description of the story itself (what happened to whom,
how, when, where), and, last but not least, an assessment of its significance to the
storyteller.

On ethics, each storyteller owns their story. Organisers must obtain permission
before using it. Storytellers must be informed about the organisers’ intended use of
their stories.

5. Select the Most Significant Stories. Another important step. The selection process
is flexible, and may be done differently at different levels. It involves the
participation of beneficiaries, whose availability depends on time, logistics, and the
basic questions underlying the study, which sometimes may preclude their
involvement.

How stories are selected depends on the unique perspective of a stakeholder, e.g.,
beneficiaries, community organisations, supporting organisations, governmental or

38
 Noted at 2/2021 CPR Meeting on 9 February 2021

non-governmental organisations, regional or national authorities, and cooperation

partners.

Different techniques are used in selecting stories, which should be done preferably
by groups, whose members will discuss and select stories by consensus, e.g., through
voting or scoring, etc…. Selection criteria may be predefined by the terms of
reference of the process. However, it is best to have open selection criteria so that
consensus can emerge though discussion.

6. Feedback. Stakeholders must be advised after the most significant stories have
been selected. Feedback may be delivered in various ways, including verbally, by
email, newsletters, or formal reports.

It is important that storytellers receive feedback, through whatever medium, about

the outcome and the conclusions of the MSC process, as well as about how selection
was conducted. This will complete the communication loop between different levels
and may enhance cooperation.

Feedback motivates project beneficiaries. Reading success stories helps generate

new ideas. However, feedback also risks generating frustration and neglect when
storytellers learn that their stories were not deemed significant.

7. Verifying Stories. Verification may not be necessary if proper selection occurred,

meaning that stories that were unreliable or biased were excluded. However, it may
be useful to return to the field and verify the MSC stories to exclude deliberately
fictionalised accounts, misunderstandings, incorrect stories, and exaggerations.

8. Quantification. While the MSC Technique emphasises a qualitative approach, it

may be applied quantitatively, as well, in these ways:

• Stories may include quantitative elements that may be analysed further.

• Elements of a selected story may be present in other stories, making this a
subject of a quantitative review.
• A quantitative analysis may be made of all SC stories.

9. Secondary Analysis and Meta-Monitoring. While not mandatory, secondary

analysis and meta-monitoring adds legitimacy and rigour to the MSC Technique by
studying how the process was implemented, by classifying and analysing content

39
 Noted at 2/2021 CPR Meeting on 9 February 2021

across a set of SC stories, as well as their attributes, such as who liked a story and
who selected it.

10. Revising the System. The MSC Technique should be used as a continuous and
flexible monitoring system. It should not be done in a ritualistic way or become a
continuous reflection on the monitoring system. This will lead to frequent changes in
the domains of change, frequency of reporting, types of participants, and the
structure of meetings to select the most significant stories, etc.

Recommendations for Using MSC

The Most Significant Change Technique is an interesting approach for monitoring

and evaluation, particularly in cases where the outcome of an intervention is not
clear and the aim is to learn lessons from actual practice.

It cannot replace a monitoring and evaluation system linked to a logical framework

with measurable indicators. Current trends are to focus on accountability, such as
through measurable indicators that are established at a project’s inception.
However, accountability risks over-emphasising the relationship between planning
and results, which may be full of uncertainties and surprises. Excess attention to paid
measurable results might blind a project team to unexpected but valuable events
that may occur. After all, the outcomes of a project depend, to a limited extent, on
the results of the planned interventions, and are subject to many other influences
that are beyond the scope of a particular intervention.

From this perspective, the MSC Technique is a simple and complementary

instrument that may help a project team learn lessons from the daily practice of
beneficiaries. While beneficiaries may not represent measurable indicators, they
certainly have important stories to tell. In this way, the MSC Technique may keep
development projects and programmes well-tuned to the reality of the target
groups.

40
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 11: Tips for Results-Based Reporting

Introduction

Project reporting is often seen as a formality or something required by cooperation

partners. Under a results-based approach, however, reporting reflects learning,
which is essential for improving project execution and achieving more meaningful
and sustainable results.

Using the Tool.

Results-based reporting can be visualised as a knowledge triangle (Figure 19) that

illustrates how to process monitoring data into meaningful data for results-based
reports.

Data collection for measuring indicators is the first step in understanding a project’s
progress toward achieving its intended results. This step typically involves raw facts
and figures, especially for quantitative data, such as the number of workshop
participants or articles published. Raw facts and figures are found in project
documents and surveys, to an extent. However, raw data is not typically meaningful
without analysis.

Data is only made

meaningful through
processing. Data must be
organised and put in
context, e.g., how many men
and how many women?
Which months had the best
attendance? Do trends
appear over time?

Subjecting raw data to

simple analysis produces
information. However,
information often does not The Knowledge Triangle
reveal why specific trends
have emerged. For example,
Figure 19. Knowledge triangle.
although assumptions can be

41
 Noted at 2/2021 CPR Meeting on 9 February 2021

made on gender imbalances in workshops, there can be no determination as to why

that is so without analysing data in an organised way. And yet, assumptions and raw
data are typically highlighted in project reports: Raw data indicate trends, which are
justified by assumptions, which have been in turn made without proper justification.

Qualitative tools, such as interviews or FGDs, can be helpful in analysing raw facts
and figures. For example, participants might be asked why they attended a
workshop, or why a certain approach was easier to apply. This results in knowledge:
information, with the understanding of why things happened. This is the goal of
project reporting: documenting trends and changes in the behaviour of target
groups, with a justified understanding of why these changes have happened.

Ideally, there should be another level of understanding for project reporting, and for
completion reports, specifically, when, why and how to use the knowledge we
generated in the previous level. This final level in our knowledge triangle is wisdom.

While the jump from knowledge to wisdom comes partially from experience, it can
be enhanced through facilitating learning sessions for the organisation and project
team (See Tool 13: Lessons Learned).

For reporting, it is important to realise that it is not sufficient include raw data in
project reports. Raw data offers an insufficient understanding as to why certain
results have been achieved or not, and is of limited use for guiding decision making
and future programming.

Tips for Report Writing

Readability
● Use active language.
● Write in the third person, e.g. ‘our project will’ becomes ‘the project will’.
● Alternate long and short sentences.
● Avoid jargon and woolly language.
● Be clear and concise: delete unnecessary words.

Appearance
● Always follow formats.
● Check spelling and grammar.
● Be consistent in style, e.g., using British vs American English, in writing currencies,
etc…
● Be consistent in layout.
● Ensure bullet points are easy to read.
42
● Use visuals: infographics, charts, pictures.
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 12: Developing and Implementing Sustainable Projects

Introduction

Results-based project management looks beyond the duration of a project. Results are
not of value if the progress a project makes in solving a problem vanishes after
implementation. Results must be sustained to be meaningful.

What Is a Sustainable Project?

A project is sustainable when it delivers an appropriate level of benefits for an

extended period of time after major financial, managerial, and technical assistance
from an external donor is terminated (DAC). The focus is on sustaining the flow of
benefits. Figure 20 describes the factors that contribute to a project’s sustainability
and how those factors may be supported.

Sustainability Description How to Support this Factor

Factor
Host Projects operate in the context of Include project components at
Government national policies. Government output level to influence policy
Policies commitments and policies that support formulation or implementation,
project objectives are critical for e.g., advocacy or support
sustaining development activities. mobilisation.
Management, Sustainability is enhanced when project Ensure high levels of participation
Organisation, objectives are well matched with the from relevant stakeholders.
and administrative and managerial Include project components to
Participation capabilities of local partner strengthen managerial and
organisations. For projects with benefits organisational capacity of local
directly associated with local partner organisations that will
populations, participation is critical for play roles in continued project
sustainability. Local participation is an interventions and results.
integral part of continuing the flow of
benefits after termination of a donor's
activity.
Finance Sustainability requires a flow of funds to Facilitate connections between
cover operations, maintenance, and target groups and potential
depreciation of investments to continue funding sources, including
the benefits generated by a project. governments and private sector,
or build their capacity to mobilise
resources in other ways.
Technology Technology chosen for the activity Ensure that technology and
must be appropriate to the country's equipment used by the project is

43
 Noted at 2/2021 CPR Meeting on 9 February 2021

financial and institutional capacities, suitable for the country context

and to the project’s goals. and financial capacities of the
target groups (maintenance).

Socio-Cultural Integration of a programme with the Ensure that project

socio-cultural setting of its beneficiaries interventions are suitable for
and their operating circumstances is the target group’s socio-cultural
important if the activity is not to be context. Include relevant
rejected after assistance ends. awareness-raising and
sensitization components in the
work plan.

Environment Unplanned development can accelerate If necessary, include an

and Ecology depletion of natural resources, environmental impact
threatening the ability of the assessment at the start of the
environment to renew itself and thus project and integrate
threatening the sustainability of recommendations in the
projects. project design.

External Development projects operate within Conduct a risk assessment in the

the context of existing political, project design phase to analyse
economic, institutional, and cultural the relative potential impact on
circumstances that are beyond a the project. Develop risk
project’s control and influence. Yet, management and mitigation
the project and its intended results plans. Ensure risks are monitored
can be deeply affected by external continuously throughout the
factors, such as political and economic project and action is taken as
instability or natural disasters. necessary.

Figure 20. Factors contributing to project sustainability.

Strengthening Project Sustainability:

• Set out activities and results to be sustained after the project’s end.
- For example, student clubs continue to organise awareness raising events to
train fellow students to understand and apply green-economy principles in
their daily practices and future businesses.

• Formulate critical questions for each sustainability factor. Scrutinise the project
purpose, results, activities, and assumptions in light of these questions, e.g.:

44
 Noted at 2/2021 CPR Meeting on 9 February 2021

- Will schools allow the students to organise such events?

- Do student clubs have the needed organisational capacity?
- Will student clubs be able to raise the financial resources to organise
activities?
- Do student clubs need specific equipment or technology to organise
activities?
- Do project activities fit well with the culture of student clubs and the
schools?
- Are activities organised in a green and environmentally friendly manner?
- What will happen if the school leadership changes?

• Based on the answers:

- Rethink or add results, activities, assumptions, or preconditions, e.g.,
▪ Consider including an output related to the level of school
management buy-in, to ensure long-term support.
▪ Consider including leadership training for the student clubs.
• Commission further studies, e.g.,
▪ An organisational capacity and development study with student clubs.
• Formulate recommendations for implementation.

45
 Noted at 2/2021 CPR Meeting on 9 February 2021

Project sustainability is about

making sure that the activities of Q: What is project sustainability?
a project can be carried on by an
organisation, generally the A project is sustainable if the outputs of
project target group, after the project can continue after the project
is completed.
project’s implementation has
ended. Sustainability, of course, is Q: How can sustainability be strengthened?
also related to environmental and
other governmental factors that • Including relevant follow-up activities
can only be partly influenced by to support implementation.
project design. Some • Through national initiatives.
sustainability factors can be • Planning for next-phase, recurring
projects.
strengthened at the project level;
• Establishing networks with other
whereas others are limited to an relevant institutions.
assessment of organisational • Conducting a thorough risk
capacity or the continuous assessment, including mitigation
monitoring of contextual factors strategies.
(e.g., political, environment).

46
 Noted at 2/2021 CPR Meeting on 9 February 2021

Tool 13: Drawing Lessons Learned

Introduction

The results-based project approach emphasises learning. This is not a coincidence.

Achieving meaningful and sustainable results through projects requires the integration
of tools so project managers can learn from challenges and mistakes to achieve better
results in the future. This also ensures project learning within the duration of the
project itself. Those involved in drawing lessons learned include all project staff, as
well as key stakeholders, such as target groups and project partners.

This toolbox offers two structured methods for drawing lessons learned: the after-
action review (AAR) method and developing a programme of action.

After-Action Reviews

The after-action review process was developed by the military as way to learn from
the experiences of troops under difficult conditions. An AAR is a structured
debriefing that is used to analyse a project in order to determine what happened,
why things happened, and how a project or activity might be better implemented in
the future.
An AAR can be used to review an activity as well as a whole project, offering input
for a progress or completion report, or as a basis for designing a next-phase project.
After-action reviews, which can also be used as a guiding tool for project review
meetings, will typically:

• focus on why things happened,

• compare intended results with realised results,
• encourage participation, and
• emphasize trust and value feedback.

AARs quickly transfer critical lessons and knowledge in order to maximise benefits.
During the review, team members directly involved with implementing activities
have an opportunity to share their experiences, which other members can use
expeditiously to improve the performance of the whole organisation in a timely
manner.

47
 Noted at 2/2021 CPR Meeting on 9 February 2021

For an AAR process to be successful, a team must discover for itself the lessons to be
learned from their experiences. The more honest and open the discussion, the
better. Typically, the questions asked during an after-action review cover these
themes:

• What was supposed to happen? What did happen? Why was there a
difference?
• What worked? What didn't work? Why not?
• What would you do differently next time?

Organising Principles for AAR Meetings:

• Be inclusive. Invite all project staff and representatives of key stakeholders.

• Mistakes should be welcomed as important sources for learning and should
not launch a blame game.
• Brainstorming is a useful methodology for AARs: All ideas are welcome, and
criticism is banned–or at least temporarily suspended.

Developing a Programme of Action for Lessons Learned

This is approach is more analytical that the AAR process. While learning refers to
increasing knowledge or skills, a lesson refers to an action programme based on the
prior experiences of ourselves and others. A lesson comprises a concrete action plan
for improving work in the future with the goal of producing better outcomes. The
process of creating lesson involves several stages:

• Searching for previous experiences, whether positive or negative.

• Analysing cause-and-effect linkages from what was observed during the search.
• Creating a lesson, i.e., a new action programme (what is to be done differently)
based on the lesson.
• Evaluating the lesson in the context of a completed project, or estimating the
consequences of adopting the lesson and speculate about what might happen in
the future if the lesson is applied. Also called prospective evaluation.

48
 Noted at 2/2021 CPR Meeting on 9 February 2021

Creating a Lesson Learned

Lessons based on the experiences of others may be devised in several ways:

• Adopting. Copying processes that were implemented elsewhere. An

organisation can also choose not to adopt such a process.
• Adapting. Adjusting processes that were implemented elsewhere to account
for differences in context.
• Hybridizing. Combining elements of processes that were implemented at
several different organisations.
• Synthesizing. Combining elements of programmes from various organisations
to create new programmes.
• Inspiration. Using programmes elsewhere as an intellectual stimulus to
develop a new programmes.

Evaluating a Lesson
The applicability of a lesson is Desirability
dependent on many factors, High Low
as per Figure 21. What will
Unwanted Technical
High

work for one organisation or Doubly Desirable

Practicality

problem might not work in Solution

another context. A
Low

prospective evaluation Siren Call Doubly Rejected

means assessing the
likelihood that a learned Figure 21. Applicability of lessons learned.
lesson will be effective if
applied to a project’s specific
situation.

The practicality of a lesson refers to the technical feasibility of a programme, e.g.,

whether approaches employed elsewhere could be done at an organisation.
Desirability refers to “political feasibility”, e.g., consistency with the values and goals
of those who are evaluating it. Siren call is cautionary. It is a warning not to wreck a
programme by adopting a lesson based on its goals without regards to means.

Conclusion

The after-action review process and the action-programme development methods

can be used for periodic monitoring or review during project implementation or
evaluation. However, for either method to be effective, discussions must be open

49
 Noted at 2/2021 CPR Meeting on 9 February 2021

and non-judgemental. All participants, e.g., project staff, partners, and key
stakeholders, must feel there is a safe space to discuss challenges and areas for
improvement without judgment or blame. Mistakes need to be viewed as
opportunities for learning. It may be helpful to ask someone from outside the team,
or an external facilitator, to facilitate such review.

50
Noted at 2/2021 CPR Meeting on 9 February 2021

51