Design data-driven

programmes that deliver

results effectively
A step-by-step guide for programme managers
in international development
Design, Capture, Understand, Act
#withAkvo
Contents

Introduction 04
1 | What is a data journey and why do you need it for your programme? 06
2 | How a design phase will help you improve data-driven decision making 09
3 | How to conduct a context analysis 11
4 | How to identify your programme’s steps toward impact 15
5 | How to design a results monitoring framework 20
6 | How to choose the right tool for effective results monitoring 22
7 | How to conduct data research for your programme in four steps 24
Conclusion 26
About Akvo 27
Credits 27

03
 Introduction

Why this eBook? programmes involve multiple partners with

This eBook guides programme managers and different resources spread across the globe.
consortium leads through every step in the design High quality data is essential in making informed
phase of the data journey1, resulting in programmes decisions and improving programme results. But
that deliver results effectively. This eBook will help gaining a comprehensive overview of what data
you to: is needed to make better informed programme
decisions can be a challenge. For programme
• Align distributed teams and bolster co- managers and consortium leads, this can lead
ownership of your programme to an overload in admin, miscalculated budgets,
• Achieve clarity and oversight on every aspect of and unclear results. So how do you design your
your programme programme to ensure optimal results and smooth
• Make confident management decisions based implementation?
on accurate and reliable data
• Showcase your results to the people who need Gaining clarity on the opportunities, challenges
to see them and objectives from the start is essential. What
• Gain lessons learned to improve future problem do we aim to address and which factors
programmes will influence our goals? What work has already
been done and who was involved? What do we
Data-driven programme design need to monitor to be accountable, steer and learn?
The challenges facing the international How can we effectively track implementation
development sector are complex. Often, across different locations? What data do we need
to monitor results? How can we ensure that the
1  https://akvo.org/our-approach/
data collected is used for decision making? These

Introduction
04
 questions are at the heart of any successful data-
driven development programme. However, if they
Design are not addressed right at the start, they can lead to
problems down the line. Why have we overspent on
Gain clarity on the context of your the budget? Why do we have so much unnecessary
programme, the problem you are trying data? Which data still needs to be collected due to
to solve, the data you need, and the roles inaccurate and inconsistent collection methods?
and responsibilities of each partner.
Share insights with the Data quality over quantity
relevant people, generate In the development sector, streams of data pass
dialogue, encourage decision
through our hands, inboxes, and online tools
making and continuously
improve your work.
every day. But collecting data is much easier than
discovering knowledge2, and many organisations
Act still struggle to make the most of the data they
collect. With the definition of the Sustainable
Capture Development Goals (SDGs), the development sector
has embraced large scale and large volume data
Capture reliable and high use. Too often, however, data is collected using
quality data from the start. inconsistent methodologies, which can lead to data
Monitor your data collection that is not useable and/or comparable. Sometimes,
to ensure accuracy and track additional data is collected just because there is
Extract the insights that matter. Clean,
progress. the opportunity to do so, leading to data-fatigue
analyse and visualise your data and turn it
and a lack of concrete results for both the collecting
into valuable information and knowledge.
organisations and the people from which the data is
collected. Lastly, data is not always shared, causing
Understand other organisations to collect the same data and
waste resources.

2  https://akvo.org/blog/what-we-talk-about-when-we-talk-about-
data-science/

Introduction
05
1 | What is a data journey
and why do you need it for The data journey methodology consists of four phases: Design,
Capture, Understand and Act. They form the starting point for

your programme? organisations to ensure data is used to contribute to lasting and

inclusive impact. These phases aren’t always consecutive or
prescriptive, there may be some overlaps, and it may be necessary to
go back to a previous phase due to findings at a later stage.
Design
Context analysis What does each data journey phase consist of?
Theory of Change
Website / reporting Monitoring framework Design
Open data sharing Data research Gain clarity on the context of your programme, the problem you are
System integration Survey design trying to solve, the results you are trying to achieve, the partners you’ll
Data informed decision making Data collection plan work with, the data you need to monitor progress and improve decision
Sampling plan making, and the roles and responsibilities of each partner. The design
Act
phase should enable you to define your data needs and prepare for a
smooth data capture process.
Capture
Prepare data collection Questions to consider include:
Data collection • What is the context you are operating in? Who is involved and what
Verify data collection is their role?
Data cleaning Monitoring cycles • Which impact do you want to achieve and which outcomes will
Data analysis contribute to it?
Data visualisation • Which data will you need for which purpose, which data already
Understand exists and which do you still need to collect?
• What does the optimal survey design look like to ensure success?
• Which sampling plan fits best and is most cost-effective?
1 | What is a data journey and why do you need it for your programme?
06
Capture Understand Act
Collect relevant, high quality data from the start. Clean, analyse and visualise your data and turn it Share insights with the relevant people, generate
Implement your data collection plan and track into valuable information. Extract the insights that dialogue, encourage decision making and
progress. Questions to consider include: you can act upon. In the Understand phase of your continuously improve your work. In the Act phase,
• Are the tools and skills and logistical plan in programme, you can generate information which you’ll share your data to influence change.
place to commence data collection? can be interpreted to extract insights.
• How can you verify and ensure the quality of • How will you share findings with the key
your data on the go? • What data sources are you planning to combine? audiences?
• How best do you organise monitoring cycles of Is your data clean and ready for analysis? • Which systems does the data need to be stored
repeated data collection? • How can you extract insights from your data? in?
• How will you visualise the data and ensure • How can you amplify your insights and create
effective data storytelling1? lasting impact?

1  https://akvo.org/blog/five-tips-for-effective-data-storytelling/

For programme managers and consortium leads,

following this data journey will ensure smooth and
successful implementation of your programme,
allowing you to focus on improving and showcasing
your results. In this eBook, we’re focusing on the
Design phase of the data journey.

1 | What is a data journey and why do you need it for your programme?
07
Round out your
data journey knowledge
Out later this year

Download the eBook now Visit our knowledge library

08
 2 | How a design phase will
help you improve data-driven
decision making

Designing a programme in a constantly changing The methods explained in this chapter are geared If you have a certain impact in mind that you want
environment can be challenging. The context in towards optimising the outputs of your programme to contribute to as a programme, a ToC helps you
which a programme operates is often complex, design and implementation while emphasising the to understand which different outcomes you need
with many different stakeholders and factors importance of the design process in itself, which will to achieve in order to reach your envisioned impact
involved. For that reason, it’s important to base the help stakeholders align and feel ownership of the and how these outcomes are interrelated. While the
design of a programme on an understanding of the programme. It’s important to carefully document word impact refers to the ultimate change that your
context and choose an approach that allows for the entire design phase in order to capture lessons programme aims to contribute to, the outcomes are
flexibility instead of stasis. That way, adaptations learned that can be shared within the sector and changes that need to happen in between. Designing
can be made when needed. Even a relatively can be used to feed into future programmes. a ToC together with all stakeholders will result in a
straightforward data collection programme needs This approach to programme design is Theory of common understanding and co-ownership of the
to keep in mind which stakeholders will be involved Change1 (ToC) based, a methodology which helps programme and will facilitate the planning of your
and which problems and opportunities exist, at the you to structure reality and understand how your activities in a participatory way. It will also help
start and throughout. programme can contribute to a process of change. you to discover what you collectively want to learn,
and therefore to decide what you want to monitor
1  https://akvo.org/blog/akvo-theory-of-change/
during the programme.

2 | How a design phase will help you improve data-driven decision making
09
This chapter takes you through the three steps Step two: Design your Theory of Change
involved in programme design before going into Define an impact that you want to reach or
depth in the following chapters. It’s important to contribute to with your programme and think
remember that, while the steps are presented in backwards. Which outcomes need to be realised to
a sequence, the three are circular in nature. For reach the impact, and how are they interconnected?
example, in order to be able to map all relevant Which strategies will help to achieve these
stakeholders, there needs to be an awareness outcomes? Make sure that all your underlying
of the context and of what the problems and causal assumptions are recorded and made explicit.
opportunities are. You might realise after the Chapter four takes you through each step of
context analysis that some important stakeholders designing a ToC.
were overlooked during the analysis. Each step in a
ToC based programme design can make you realise Step three: Build a monitoring framework
that something was missing or not clear enough in a From the ToC, collectively agree on what key
previous step and may lead to revisions. expected outcomes (and impact) all stakeholders
want to monitor. For those, design a planning,
How it works in three steps: context monitoring, evaluation and learning (PMEL)
analysis, Theory of Change, and monitoring framework, with indicators and means of
framework verification. In addition, monitor the causal
assumptions that you are unsure about. Based on
Step one: Conduct a context analysis your monitoring findings, the ToC should be revised
Before designing the ToC, you need to have a on a yearly basis and adjusted accordingly. Chapter
thorough and common understanding of the five and six explain the best way of building your
context in which your programme is operating. results framework and choosing your monitoring
Therefore it is good practice to start with a context and reporting tool.
analysis, which consists of a factor, issue and
stakeholder analysis, and maps of the findings. All three steps require a highly participatory
Chapter three describes in detail how you can approach, to ensure relevance and co-ownership
conduct a context analysis. from the start.

2 | How a design phase will help you improve data-driven decision making
10
 3 | How to conduct a context analysis

Before designing a programme, it is crucial to understand the context in which it

is operating to ensure that everyone involved has a similar understanding of the
situation and that the programme is designed to address the relevant issues with
the right people. Context analyses are often outsourced to external consultants.
However, our experience shows that some of the knowledge, understanding
and connections that the external consultant acquires during the analysis may
be lost in the transfer of information. Therefore, we suggest having the context
analysis done in a participative way with skilful insiders; having the wisdom
in the room by inviting the right people. This way, the exercise can result in a
deeper understanding and higher usefulness to the design of the programme.
The context analysis usually consists of two interlinked exercises:
• Mapping and analysing stakeholders
• Mapping and analysing factors

Map and analyse the stakeholders

The participatory stakeholder analysis helps you to identify and map all relevant
actors and their roles, responsibilities, relationships, interests, and relative
influence/power. Relevant refers to the actors who have something to do with
the impact the programme aims to contribute to. It is important to make sure
all actors are taken into account, including vulnerable and underprivileged
groups who may otherwise be overlooked. This exercise is of most value
when performed in a group or workshop setting, and will help identify which
3 | How to conduct a context analysis
11
strategic stakeholders need to be involved in activist campaigning for health rights. Informal coded cards (different colours for government,
your programme, and in what way. During the stakeholders can be very influential but are easily civil society, private sector, knowledge institutes),
programme, power relations may change and overlooked. and organised according to the level at which they
new stakeholders may appear or knowledge gaps are most active (international, national, regional,
regarding existing stakeholders may be filled. It Create a stakeholder map district, community). Links between actors can be
is therefore advisable to review your stakeholder In a stakeholder map, all actors that are relevant for indicated with different types of lines.
analysis on a regular basis. the programme’s success are noted down on colour-

There are three types of stakeholders for you to

consider: Stakeholder map
1. Communities: The people who experience the

Regional International
problem directly, and interact with problem
Platform/ Private
solvers. CSO
network sector actor
2. Problem solvers: The civil servants, non
governmental organisation (NGO) staff, frontline
responders, and others on the ground.
Platform/
3. Policy and decision makers: The people who CSO
network
have access to resources and control allocation,
or can influence decision making.
Public
Public Private
National
Private CSO sector actor
It is important to include informal stakeholders in CSO
sector actor sector actor
the analysis and not only the formal ones. Formal sector actor
stakeholders are institutions or people with legal
status, such an government entities, private
Public Private
District

Public
corporations or NGOs. An informal stakeholder is a CSO
sector actor sector actor sector actor
person or group of people without legal status, but
with a vested interest in the impact the programme
aims to contribute to. For example, a formal Private
Village

stakeholder could be a member of the ministry of CSO sector actor CSO

health, and an informal stakeholder could be a local
3 | How to conduct a context analysis
12
 Create an influence-interest matrix • What is their influence on the problem? • What could they do to undermine the
Once all stakeholders have been made explicit, you • How might this person benefit from the programme?
can create an influence-interest matrix to discover programme? • What is the best way to keep them engaged?
more about each actor’s interest in and influence on • What could this person do with better data on • How can they contribute to a solution?
the data collection and data-based decision making the problem?
in the programme. The following questions can be • How does data support this person’s decision All of the actor cards can be plotted on four
useful in this discovery: making now? quadrants of the y-axis (interest) and x-axis
(influence), according to how interested they are
in the success of the specific programme, and how
Influence - interest map much influence they have in making it happen.
Powerful stakeholders can also have a strong
Public
negative effect on the success of the programme.
sector actor CSO Platform/ Public
During this exercise, it is possible that actors who
CSO network sector actor
had not yet been identified in the stakeholder map
Platform/
are included.
network
CSO Public
High interest

sector actor While doing both exercises, keep in mind that there
CSO
Private Private are both formal and informal power structures to
sector actor sector actor take into account. Government would be a formal
CSO power structure, while activist groups would be an
informal power structure.
Public
Private CSO
sector actor
sector actor These exercises produce the richest insights when
done in a participatory way in order to include
Private
different perspectives. The discussions generated
sector actor
Low interest

by the exercises help to bring out the different

CSO Private
perspectives at a time when these can be taken
sector actor
on board and to create co-ownership and a shared
focus for all involved.
Low influence High influence
3 | How to conduct a context analysis
13
Map and analysis the factors factors related to the desired impact: what is programme, and what the scope of the programme
Besides stakeholders, external factors also hampering the achievement of this impact? Why is should be.
need to be taken into account when designing a it not happening now? The next step is to cluster
programme. Are there any environmental, historic, the cards according to topics, and then organise By doing these exercises in a participatory manner,
political, cultural or socio-economic factors them in cause and effect relationships on a map. the relevant stakeholders should reach a common
that are likely to have an effect on the success In our experience, no more than 25 factors works understanding of the problem that the programme
of the programme, and in turn, on that which best to avoid spreading the focus too thin. Such is trying to address. What are the issues that lead to
the programme can have an effect? Identifying a map, or conceptual model, helps to create a the overall problem and how are they interrelated?
these factors will help to determine the problems common understanding of the problems, how they
and opportunities that need to be addressed. are interrelated, and what the root causes are. It Once you’ve conducted your context analysis, you
Documenting all factors at play will help to can also help to distill what activities should be put can use all of your knowledge and information to
justify decisions on the programme’s scope and into play to address the problems outlined by the design a Theory of Change.
focus. Besides data research (chapter seven),
brainstorming and interviewing key stakeholders is
helpful in identifying and documenting factors. This
can be done by creating a problem tree.
factor
Conceptual model: Which factors
hamper impact from happening?
Create a problem tree
A helpful method for mapping out factors is creating factor factor factor
a problem tree1 or a conceptual model. Start by
defining the impact that the programme aims to
factor factor factor
address, such as sustainable and inclusive water,
sanitation and hygiene (WASH) services. Write it on factor impact
a card and stick it to the wall. Using the knowledge factor factor factor
of the people in the room, brainstorm the various
factor

factor
1  More guidance on how to do a problem tree: https://www.odi.org/
publications/5258-problem-tree-analysis factor factor

factor

3 | How to conduct a context analysis

14
4 | How to identify your programme’s
steps toward impact
As the name suggests, a Theory of Change is a hypothesis of how we think
change occurs. It consists of a visual diagram and a narrative with causal
Glossary assumptions - “if we do X, Y happens because we believe Z.” ToCs are also
referred to as Intervention Logics or Results Chains. When designed with an
Impact: An impact is a change at the level of end-users, communities or
understanding of factors and stakeholders, a ToC helps to make sense of and
constituents. Impact is in our sphere of interest, but we can only contribute to it.
navigate the complex environment in which the programme is operating. A
ToC is a perception of reality which is shaped by the norms, values, experience
Outcome: An outcome is defined as a change in the behaviour, relationships,
and beliefs of the people who create it. It’s therefore important to involve
actions, activities, policies, or practices of an individual, group, community,
different stakeholders, to be sure different perspectives are captured in the ToC.
organisation, or institution. The formulation describes which specific local
Designing a ToC together will:
stakeholder is doing what differently. Outcomes are in our sphere of influence.
• Lead to a common understanding of how change happens
• Create awareness on different norms and values between stakeholders
Strategy: Strategies are a general description of what the programme needs to
• Generate co-ownership of the programme
do to make the expected outcomes happen.
• Help to decide on the scope of the programme
• Support decision-making on what interventions should be pursued to
4 | How to identify your programme’s steps toward impact
15
 achieve the biggest impact by whom Specify your strategies
• Expose gaps in your activities or show you where there is an overlap with the Once the stakeholders have identified one or multiple pathways of change,
activities of other actors or programmes. they can come up with strategies to set into motion the causal chain of events.
Strategies are a general description of what the programme needs to do to make
Below are the steps to defining a Theory of Change the expected outcomes happen. Every strategy will have a pathway of change.
Strategies are a general description of what needs to be done; the more specific
Define your desired impact activities in the strategies will be defined later. In reality, strategies may already
When designing a ToC, the first thing that needs to be identified is the desired be determined before the expected outcomes are mapped and an impact is
long-term impact the programme wants to achieve or contribute to. After defined. In that case, the ToC design exercise will help to identify the expected
identifying the intended impact, the participants can determine which outcomes outcomes and their causal relationships and understand how the strategies will
need to be achieved in order to reach that long-term impact. An example of an lead to the envisioned impact. For example, we support the ministry of water
impact statement is sustainable and inclusive water, sanitation and hygiene with data collection and analysis for water point mapping.
(WASH) services.
Connect your pathways of change
Determine your outcomes Once the strategies, expected outcomes, impact(s), and their linkages are
An outcome is a change in the behaviour, relationships, actions, activities, identified, the underlying causal assumptions should be made explicit. Trying to
policies, or practices of an individual, group, community, organisation, or document these assumptions can lead to the identification of weak spots in the
institution. In order to phrase an expected outcome, it helps to use the following Theory of Change and, at the same time, result in stakeholders becoming aware
mnemonic: “who should be doing what differently?” For example, the ministry of of each other’s visions of reality. When phrasing causal assumptions, it can be
water use the online water quality dashboard to make decisions on investment tempting to start a circular reasoning.
priorities.
Avoid phrases like “A leads to B, because B is the result of A”. Instead, try
Identify your pathways of change reasoning as follows: “If we do action X, we will contribute to outcome Y because
Once all expected outcomes have been identified on cards, they can be we believe that Z.” For example, if we support the ministry of water with data
organised on the wall in logical cause and effect relationships. When a collection and analysis for water point mapping, then they will use the map for
programme has several topics or issues, separate pathways of change can be decision making on investment priorities, because they were actively involved
built for each of them, with small teams who can zoom into each specific topic. in the identification of the problem (no updated information on functionality of
The resulting pathways of change can then be collectively connected at the end water points) and feel co-ownership of the solution (data collection for evidence-
to have the full Theory of Change. based decisions).

4 | How to identify your programme’s steps toward impact

16
1. Expected impact 2. Expected outcomes

3. Strategies 4. Cause - effect

relations

4
5. Causal 3 6. Theory of
1 2
assumptions 5 change diagram
1: If we do X, Y happens. See next page for larger
6
2: We expect that when model.
this outcome happens,
the next outcome
happens because Z.
3: Etc.

4 | How to identify your programme’s steps toward impact

17
Once you’ve designed your Theory of Change, you can build your tailor-made
planning, monitoring, evaluation and learning (PMEL) framework, which will
allow you to accurately monitor your results.

Theory of Change diagram

outcome outcome

strategy outcome outcome outcome outcome

outcome outcome

outcome outcome

outcome

outcome
outcome
impact
outcome outcome

strategy outcome outcome outcome outcome

outcome

outcome
strategy outcome outcome

outcome

4 | How to identify your programme’s steps toward impact

18
 Case study

Creating Theories of Change with the Watershed programme

Challenge Partnership Change
Watershed is comprised of multiple partners Creating a ToC is always a collaborative process, A year into the programme, monitoring data
in different locations, with different resources, which Akvo facilitated with multiple workshops in showed the Watershed teams what progress had
interests, capacities and responsibilities. several of the Watershed countries. been made, but also helped to reflect on the Theory
of Change with new understanding. By looking
Contributing effectively to the desired impact for Once the programme-level ToC was created, Akvo forward, looking back, zooming out, and zooming
Watershed, to achieve sustainable WASH for all in facilitated the development of country-level ToCs, in, Akvo helped to create clarity for everyone
each focus country, is deeply complex. taking into account the specific local context of each involved, ensuring that all actions are relevant and
country. strategic, all the while empowering the best people
to do the job.

4 | How to identify your programme’s steps toward impact

19
 5 | How to design a results monitoring framework

A well-formulated results framework, also known framework. All of the steps between the strategy to measure the progress towards this outcome. For
as a planning, monitoring, evaluation and learning and the impact are expected outcomes that in example:
(PMEL) framework, is designed to let you know theory can be monitored, but in practice should
whether or not your programme is succeeding. For not be. Only a small selection of the outcomes Outcome: The local government effectively
programme managers, a good results framework is should be selected and monitored. For example, monitors WASH infrastructure in district X.
key to gaining a comprehensive overview of how the out of 25 expected outcomes in the ToC, monitor Indicator: Number of water points monitored on
programme is performing and what decisions need four or five. Together, the stakeholders can define functionality and water quality in district X.
to be made to improve results. Besides improving which outcomes are the most important, as well
your programme’s results, a good results framework as practical, to monitor across the programme. Outcome: The national government allocates more
will allow you to monitor the causal assumptions Use your ToC as your guide here, choosing the budget to WASH.
laid out in your Theory of Change that you’re unsure outcomes that are both measurable and contribute Indicator: Amount of budget that is allocated to the
about. Based on your monitoring findings, you significantly to the desired impact. Your resources ministry of water on a yearly basis.
can adjust your ToC accordingly. In this chapter, will also play a role, as some outcomes will be less
we introduce the key steps in designing a results cost-effective to monitor than others. Together, these indicators constitute the
monitoring framework that will save resources, programme’s monitoring framework. For each
align distributed teams, and improve programmes. Translate your outcomes into indicators indicator, the method used to measure it needs to
Once you’ve decided which outcomes to monitor, be determined, as well as the baseline values and
Monitor your outcomes you can translate your outcomes into indicators target values.
A strong, well formulated Theory of Change is that can be measured. Each expected outcome can
the foundation of a relevant and useful PMEL have one or more indicators that make it possible

5 | How to design a results monitoring framework

20
Establish the baseline values Choose expected outcomes to monitor
You should be able to measure your indicator accurately using either qualitative
or quantitative data. Before you set your target value, you need to establish
baseline values in order to monitor change. Primary or secondary data sources
can be used to establish your baseline values. Baseline data may already be A
available, or you may have to collect it yourself.

E
Define your target values
The next step is to set the target value you expect to achieve with specific B
timeframe given the scope and resources of your project. Targets help to
determine whether progress is being made compared to projected expectations. C F

If actual data diverges from a set target notably, adjustments should be triggered
in programme implementation and/or design and expectations. D

Determine your data collection methods

Data collection is dependent on the kind of indicators and key questions
established. There are a variety of tools and methods for data collection and
they may vary by indicator type. For example, for quantitative indicators you Monitoring framework
might use questionnaires, activity logs, or before/after surveys. For qualitative
indicators, you could use interviews, field observation visits, stakeholder Outcomes Indicators Description Method Frequency of Responsibility Costs
meetings, or group discussions. You should also determine here at what interval to monitor of indicator measurement
each indicator should be measured.
A
B
Identify the data sources C
Identify those responsible for collecting or providing the data. This can be
D
project staff, implementing partners, independent evaluation teams, end users,
E
sensors, or secondary sources. Remember that there may be a lot of useful
F
datasets already available. Once these steps have been covered, you can put it
into a results monitoring framework.

5 | How to design a results monitoring framework

21
6 | How to choose the right tool for effective results monitoring

Choosing the right tool to set up your results monitoring framework depends correct values for Burkina Faso after I corrected the last version?
on many factors, from your programme’s resources to the geographic spread of
your team. At present, two of the most common tools used by M&E managers Collecting results data in timely, organised way with a system that ensures
are Excel, and more recently Google Sheets, both of which are effectively data quality is not easy. A single online platform, accessible anywhere with
free. However, the challenges in using those tools after the results monitoring connectivity, can mitigate this challenge; it eliminates the search for scattered
framework has been established and the programme has commenced can incur documents and data while providing transparency on who has input the
hidden costs. In this chapter, we’ve outlined some of the deciding factors in encoded data.
choosing your results monitoring tool.
Consider staff turnover
Align your distributed teams Another common occurrence when working on large multi-country, multi-
Programme managers working with distributed teams spread across several year programmes is that staff often do not stay for the entire duration of the
implementing countries know that once a results framework is finalised and programme. For a programme manager this can present challenges, especially
project activities have started, working off a single Excel document becomes when it comes to the quality of results data. With changing programme staff,
frustrating. Not long after implementation, the master results framework can it can become difficult to know exactly who submitted what results data and
become riddled with incorrect results data. Which document version was the when. This uncertainty can diminish data integrity and ultimately leads to
monitoring and evaluation officer in Beirut working in? Am I looking at the additional costs in the form of more staff time.

6 | How to choose the right tool for effective results monitoring

22
 Online tools that tag each indicator results update with useful information can
solve this problem. This way, you can easily find out who entered the results,
who approved the results, and when. The result is an auditable data trail,
providing programme managers with a comprehensive oversight of results
throughout the programme.

Discover your reporting needs

Making sense of data collected in multiple programmes in different countries
and turning it into unified, digestible and insightful reports, often under time
pressure, can be a real challenge. Working in an intuitive monitoring and
results platform that allows you to create high quality, appealing reports can
significantly reduce the admin burden for programme managers. An online
environment can also make sure results are easier to share externally. Finding a
tool that allows you to report to International Aid Transparency Initiative (IATI)
standards means that you don’t have to carry out duplicate reporting to the IATI
Standard in a separate process. It also allows your data to be easily compared
with data published by other organisations reporting their development
cooperation activities to IATI.

The monitoring framework of a programme shows what data needs to be

collected to describe or score the indicators which are used to monitor
progress, as well as which methods will be used. Finding the right tool for your
programme’s needs and data uses is an essential step in the design process. This
is how Theory of Change based programme design links to the chapter seven,
data research, where we assess whether any of the necessary data is already
available from secondary sources, and what data still needs to be gathered by
the programme.

6 | How to choose the right tool for effective results monitoring

23
 7 | How to conduct data research for
your programme in four steps

Data research is a method that helps you to while other data may still need to be captured. You Evaluate existing data
systematically assess existing data and data can start off by making an inventory of existing
sources, allowing you to identify where there are data. Once you’ve created an inventory of existing data
gaps in the data you need and where you can add sources, it is important to evaluate the existing
value with your programme. At the same time, data First of all, look into the data resources of your own data on its accessibility, granularity, credibility and
research gives you the tools to think about your organisation, including what is gathered in reports relevance. The following questions can help you
stakeholders and audience. In this chapter, you’ll and stored in databases. Consider both quantitative understand whether the existing data is available
find an overview of four consecutive steps that will data, expressing a certain quantity, amount for usage, detailed enough and has the right
help you in conducting data research: or range, and qualitative data, which is more scale, and reliable enough for you to use in your
• Make an inventory of existing data/evidence descriptive, resulting from small scale surveys, focus programme:
• Evaluate existing data group discussions, observations and interviews. You
• Perform a gap analysis can then think about what data may be available • Is the data openly available, or does it require
• Understand who will use your data and easily accessible outside of your organisation. special permission to access? (Accessibility)
Are there any data sharing platforms or other • Is the data structured in a way that is useful for
Make an inventory of existing data/ organisations that deal with the same problem or your programme? (Relevance)
evidence try to answer the same question? What data do • How often is the data collected? (Granularity)
Once you’ve identified what data needs you have they have on this problem? Is it open access? Even if • How granular or detailed is the data
within your programme, you will need to start data is not openly accessible, it might be possible to geographically? (Granularity)
gathering it. Some data may be readily available, persuade this organisation to share its data. • How granular or detailed is the data

7 | How to conduct data research for your programme in four steps

24
 demographically? (Granularity) Once you have identified all the data gaps, take problem empowers them to take action. However,
• When was the data collected? How long has it a critical look at the data you’ve identified as this involves thinking about how to share the data
been retained? (Relevance and Granularity) necessary. Do you really need to collect all that in an understandable and accessible way. In remote
• Do the current problem solvers use it for data? And what are you going to use all the different communities, accessing the data online may prove
decision making, evaluation, or something else? elements for? Although it’s tempting to collect data to be difficult, and radio stations or distribution
(Credibility) that you may think will be useful in future, a general of offline materials may be a better mode of
• Who collected the data? What was the purpose rule of thumb is that less data is more. It’s better to dissemination. You might want to consider making a
of their data collection? Has the data been focus on the things that really matter and minimise data dissemination plan, in which you identify your
cleaned and/or analysed? And if so, in what complexity. It’s less expensive, less time consuming, stakeholders and their respective communication
way? (Credibility) and you don’t run the risk of collecting the wrong channels.
data.
Perform a gap analysis Data research is an approach that will help you
Now that you’ve identified the data sources that are Understand who will use your data to create focus in your programme. Thinking in
available to you and what data you can use for your If you are collecting data to contribute to solving this structured way about data gathering will
programme, you need to think about what data you a problem, or to underline the importance of avoid collection of duplicate data and encourage
still need to collect to answer your questions. To do addressing a certain problem, keep in mind that it everyone involved to determine the quality and
so, it helps to ask the following: what data do I need is crucial to involve all relevant stakeholders from usefulness of available data. This method also
to answer my questions or describe my indicators? the start of the data research process. This will allows you to assess whether the data you are
It’s important that, in the first instance, you don’t create ownership of the data, ensure relevance and collecting is truly relevant to your programme, and
think about restrictions that might be apparent in usefulness of the data, result in communities feeling the different stakeholders involved, and forces you
collecting this data. Only after identifying the data represented by the data, and avoid decision makers to think about how to disseminate the data to them
you need should you start considering potential turning a blind eye or questioning the credibility of before the data collection has actually started.
restrictions, such as time, resources and feasibility. the data. Start your data collection exercise with an
It might turn out that data you initially deemed inventory of what the different stakeholders want to
infeasible to collect isn’t as difficult to gather after know and how you are going to reach them. Sharing
all. the data with the people directly involved in the

7 | How to conduct data research for your programme in four steps

25
 Conclusion

The Theory of Change based approach to data-driven programme design is an

effective way of working collaboratively towards a common understanding of
what the programme should achieve and what activities the programme should
focus on to contribute to impact. It is important to note that a ToC by nature is
subject to constant change. During the implementation of a programme, the
context may change, resulting in the need for an adaptation of the ToC. During
implementation, you may realise that outcomes are missing, causal assumptions
need to be adapted, or interrelations changed. Theories of Change therefore
need to be reviewed at least once per year, based on monitoring findings and a
context analysis update. At the same time, a ToC revision workshop can form an
excellent basis for writing an annual activity plan, keeping the ToC at the heart of
your programme at all times.

By conducting a thorough design phase, you’ll achieve clarity and oversight

on every aspect of your development programme and align distributed teams
from the start. This will allow you to make decisions with confidence based
on accurate and real time data, reduce your admin burden, and focus on
showcasing your results, securing funding and driving impact.

Conclusion
26
 About Akvo Credits

We believe in equal access to public services, reliable infrastructure Project manager

and a safer environment for everyone. We are convinced that this will Laura Tufis
happen faster if governments and non-governmental organisations
become more effective, accountable and collaborative. Authors
Anita van der Laan, Anthony Gonzalez, Ethel Mendez, Marten Schoonman and
Since 2008, we’ve worked with over 20 governments and 200 Veerle van Loevezijn
organisations in more than 70 countries to improve the way they
implement development projects and make decisions using data. We Editor
call them partners. Georgia Walker

With our combination of tools, services, local expertise and sector Graphic designer
knowledge, our partners improve the management of water, Linda Leunissen
sanitation and agriculture, with a strong commitment to accelerating
the progress of the sustainable development goals. Photographers
P04 SmartSeeds by Stefan Kraus (RGB Collective). P05 (from left) Stefan Kraus,
With our unique approach to development, we help our partners Tiipaalga, Greencoffee by Lissy van Noort, Oxfam. P07 WWF by Stefan Kraus
design their projects so that they can capture and understand (RGB Collective). P9 Kopernik by Ima Puspita Sari. P10 WWF by Stefan Kraus.
reliable data which they can act upon. P12 Oxfam. P16 Watershed. P20 Watershed. P24 CARDS by Joseph Thomas. P27
Sustainable Harvest International.
Visit us at www.akvo.org to learn more.

The contents of this eBook are partly based on the contents of the
AfriAlliance Handbook (see www.afrialliance.org). The AfriAlliance
project has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement
No 689162.

27
Looking to unlock the power
of data for sustainable change?
Embark on a data journey
#withAkvo and amplify the impact
of your development work

Get in touch now

Credits
28