Rainbow Framework

There are so many different options (methods, strategies and processes) in evaluation that
it can be hard to work out which ones to choose for an evaluation.
BetterEvaluation organises options into 34 different evaluation tasks, grouped by 7 colourcoded clusters to make it easier for you to choose and use appropriate methods, strategies
or processes. It also shows approaches (which combine a package of options) such as
Randomized Controlled Trials (RCTs) and Outcome Mapping (OM).
The planning tool can be used to: commission and manage an evaluation; plan an
evaluation; check the quality of an ongoing evaluation; embed participation thoughtfully in
evaluation; develop evaluation capacity.
Send suggestions for additions or revisions to us via http://betterevaluation.org

BetterEvaluation is an international collaboration to improve evaluation theory and practice by sharing information
about evaluation options (methods, strategies, processes) and approaches (collections of methods). We provide an
interactive and freely accessibly website and related events and resources. Visit BetterEvaluation at
http://betterevaluation.org and register to contribute material, add comments and ask questions. We support
individual evaluators, managers of evaluation and practitioners as well as organisations across disciplinary and
organisational boundaries, sectors, languages and countries.
Founding partners: Institutional Learning and Change (ILAC) initiative of the Consultative Group on International
Agriculture (CGIAR), Overseas Development Institute (ODI), Pact, RMIT University (Royal Melbourne Institute of
Technology).
Financial support: Australian Government Department of Foreign Affairs and Trade (DFAT), International Fund
for Agricultural Development (IFAD), The Rockefeller Foundation, Netherlands Ministry of Foreign Affairs,
International Development Research Centre (IDRC).

You may use this document under the terms of the Creative Commons Attribution-Non Commercial Unported licence available
at http://creativecommons.org/licenses/by-nc/3.0/.

1. MANAGE an evaluation or evaluation system

Manage an evaluation (or a series of evaluations), including deciding who will conduct the evaluation and who
will make decisions about it.

Understand and engage stakeholders

Who needs to be involved in the evaluation? How can they be identified and engaged?
Understand stakeholders:
1. Community scoping
2. Stakeholder mapping and analysis

Engage stakeholders:
3. Community fairs
4. Fishbowl technique
5. Formal meeting processes
6. Informal meeting processes

Establish decision making processes

Who will have the authority to make what type of decisions about the evaluation?
Who will provide advice or make recommendations about the evaluation?
What processes will be used for making decisions?
Types of structures:
1. Advisory group
2. Citizen juries
3. Steering group

Ways of making decisions:

8. Consensus decision making
9. Hierarchical decision making
10. Majority decision making

Ways of exploring issues:

4. Formal meeting processes
5. Informal meeting processes
6. Round robin
7. Six Hats Thinking for exploring decision making
Approaches:
Participatory evaluation

Decide who will conduct the evaluation

Who will actually undertake the evaluation?
1.
2.
3.
4.

Community
Expert review
External consultant
Hybrid - internal and external

5. Internal staff
6. Learning alliances
7. Peer review

Approaches:
Horizontal evaluation
Positive deviance
Participatory evaluation

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 2

Determine and secure resources

What resources (time, money, and expertise) will be needed for the evaluation and how can they be obtained?
Consider both internal (e.g. staff time) and external (e.g. previous participants time) resources
Determine resources needed
1. Evaluation budget matrix
2. Evaluation costing
3. Resources stocktake

Secure resources needed

4. Designated staff time
5. Grant funding
6. Institutionalised budget allocation
7. Leveraging partnerships
8. Strategies to reduce costs

Define ethical and quality evaluation standards

What will be considered a high quality and ethical evaluation?

How should ethical issues be addressed?
1. Cultural competency
2. Ethical guidelines

3. Evaluation Standards
4. Institutional Review Board (IRB)

Document management processes and agreements

How will the evaluations management processes and agreements be documented?

Document what is needed in an evaluation:
1. Expression of Interest (EoI)
2. Request For Proposal (RFP)
3. Scope of Work (SoW)
4. Terms Of Reference (ToR)

Document how different organisations will work

together:
5. Contractual agreement
6. Memorandum of Understanding (MoU)

Develop planning documents for the evaluation

What needs to be done to design, plan and document the evaluation?

What planning documents need to be created?
1.
2.
3.
4.

Aide memoire
Evaluation framework
Evaluation plan
Evaluation work plan

5. Gantt chart
6. Inception report

Review evaluation (do meta-evaluation)

How will the evaluation itself be evaluated including the plan, the process and report?
1. Beneficiary exchange
2. Expert review for meta-evaluation
3. Group critical reflection

4. Individual critical reflection

5. Peer review for meta-evaluation

Develop evaluation capacity

How can the ability of individuals, groups and organisations to conduct and use evaluations be strengthened?
1.
2.
3.
4.
5.
6.
7.
8.

Community of practice
Conferences
Coaching
Evaluation competencies
Evaluation library
Evaluation policy
Evaluation societies and associations
Learning circle

9.
10.
11.
12.
13.
14.
15.

Mentoring
Organisational policies and procedures
Peer coaching
Peer review for meta-evaluation
Reflective practice
Supervised practice in teams
Training and formal education

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 3

2. DEFINE what is to be evaluated

Develop a description (or access an existing version) of what is to be evaluated and how it is understood to work.

Develop initial description

What exactly is being evaluated?

1. Existing project description

2. Peak experience description
3. Thumbnail description
Approaches
Appreciative inquiry

Develop programme theory / logic model

How is the intervention understood to work (programme theory, theory of change, logic model)?
Ways of developing logic models:
1. Articulating mental models
2. Backcasting
3. Five whys
4. Group model building
5. Previous research and evaluation
6. SWOT analysis

Ways of representing logic models:

7. Tiny Tools Results Chain
8. Logframe
9. Outcomes hierarchy
10. Realist matrix
11. Results chain

Approaches
Collaborative outcomes reporting
Outcome mapping
Participatory impact pathways approach
Realist evaluation

Identify potential unintended results

What are possible unintended results (both positive and negative) that will be important to address in the
evaluation?
1. Key informant interviews
2. Negative programme theory
3. Risk assessment

4. Six Hats Thinking about unintended results

5. Unusual events reporting

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 4

3. FRAME the boundaries of an evaluation

Set the parameters of the evaluation its purposes, key evaluation questions and the criteria and standards to be
used.

Identify primary intended users

Who are the primary intended users of this evaluation?

(This task has resources only)

Decide purpose

What are the primary purposes and intended uses of the evaluation?
Using findings:
1. Contribute to broader evidence base
2. Inform decision making aimed at improvement
(formative)
3. Inform decision making aimed at selection,
continuation or termination (summative)
4. Lobby and advocate

Using process:
5. Build trust and legitimacy across stakeholders
6. Ensure accountability
7. Ensure diverse perspectives are included,
especially those with little voice

Specify the key evaluation questions

What are the high level questions the evaluation will seek to answer? How can these be developed?
(This task has resources only)

Determine what success looks like

What should be the criteria and standards for judging performance?

Whose criteria and standards matter? What process should be used to develop agreement about these?
Formal statements of values:
1. Sustainable development goals
2. OECD-DAC Criteria
3. Millennium Development Goals (MDGs)
4. Standards, evaluative criteria and benchmarks
5. Stated goals and objectives

Negotiate between different values:

13. Concept mapping
14. Delphi study
15. Dotmocracy
16. Open space technology
17. Public consultations

Articulate and document tacit values:

6. Hierarchical card sorting
7. Open space technology
8. Photovoice
9. Rich pictures
10. Stories of change
11. Values clarification interviews
12. Values clarification public opinion questionnaires

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 5

Approaches
Critical system heuristics
Participatory evaluation

4. DESCRIBE activities, outcomes, impacts and context

Collect and retrieve data to answer descriptive questions about the activities of the project/programme/policy,
the various results it has had, and the context in which it has been implemented.

Sample

What sampling strategies will you use for collecting data?

Probability:
1. Multi-stage
2. Sequential
3. Simple random
4. Stratified random
Convenience:
5. Convenience
6. Volunteer

Purposive (or Purposeful):

7. Confirming and disconfirming
8. Criterion
9. Critical case
10. Homogenous
11. Intensity
12. Maximum variation
13. Outlier
14. Snowball
15. Theory-based
16. Typical case

Use measures, indicators or metrics

What measures or indicators will be used?

Are there existing ones that should be used or will you need to develop new measures and indicators?
(This task has resources only)

Collect and/ or retrieve data

How will you collect and/or retrieve data about activities, results, context and other factors?
Information from individuals:
1. Deliberative opinion polls
2. Diaries
3. Goal attainment scales
4. Interviews with individuals:
- Convergent
- In-depth
- Key informant
5. Hierarchical card sorting
6. Keypad technology
7. Questionnaires (or surveys):
- Email
- Face-to-face
- Internet
- Mail
- Mobile phone (see Mobile Data Collection)
- Telephone
8. Mobile data collection
9. Photolanguage

Information from groups:

17. After action review
18. Brainstorming
19. Card visualization
20. Concept mapping
21. Delphi study
22. Dotmocracy
23. Fishbowl technique
24. Interviews with groups
- Focus groups discussion
25. Future search conference
26. Mural
27. ORID (Objective, Reflective, Interpretive,
Decisional)
28. Q-methodology
29. SWOT analysis (Strengths, Weaknesses,
Opportunities, Threats)
30. World cafe
31. Writeshop

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 6

10.
11.
12.
13.
14.
15.
16.

Photovoice
Polling Booth
Postcards
Projective techniques
Seasonal calendars
Sketch mapping
Stories

Physical:
37. Biophysical
38. Geographical

Observation:
32. Field trips
33. Non-participant observation
34. Participant observation
35. Photography/video recording
36. Transect

Existing documents and data:

39. Big data
40. Logs and diaries
41. Official statistics
42. Previous evaluations and research
43. Project records
44. Reputational monitoring dashboard

Manage Data

How will you organise and store data and ensure its quality?
1. Consistent data collection and recording
2. Data backup
3. Data cleaning

4. Effective data transfer

5. Secure data storage
6. Archive data for future use

Combine qualitative and quantitative data

How will you combine qualitative and quantitative data?

When data are gathered:
1. Parallel data gathering
2. Sequential data gathering
When data are combined:
3. Component design
4. Integrated design

Purpose of combining data:

5. Enriching
6. Examining
7. Explaining
8. Triangulation (confirming; rejecting)

Analyse data

How will you investigate patterns in numeric or textual data?

Numeric analysis:
1. Correlation
2. Cross-tabulations
3. Data mining
4. Exploratory techniques
5. Frequency tables
6. Measures of central tendency
7. Measures of dispersion

8.
9.
10.
11.
12.

Multivariate descriptive
Non-parametric inferential
Parametric inferential
Summary statistics
Time series analysis

Textual analysis
13. Content analysis
14. Framework matrices
15. Thematic coding
16. Timeline and time-ordered matrices

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 7

Visualise data

How will you display data visually?

See relationships among data points:
1. Scatterplot
2. Matrix chart
3. Network diagram

See the parts of a whole:

9. Icon array
10. Pie chart
11. Treemap

Compare a set of values:

4. Bar chart
5. Block histogram
6. Bubble chart

Analyse text:
12. Phrase net
13. Word cloud
14. Word tree

Track rises and falls over time:

7. Line graph
8. Stacked graph

See the world:

15. Demographic mapping
16. Geotagging
17. GIS mapping
18. Interactive mapping
19. Social mapping

5. UNDERSTAND CAUSES of outcomes and impacts

Collect and analyse data to answer causal questions about what has produced outcomes and impacts that have
been observed.

Check the results support causal attribution

How will you assess whether the results are consistent with the theory that the intervention produced them?
Gathering additional data:
1. Key informants attribution
2. Modus operandi
3. Process tracing

Approaches:
Contribution analysis
Collaborative outcomes reporting

Analysis:
4. Check dose-response patterns
5. Check intermediate outcomes
6. Check results match a statistical model
7. Check results match expert predictions
8. Check timing of outcomes
9. Comparative case studies
10.Qualitative comparative analysis
11.Realist analysis of testable hypotheses

Multiple lines and levels of evidence

Rapid outcomes assessment

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 8

Compare results to the counterfactual

How will you compare the factual with the counterfactual what would have happened without the
intervention?
Experimental:
1. Control group
Quasi-experimental:
2. Difference-in-difference
3. Instrumental variables
4. Judgemental matching
5. Matched comparisons
6. Propensity scores

7.
8.
9.

Regression discontinuity
Sequential allocation
Statistically created counterfactual

Non-experimental:
10. Key informant
11. Logically constructed counterfactual

Approaches:
Randomised Controlled Trials

Investigate possible alternative explanations

How will you investigate alternative explanations?
1.
2.
3.
4.

Key informant
Force field analysis
General elimination methodology
Process tracing

Approaches:
Contribution analysis
Collaborative outcomes reporting

5. Rapid outcomes assessment

6. Ruling out technical explanations
7. Searching for disconfirming evidence/Following
up exceptions
8. Statistically control for extraneous variables

Multiple lines and levels of evidence

Rapid outcomes assessment

6. SYNTHESISE data from one or more evaluations

Combine data to form an overall assessment of the merit or worth of the intervention, or to summarise evidence
across several evaluations.

Synthesise data from a single evaluation

How will you synthesise data from a single evaluation?

Processes:
1. Consensus conference
2. Expert panel

Techniques:
3. Cost benefit analysis
4. Cost effectiveness analysis
5. Cost utility analysis
6. Lessons learnt
7. Multi-criteria analysis
8. Numeric weighting
9. Qualitative weight and sum
10. Rubrics
11. Value for money

Approaches:
Social return on investment

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 9

Synthesise data across evaluations

Do you need to synthesise data across evaluations? If so, how should this be done?
1. Best evidence synthesis
6. Realist synthesis
2. Lessons learnt
7. Systematic review
3. Meta-analysis
8. Textual narrative synthesis
4. Meta-ethnography
9. Vote counting
5. Rapid evidence assessment

Generalise findings

How can the findings from this evaluation be generalised to the future, to other sites and to other programmes?
1. Analytic generalisation
2. Statistical generalisation
Approaches:
Positive deviance
Horizontal evaluation

7. REPORT AND SUPPORT USE of findings

Develop and present findings in ways that are useful for the intended users of the evaluation, and support them
to make use of them.

Identify reporting requirements

What timeframe and format is required for reporting?

1. Communication plan
2. Reporting needs analysis

Develop reporting media

What types of reporting formats will be appropriate for the intended users?
Written:
1. Aide memoire
2. Executive summaries
3. Final reports
4. Interim reports
5. Memos and Email
6. News media communications
7. Newsletters, bulletins, briefs and brochures
8. Postcards
9. Website communications
Presentation events:
10. Conference
11. Feedback workshops
12. Teleconference
13. Verbal briefings
14. Videoconference
15. Web-conference

Presentation materials:
16. Flip charts
17. Displays and exhibits
18. Posters
19. Power-point
20. Video
Creative:
21. Cartoons
22. Photographic reporting
23. Poetry
24. Reporting in pictures
25. Theatre
Graphic Design:
25. Arrangement
26. Color
27. Images
28. Type

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 10

Ensure accessibility

How can the report be easy to access and use for different users?
General accessibility:
1. Applied graphic design principles
2. Descriptive chart titles
3. Eliminate chartjunk
4. Emphasis techniques
5. Headings as summary statements
6. One-Three-Twenty-Five (1:3:25) principle
7. Plain language

Specific accessibility barriers:

8. Colour blind audience
9. Low vision and blind audience

Develop recommendations

Will the evaluation include recommendations?

How will these be developed and by whom?
1.
2.
3.
4.

Beneficiary exchange
Chat rooms
Electronic democracy
External review

5.
6.
7.
8.

Group critical reflection

Individual critical reflection
Participatory recommendation screening
World cafe

Support use

In addition to engaging intended users in the evaluation process, how will you support the use of evaluation
findings?
1. Annual reviews
2. Conference co-presentations
3. Data use calendar

4. Policy briefings
5. Recommendations tracking
6. Social learning

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 11

Approaches
Appreciative Inquiry
A participatory approach that focuses on existing strengths
rather than deficiencies - evaluation users identify instances
of good practice and ways of increasing their frequency.

Most Significant Change

Collects and analyses personal accounts of change, includes
processes for learning about what changes are most valued
by individuals and groups.

Beneficiary Assessment
An approach that assesses the value of an intervention as
perceived by the (intended) beneficiaries, thereby aiming to
give voice to their priorities and concerns.

Outcome Mapping
Unpacks an initiatives theory of change, provides a
framework to collect data on immediate, basic changes that
lead to longer, more transformative change, and allows for
the plausible assessment of the initiatives contribution to
results via boundary partners.

Case study
A research design that focuses on understanding a unit
(person, site or project) in its context, which can use a
combination of qualitative and quantitative data.
Collaborative Outcomes Reporting
An approach that builds on contribution analysis, adding
expert review and community review of the assembled
evidence and conclusions.
Contribution Analysis
An approach for assessing the evidence for claims that an
intervention has contributed to observed outcomes and
impacts.
Critical System Heuristics
An approach used to surface, elaborate, and critically
consider boundary judgments, that is, the ways in which
people/groups decide what is relevant to the system of
interest (any situation of concern).
Developmental Evaluation
An approach appropriate for evaluations of adaptive and
emergent interventions, such as social change initiatives or
projects operating in complex and uncertain environments.
Horizontal Evaluation
An approach that combines self-assessment by local
participants and external review by peers.
Innovation History
A way to jointly develop an agreed narrative of how an
innovation was developed, including key contributors and
processes, to inform future innovation efforts.
Institutional Histories
An approach for creating a narrative that records key points
about how institutional arrangements have evolved over
time and have created and contributed to more effective
ways to achieve project or programme goals.

Participatory Evaluation
A range of approaches that engage stakeholders (especially
intended beneficiaries) in planning, conducting, analysing the
evaluation and/or making decisions about the evaluation.
Participatory Impact Pathways Analysis
Participatory Learning for Action
Formerly known as Participatory Rural appraisal. Enables
farmers to analyse their own situation and develop a
common perspective on natural resource management and
agriculture at village level.
Positive Deviance
Involves intended evaluation users in identifying outliers
those with exceptionally good outcomes and
understanding how they have achieved these.
Randomised Controlled Trials
An approach that produces an estimate of the mean net
impact of an intervention by comparing results between a
randomly assigned control group and experimental group or
groups.
Realist Evaluation
A form of theory-driven evaluation that seeks to understand
what works for whom, where and why taking into account
how context makes a difference to programme results.
Social Return on Investment
Identifies a broad range of social outcomes, not only the
direct outcomes for the intended beneficiaries of an
intervention.
Utilisation-Focused Evaluation
Uses the intended uses of the evaluation by its primary
intended users to guide decisions about how an evaluation
should be conducted.

BetterEvaluation Rainbow Framework - August 2014 - www.betterevaluation.org

Page 12