Practical Monitoring and Evaluation: A Guide For Voluntary Organisations

CSE COVER new:CSE cover 2005 07/09/2009 16:51 Page 1
Practical monitoring and evaluation
4 Coldbath Square
London EC1R 5HL
+44 (0) 20 7713 5722
+44 (0) 20 7713 5692
[email protected]
www.ces-vol.org.uk
Charities Evaluation Services
helping you do better what you do best
Practical
monitoring
and
evaluation
a guide for voluntary
organisations
helping voluntary
organisations to
be more effective
Since 1990, Charities Evaluation Services
(CES) has worked with a wide variety of
voluntary organisations and their funders.
Our aim is to promote accessible monitoring
and evaluation practice, which organisations
can carry out within the resources available
to them. CES provides training, advice
and technical help, and also carries out
independent evaluations as part of its
commitment to strengthening and improving
the effectiveness of the voluntary sector.
CES produces a range of publications
including PQASSO, the practical quality
assurance system for small organisations.

4 Coldbath Square
London EC1R 5HL
t +44 (0) 20 7713 5722
f +44 (0) 20 7713 5692
e [email protected]
w www.ces-vol.org.uk
Company limited by guarantee
Registered office 4 Coldbath Square London EC1R 5HL
Registered in England and Wales number 2510318
Registered charity number 803602
Copyright
Unless otherwise indicated no part of this publication
may be stored in a retrievable system or reproduced in
any form whatsoever without prior written permission
from Charities Evaluation Services. Charities Evaluation
Services will give sympathetic consideration to requests
from small organisations for permission to reproduce
this publication in whole or in part but terms upon
which such reproduction may be permitted will remain
at Charities Evaluation Services discretion.
Charities Evaluation Services, 2002
Third edition, 2008
ISBN 978-0-9558849-00
Published by Charities Evaluation Services
Designed by Alexander Boxill
New edition artwork by Artloud Ltd
Edited by Wordworks, London W4 4DB
Printed by Lithosphere Print Production
pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 1
acknowledgements
charities
evaluation
services
Developed and written

by Dr Jean Ellis for Charities
Evaluation Services.
CES trustees, staff and associates

For reading drafts and providing
material for the first edition text
Rowan Astbury
Libby Cooper
Sally Cupitt
For their contributions
Jean Barclay
Mark Bitel
Sara Burns
Vijay Kumari
John Morris
Jayne Weaver
Readers
For reading and commenting on the first edition text
Jenny Field, Bridge House Estates Trust Fund
Ciarn McKinney, Streetwise Youth
Tom Owen, Help the Aged
Georgie Parry-Crooke, University of North London
Professor Helen Simons, University of Southampton
Chris Spragg, NCH Action for Children
James Wragg, Esme Fairbairn Foundation
Funders
We are very grateful to the Calouste Gulbenkian
Foundation and the Wates Foundation for the
grants they provided for the first edition of this
publication.
charities
evaluation
services
foreword
A central part of the mission of NCVO is to help
voluntary organisations to achieve the highest
standards of practice and effectiveness. We know
from our work with organisations across the
country that the vast majority are firmly committed
to improving their work. However, while the
motivation is undoubtedly there, many
organisations lack the experience and practical
know-how to assess the effectiveness of their work
without help or guidance.
Charities Evaluation Services has an impressive track
record in developing clear and accessible material
for voluntary organisations. This stems from their
long experience of working closely with
organisations of all shapes and sizes.
CES Practical Monitoring and Evaluation was
first published in 2002, providing much-needed
guidance on evaluation for voluntary organisations
both large and small. The guide is now in its third
edition, with a revised further reading list, and text
additions which acknowledge the recent shifts in
the policy and funding context and developments
in monitoring and evaluation software and other
resources.
This guide will be invaluable to any voluntary
organisation that is serious about measuring the
effectiveness of its work. It has been carefully
designed to be of relevance to organisations
relatively new to this sort of approach, as well as
providing more demanding material for those who
need something more sophisticated.
I would encourage any voluntary organisation that
is committed to improving its work to use this
guide. The sector is under increasing scrutiny from
the public, the government and funders
particularly in our role in delivering public services.
There has never been a more important time for
the sector to demonstrate its effectiveness. I am
sure that this guide will continue to play an
important part in helping us do this.
Stuart Etherington
Chief Executive, National Council for Voluntary
Organisations
Now, more than ever, organisations which

stand still run the risk of stagnating. One of the
Commissions hallmarks of an effective charity is
that it considers how to identify, measure and
learn from its achievement and outcomes - not
least the positive (and negative) effects it has on
its beneficiaries.
Dynamic, effective organisations put learning and
developing at the heart of their development
strategy, and advice and guidance on monitoring
is invaluable in helping them do so. So Im
delighted to welcome the third edition of CES
Practical monitoring and evaluation.
Charities are increasingly aware of the importance
of monitoring and evaluating the effectiveness
their work being able to demonstrate how they
do this sends a strong and positive message of
accountability to beneficiaries, funders and
supporters alike.
The fundamental principles in this guide are
equally relevant to those setting up a charity as to
those already up and running. It provides back-tobasics advice for organisations with no, or limited,
experience of monitoring and evaluation as well as
more advanced guidance for those organisations
which are keen to develop a more nuanced
understanding.
Monitoring and evaluation are an integral part of a
well-run and evolving charity and should be seen
as vital tools in helping an organisation adapt and
develop their services to meet need.
I hope the third edition of Practical monitoring
and evaluation will cement the importance of
good governance. It is more fundamental to the
sector today than ever before.
Dame Suzi Leather

Chair, Charity Commission
this guide
charities
evaluation
services
Practical monitoring and evaluation has

an introductory and four main sections:
Planning
Monitoring
Evaluation
Utilisation.
The final section is a Practical toolkit.
The four main sections offer practical
guidance on the four key areas of the
monitoring and evaluation cycle:
feedback for internal learning and planning,

or whose funders ask for one-off evaluations,
or who have reporting requirements that
demand a more complex understanding of
monitoring and evaluation. This level is
particularly relevant for larger projects or
organisations which have greater resources
for evaluation.
The Practical toolkit has practical guidance
on data collection methods, gives examples
of some data collection tools, and discusses
some essential points on data analysis.
Planning
Practical monitoring and evaluation also

provides a comprehensive glossary of terms
and suggestions for further reading.
Utilisation
or
Monit ing
E v a l u a t io n
Within each section, guidance is offered at a

basic and a further level. The basic level
provides a guide to self-evaluation. This is
intended to help voluntary organisations
develop a system both for accountability
purposes and to provide essential information
for good management and service
development. Read together with the
Practical toolkit, this level is designed as an
introduction for organisations with a limited
amount or no previous experience of
evaluation, particularly small organisations.1
Even for organisations familiar with evaluation,
it may be helpful to check practice against the
different stages described.
The further level within each section follows
and builds on the basic level. It aims to meet
the needs of organisations wanting more
1
For a short introduction to monitoring and evaluation,

see also Charities Evaluation Services (2002), First steps
in monitoring and evaluation, London
A resource for PQASSO users

Although it is intended to stand alone, this
guide has also been designed as a resource
for users of PQASSO (Practical Quality
Assurance System for Small Organisations).
CES designed and introduced PQASSO in
1997 specifically for the voluntary sector.
Since then it has been used by thousands
of organisations as a tool to increase
organisational efficiency and effectiveness.
PQASSO provides a flexible, three-level
approach for stage-by-stage improvements
across 12 quality areas.
In a number of quality areas, PQASSO calls
for monitoring and evaluation activities
at three levels of complexity. Many small
organisations and projects work to PQASSO
Level 1. The basic level in Practical
monitoring and evaluation is designed to
meet the needs of PQASSO users at this
stage, as well as other small organisations
with limited resources.
The further level of guidance provides a
resource for those moving on to PQASSO
Levels 2 and 3, who need more complex
approaches to monitoring and evaluation.
A resource
for PQASSO
users
charities
evaluation
services
Terminology
There is a comprehensive glossary in the
Practical toolkit. But here are some basic
definitions of terms used in this guide.
Project
For simplicity, the guide uses the term
project to imply a fairly limited set of
activities or services with a common
management and overall aim, whether
carried out independently or as part of a
larger organisation. However, this guide is
relevant to, and can be used by, the full
range of voluntary organisations, including
large organisations which may provide a
complex range of services at a number of
sites. The principles and practice also apply
to the evaluation of programmes, in which
a number of projects are funded within
the framework of a common overall aim.
Participant
This means any person, group or
organisation taking part in project activities,
where user is not an appropriate term
as no services are offered. The term
participant is also used for those who
take part in evaluation activities.
Respondent/informant
A respondent is someone who provides
information directly, usually by answering
questions asked by an evaluator. Informants
might provide information both directly
and indirectly, for example through
being observed.
Information/data
Information is collected during monitoring
and evaluation. This becomes data when
it is gathered for a specific purpose and is
linked to specific evaluation questions.
Staff/volunteers
Many voluntary organisations have no paid
staff, and depend on volunteers to deliver
their services. This is recognised in the
guide, which often refers to both staff and
volunteers. However, for brevity, the term
staff is also used as an umbrella term to
include everyone working for a project,
whether fully employed, sessional staff,
consultants or volunteers.
Symbols
The following symbols are used in the guide:
There is more about this in the

Practical toolkit.
User
This term is used to mean any person,
group or organisation that may use an
organisations services, either directly or
indirectly. This can include clients, casual
callers, referral agencies and researchers.
Further reading relevant to this section.
inside this guide
Introduction
Evaluation and the voluntary sector
The partnership between monitoring and evaluation
Approaches to monitoring and evaluation
Challenges for voluntary sector evaluation
11
Section 1 Planning
13
Basic planning
15
Further planning
27
Section 2 Monitoring
33
Basic monitoring
35
Further monitoring
45
Section 3 Evaluation
49
Basic evaluation
51
Further evaluation
65
Section 4 Utilisation
79
Basic utilisation
81
Further utilisation
84
Section 5 Practical toolkit
89
Collecting and analysing data
91
Data collection tools 111

Further reading 130
Glossary 132
charities
evaluation
services
charities
evaluation
services
pages 7-88 new jp:basic corrected + extra pages 03/09/2009 13:33 Page 7
introduction
Evaluation and the voluntary
sector
Practical monitoring and evaluation is a
comprehensive introduction to monitoring
and evaluation for voluntary organisations
and their funders. It is intended to support
organisations which carry out their own
evaluation activities, as well as those that
use external consultants to help develop
their systems or to carry out an external
evaluation. The need for this guide has
become increasingly pressing. The voluntary
sector is complex and diverse, and is
affected by changing demands from funders,
from those in whose interests it works, and
from the general public.
This funding makes organisations more
formally accountable. Voluntary
organisations are more likely to have to
prove that their performance is of a high
quality and to adopt more business-like
management. The general public, funders
and grant givers, and service users
themselves, expect to see evidence that
projects are making a difference and that
they provide value for money.
A significant amount of voluntary and
community grant income comes from
government sources, both national and
local. Some major voluntary organisations
receive large sums from central and local
government. Local authority funding is now
increasingly available through contracts for
service delivery and more and more local
community sector organisations are funded
from local budgets. The sector is also highly
dependent on grant-aid from charitable
foundations, particularly for new initiatives,
as well as on public donations, and is
sensitive to any reduction in available
funding and greater competition for
dwindling funds.
This funding makes organisations more
formally accountable. Voluntary
charities
evaluation
services
organisations are more likely to have to

prove that their performance is of a high
quality and to adopt business-like
management. The general public, funders,
commissioners and grant givers, and service
users themselves, expect to see evidence
that projects are making a difference and
that they provide value for money.
Major policy changes have affected the
sector. Increasingly, voluntary organisations
have been asked to demonstrate proof of
their performance, often against targets.
But organisations are now asked not just to
show that they work to high standards. They
have to show how their standards can be
guaranteed, and how well they perform in
comparison to others. There is an increasing
emphasis on understanding not just what
works, but why and how it works. This level
of accountability is here to stay, so the
sector needs practical guidance on how to
demonstrate its performance effectively.
The nature of the voluntary sector today
means that organisations have to be clear
about what they are trying to achieve,
and need to develop very specific aims
and objectives along with strategic plans
and targets. Practical monitoring and
evaluation describes a model of evaluation
that recognises this emphasis on aims and
objectives, while describing the diverse and
complex range of evaluation approaches and
activities from which organisations
can benefit.
7
introduction
Evaluation
and the
voluntary
sector
charities
evaluation
services
The partnership between

monitoring and evaluation
Although monitoring and evaluation work
hand-in-hand, you need to distinguish
between them.
Monitoring
All organisations keep records and notes,
and discuss what they are doing. This simple
checking becomes monitoring when
information is collected routinely and
systematically against a plan. The information
might be about activities or services, your
users, or about outside factors affecting your
organisation or project.
Monitoring information is collected at specific
times: daily, monthly or quarterly. At some
point you need to bring this information
together so that it can answer questions such
as:
How well are we doing?
into monitoring, the fewer you should need

for evaluation. You can develop this close
connection further by involving the people
who evaluate your project in the design of
your monitoring systems wherever possible.
Approaches to monitoring
and evaluation
The following questions are key to making basic
decisions about monitoring and evaluation:
Why are you doing it?
Who is it for?
What are the key issues or questions
you wish to address?
When will you do it?
How will you do it?
Who will carry it out?
How will the information be managed and
analysed?
Are we doing the right things?

What difference are we making?
At this point you are starting to evaluate.
While monitoring is routine and ongoing,
evaluation is an in-depth study, taking place
at specific points in the life of the project.
Evaluation
Evaluation aims to answer agreed questions
and to make a judgement against specific
criteria. Like other research, for a good
evaluation, data must be collected and
analysed systematically, and its interpretation
considered carefully. Assessing value or the
worth of something and then taking action
makes evaluation distinctive. The results of
an evaluation are intended to be used.
Although monitoring and evaluation are
different, certain types of evaluation may
involve a lot of monitoring activity. It can
be difficult to carry out an evaluation
unless monitoring data has already been
collected. The more resources you put
When and how will the information

be communicated?
There are many different perspectives and
approaches to evaluation. Answering the
questions above will help you decide
whether you wish to self-evaluate or to
have an external evaluation. The questions
will help you to think about what you want
to focus on. For example, this could be:
your organisational structure and how
it works
how you carry out your services
or activities
how users experience the project
what changes or benefits the project
brings about.
The basic guidance in Practical monitoring

and evaluation focuses on helping projects to
self-evaluate, although the principles are also
useful for external evaluations. It is based on the
aims-objectives model widely used by funders
charities
evaluation
services
and voluntary organisations. In this model,

evidence is gathered in relation to planned and
clearly defined aims and objectives.
However, evaluation can be more complex
and varied. Case study evaluation, for
example, is often used to develop good
practice. This, and other approaches, are
discussed in Further monitoring and
evaluation, which also looks in more detail
at the management of both self-evaluation
and external evaluation.
Why evaluate?
The approach we take in this guide is that
monitoring and evaluation not only measure
how well you are doing, but also help you
to be more effective.
Funders and others who sponsor an
evaluation will want to know whether a
project has spent its money appropriately,
or whether it provides value for money.
There is considerable pressure from funders
to prove success. Many projects have to
respond to this demand in order to survive.
But there is a danger that an evaluation
that is unable to prove success, or defines
success unacceptably, will be rejected, and
important learning from the evaluation will
be lost. Evaluation can help you to manage
and develop your work and this is a valid
purpose in its own right.
Evaluation has two main purposes:

Evaluation for accountability to
demonstrate achievements.
The purpose of evaluation will change the

type of questions asked. For
accountability, the questions might be:
There may be some tension between these

two approaches. But it is important for many
voluntary organisations to find ways in which
both the need for accountability and the
need for learning can be met. Many funders
are also becoming more interested not only
in whether a project has worked, but why.
introduction
Has the project worked?

How has money been spent?
Should the project continue?
For learning, you might ask:
What are the projects strengths
and weaknesses?
What are the implementation problems?
Why have things worked, or not?
What are the good practice issues?
Evaluation should not only answer questions.

It should also prompt fresh thinking within
your organisation and in your contacts with
external agencies. If you have asked the right
questions, an evaluation will tell you not
only what you have achieved, but also how
you did it and what was most effective. It will
help you find the areas where improvement
or change is needed, and help you to provide
the best service to users.
Who is evaluation for?

Many different people and groups may
be interested in your evaluation. These
stakeholders may include:
the project management
staff and volunteers
funders and commissioners
Evaluation for learning and development

using evaluation to learn more about an
organisations activities, and then using
what has been learnt.
users
members
policy-makers and decision-makers
other agencies and project partners.
How you involve stakeholders in the
evaluation will affect how the evaluation
findings are accepted and, in turn, how
they are used. People are more likely to
respond and make changes if they have
The
partnership
between
monitoring and
evaluation
Approaches to
monitoring and
evaluation
charities
evaluation
services
10
been personally involved in decision-making.

Evaluation needs to answer their questions
and use methods that they think are credible
and that will provide adequate and convincing
evidence. It is therefore useful to think about
what part they will play in carrying out the
evaluation, and how they want the evaluation
findings reported back.
Quality and evaluation

The term quality is often used in a vague,
blurred way. If someone talks about working
on quality, they may simply mean activities
designed to improve the organisation and
its services. When the term quality assurance
system is used, then it means something
definite: a formal system you can use to raise
standards of work and help strengthen your
organisation.
Quality is concerned with expectations about
performance, setting standards and making
improvements. Each quality assurance system
will suggest particular areas for which you
need to make judgements about performance
against quality standards.
Many voluntary organisations have developed

quality standards and systems. They monitor
different aspects of their organisation and
projects as part of a quality check or audit. This
has led to some confusion about whether their
quality system is in itself a form of evaluation.
Quality systems sometimes use a simple form
of self-assessment to find out how well a
project is performing against quality standards.
This might, for example, use peoples
subjective judgements to reach agreement
on improvements to be made. However, this
process may not involve the level of proof or
evidence that an evaluation might require.
On the other hand, a more rigorous quality
assessment or audit will need more substantial
evidence about what the organisation is doing,
how it is doing it, and its results. Here, you
must have a monitoring and evaluation system
to give you the information you need.
Monitoring and evaluation can therefore serve
as important tools for improving quality. Work
on quality, and on monitoring and evaluation,
should be co-ordinated and feed into each
other. However, it is helpful to recognise them
as two different things.
The quality and evaluation link

asks for data
o th
Monitoring and
evaluation system
assesses the project
against a varied range
of agreed evaluation
questions
er inform ation n e
Quality assurance
system
assesses the project
against its own
expectations and
standards
ed
provides data and judgements

Example: A quality assurance system may
state that, the organisation should seek
regular user feedback on its services.
The monitoring and evaluation system
can give you data to help you show that

you meet that standard. It can also help
you understand what that feedback was,
and the reasons behind it.
charities
evaluation
services
Self-evaluation
The use of self-evaluation techniques allows

the people involved managers, staff or users
to take a fresh look at the project. Evaluation
findings should provide information for service
reviews and planning cycles, and can lead to
rethinking the very aims and objectives of the
project. Were objectives too limited or overambitious? Did they reflect the needs of
members of the community?
When an organisation uses its own

expertise to carry out its evaluations this
is known as self-evaluation. Monitoring
and evaluation is built into the everyday
life of the project and is an essential part
of project management and development.
It should not be confused with individual
staff self-assessment or appraisal. Funders
increasingly rely on self-evaluation to
provide important information on what
you are doing and how effective you are.
Self-evaluation processes and findings need

to be carefully managed. Here are some issues
to consider:
Self-evaluation should be integrated into
work activities.
Self-evaluation cycle
Self-evaluation will need resources, both in

terms of peoples time and money without
these, the quality of the evaluation will
suffer.
Ut
se
Set aims and

objectives:
what do you
want to achieve
and how?
Findings may stir up discomfort or conflict

within the organisation.
an
Pl
ili
Needs
assessment
Set performance
indicators:
to measure your
progress
Review project
and implement
recommendations
at
e
on
lu
i to
a
Ev
Deliver work
programme
Challenges for voluntary

sector evaluation
The government drive for evidence-based
policy has brought increasing pressures to
establish a direct relationship between a
programme or project and some measured
outcomes demonstrating what works.
There are a number of problems for voluntary
organisations in demonstrating outcomes and
impacts:
Self-evaluation may be carried out by

one member of staff, by an internal
team responsible for evaluation, or
responsibility may be spread throughout an
organisation. The emphasis is on
user-friendly techniques and methods,
and on collaboration and participation.
Who will decide how much change is

needed for a project to be successful?
People may have different perspectives.
And it may be different for different
projects. How does a project which
demonstrates less change for more
people compare to one which shows
dramatic benefits for a few?
Community change initiatives are putting
more value on joint working, and yet
there is a desire to show that each
individual project has caused a result.
11
introduction
Challenges for
voluntary sector
evaluation
charities
evaluation
services
12
Changes in complex initiatives may also

take place over a longer time period,
although funders are looking for
outcomes earlier in the process.
Outcomes sometimes appear to worsen
over time. Clients may report their
views more accurately as their trust
or awareness grows, and this can lead
to client outcomes appearing to be
more negative.
Some benefits to the target group are
very hard to define and measure. It can
be particularly hard for organisations
involved in campaigning or in education
and prevention work to monitor
outcomes. The work is often long term
and a small voluntary organisation
cannot measure it economically.
It is often difficult before a project
starts to collect good data against
which to measure change.
There are, then, problems in proving that
changes are attributable to the programme
or project itself and not to some other
cause. This is especially true when a project
is part of complex and collaborative
community initiatives, in which it is difficult
to isolate or show the effect any one group
is having.
There is a particular problem for projects
which try to measure outcomes for
campaigning and advocacy work. The
organisation can measure outputs, such as
the level of participation in campaigning,
and there is often information publicly
available about policy change. However,
unless policy-makers are willing to admit
that campaigning or advocacy has
influenced the change, it may be difficult or
impossible to prove the link between the
two.
This does not mean that such projects
should not try to find out how effective they
are. There is value in describing the process
of the work, and how well particular
strategies are working. For campaigning
organisations, building a persuasive evidence

base and forming alliances with other
advocates for change and those in a position
to influence policy may lead to
developments in the longer term. Recording
and reporting on those processes and
focusing on important intermediate
outcomes may be more useful than trying to
measure more long-term ones. It is also
important to research and acknowledge
other factors that may have affected the
changes that have happened.
For the voluntary sector to have confidence
in evaluation, there needs to be clarity about
how it can be used to build and develop
organisational capacity. It is also important
to understand where there are limits on what
can be reliably or usefully measured within
the time and resources available.
Clarifying your success criteria,
maintaining good monitoring systems,
and achieving good quality data
collection and reporting are essential for
credibility. This guide leads you through
several important processes: how to
clarify your aims and objectives, how to
identify indicators of success, and how to
develop and implement simple and
effective monitoring and evaluation
systems. It also explores the scope and
potential of evaluation activities that the
voluntary sector can use and the wide
range of approaches and techniques that
can be applied.
section1 planning
Basic planning
15
Understanding and clarifying your project 15

Performance indicators 21
Setting targets 23
Developing a year plan 24
Planning for monitoring and evaluation 26
Further planning
27
Linking evaluation to project planning 27

Designing an evaluation 30
charities
evaluation
services
13
charities
evaluation
services
basic planning
Basic planning describes the foundation blocks
on which monitoring and evaluation are built.
This section tells you how to make clear
statements about what your organisation is
doing, with whom, and why. It shows you how
to word these statements so that your
activities, and the resulting changes, can be
measured and reported on. It looks at a simple
year plan and sets out the points you must
remember when planning a monitoring and
evaluation system.
This section describes a sequence of

planning steps:
Stakeholders expect clear statements about the

reason for the projects existence and what it is
doing. This clarity is also needed
to plan the projects work and to monitor
and evaluate it. These statements are usually
expressed in the following way:
mission or overall aim your
organisational or project purpose
values the organisational principles you
work to
specific aims the changes or benefits
you intend to achieve
objectives the activities you carry out.
Think about which of your stakeholders will
be involved in thinking these through:
the trustees
staff and volunteers
users
Clarify the target group

Identify outputs and outcomes
Set performance indicators
Developing a mission statement

A mission statement expresses the overall
purpose of an organisation. It is sometimes
called a strategic or overall aim. It defines
and summarises the long-term change or
difference you want to make. Because it
expresses this in broad terms, it will need to be
supported by more specific aims.
If no mission statement exists, you may want to
develop one. It is often helpful for staff and
trustees to develop the statement together.
It should be short and clear, and cover the
essential aspects of your work. Think about the
people you work with, the nature of the work
and the broader or long-term changes that you
want to result from your work.
It is useful to check whether your mission
statement accurately and actually describes
the change you want to make, and then to
review it every three to five years, particularly
if there are obvious external changes that
might affect it.
other community members

specialists or experts in the field.
You also need to be clear about the target
group or target groups for your activities
or services.
If these statements were not clearly
developed when the project was first set
up, it will be helpful to do this as a first step
in monitoring and evaluating the project.
15
basic
planning
Clarify project aims, objectives and values
Set targets
Understanding and
clarifying your project
charities
evaluation
services
Example mission statement: Thursbury

Community Theatre Project
The Community Theatre Project works with
young people in schools, community centres
and other community venues. This is its
mission statement:
To enhance the lives of young people in

Thursbury through the integration of drama
into education and community activities.
Understanding
and clarifying
your project
charities
evaluation
services
16
Expressing values
Mission statements usually include some
core values that describe an organisations
approach when carrying out its activities,
for example, the way you work with service
users. Values may also be expressed in a
separate statement.
Organisations often use words such as
these to express the way they work: flexible,
respectful, caring, empowering, providing
equal opportunity.
Various expressions are used to describe the
basis of relationships in the organisation:
involving users in service planning, working in
partnership, collaborative.
People inside an organisation often make
assumptions about organisational values,
sometimes wrongly. Be sure to discuss values
openly, so that everyone can agree them. You
can then incorporate the statement into your
promotional material and any other
descriptions of your project, for example
materials prepared for funders and other
agencies.
Values that have been agreed and clearly
stated can be integrated into everyday work,
and monitored to show how successfully you
are working within them. Funders sometimes
have a statement about their own values, and
they may expect you to demonstrate that you
are working towards them. This may be a
problem if no-one decides how these values
might affect the way the project works.
It is helpful to identify and distinguish

between your first-line users and your end
users. Your first-line users may not be your
actual target group, but may be their
parents or carers, or other agencies
providing services for them. The end users
will be the children or people being cared
for. You may also have indirect users, such
as other agencies which refer people to you.
It is important to be clear about who these
different groups of users are.
Needs assessment
The starting point for any project is to
understand the needs of the group it plans to
work with, whether they are potential users,
partner agencies or those it wants to
influence.
A needs assessment is a form of evaluation
activity. It will identify the existing problems
in the community, how bad these are, the
services available and the needs that are not
being met. Look for local data and any
relevant national statistics and reports, and
talk to potential users and other
stakeholders to get information on:
the physical and economic characteristics
of the projects environment
the nature of the target population and
where they are
existing community groups and projects,
current service provision and gaps in
provision, and whether partnerships are
possible
problems and possible solutions.
Defining your target group

Your target group is the main group or groups
in society that you work with. It is important
to be clear about who they are. They could be
identified by geographical area or a place, such
as a housing estate or school, by age, gender,
sexual orientation, ethnicity, or by health
status or disability. You may also want to
identify socio-economic groups, such as single
parents, children excluded from school or
homeless people.
You can then plan what is feasible and clarify

what information you need. This avoids
collecting large amounts of data that cannot
be analysed.
Target your questions to different populations
or groups. Hard-to-reach groups may be the
very groups you want to work with. When
contacting ethnic minorities, talk to
community workers and members as well as
charities
evaluation
services
community leaders. Think about using

translators in your needs analysis, and talking
widely within the community to reach
hidden groups, including women, young
people, smaller religious or ethnic groups
and disabled groups.
Different groups or individuals may well
have different ideas about what is needed.
So ask about problems and the available
resources and services, rather than for lists
of needs or solutions which may give you
contradictory answers or raise unrealistic
expectations.
what you are doing. It is useful anyway to

review them when you start your selfevaluation activities, to make sure they are
clear, that they continue to be relevant, and
that they are agreed and understood by
everyone. Try not to have too many aims
and objectives and dont make them vague
or too ambitious.
Aims will usually describe:

the people the service or activity is
intended for
the intended benefits, or changes you
expect to make, for your target group.
Aims and objectives

Your mission or overall aim is likely to be too
broad a statement to allow you to plan your
work in detail, or to provide guidance on
what you might monitor and evaluate.
Specific aims statements about different
aspects of your main purpose will allow you
to think clearly about what you want to
achieve.
Aims should show an understanding of the
needs of the target group. These more
precise statements are also easier to evaluate.
Specific aims are likely to remain relevant for
several years, but should be reviewed
regularly.
Aims and objectives are linked in the
following way:
Aims are the benefits or changes you are
trying to achieve.
Objectives are the methods or the
activities by which you achieve your aims.
It is important to make this distinction in
order to be clear about what you are doing,
why you are doing it, and to be able to assess
your achievements. Projects often collect
information fairly randomly because they are
not clear about what their key activities are.
You will usually set your aims and objectives
as part of the project planning process. If
you did not, it is never too late to clarify
Aims may also identify the area in which

the project will work. Although they should
be clear, aims do not specify exact details
of what will be done to achieve them, nor do
they spell out a timescale.
Example aim: Thursbury Community
Theatre Project
The Community Theatre Project has three
specific aims. Its first is about the performance
work it does with young people, that is:
To change young peoples attitudes about

social issues through drama.
Objectives identify what the organisation
will do and the services it will provide. Each
aim will have one or more objective, and
some objectives may relate to more than one
aim.
Example objective: Thursbury Community
Theatre Project
The following objective of the Community
Theatre Project relates directly to the aim of
changing young peoples attitudes,
describing what it will do to achieve it:
To perform plays in schools and community

venues in co-operation with other voluntary
organisations.
The language you use will help you keep your
aims and objectives distinct.
17
basic
planning
charities
evaluation
services
18
Aims tend to start with words that indicate

a change
Objectives tend to start with words that

indicate activity
to increase
to reduce
to expand
to organise
to conduct
to provide
to enable
to develop
to improve
External aims and objectives relate very

specifically to the organisations mission and
the tasks it is trying to achieve. The triangle
diagram below shows how to organise aims
and objectives. It shows how all the aims and
objectives of the Community Theatre Project
feed directly into one main aim or mission
statement. The triangle should describe all the
projects external work, making it clear what a
project does and why. Keep the number of
aims and objectives limited and make them as
focused as you can.
to distribute
to produce
to set up
To fully understand the diagram, it is helpful

to understand a little more about the
theatre project. It works throughout
Thursbury, but focuses its efforts on four
wards where there are a number of failing
schools and high levels of teenage crime
and drug problems. The project has a
coordinator, an administrator and two
project workers who research social issues
and liaise with schools, social services
and voluntary organisations working with
young people.
s
i ve
ct
je
Ob
To enhance the lives

of young people in
Thursbury through the
integration of drama
into education and
community activities
To change
young peoples
attitudes about
social issues
through drama
Sp
ec
i fi
ca
im
Ov
er
all
aim
The Community Theatre Project
To perform plays in
schools and community
venues in co-operation
with other voluntary
organisations
To enable
young people
to express
themselves
through dance
and drama
To run dance
and theatre
workshops for
young people
To enable
teachers and
community
workers to use
drama in teaching
and play activities
To hold theatre
skills courses for
teachers and
community
workers
To work with
schools on
integrating drama
into the school
curriculum
charities
evaluation
services
You can do the same thing with internal or

organisational aims and objectives. These
are mainly concerned with helping the
project achieve its external objectives. They
relate, for example, to management, finance
and personnel.
Once you have described your project in
clear statements, with an overall aim and
more specific aims and objectives, you will
be able to describe your project activities
and their effects in more detail, and in
ways that can be evaluated. The terms

output , outcome and impact are used. These
relate directly to aims and objectives what
you plan to do as shown in the diagram
below.
Inputs are the resources you put into the
project, such as staff and volunteer time,
funding and technical resources. Inputs
will directly affect the quality and level of
your outputs. These in turn will affect your
project outcomes.
Impact
Overall aim
Why we do it
Specific aims
Objectives
Outcomes
What we do
Outputs
Inputs
Outputs
The outputs of the Community Theatre Project
are the work generated by the project, and relate to
the projects objectives. The following are examples:
Objectives
Outputs
To perform plays in schools and community venues in

co-operation with other voluntary organisations
Meetings with other agencies

Rehearsals
Publicity leaflets
Performances
To run dance and theatre workshops for young people
Weekend workshops
One- and two-day workshops in schools
After-school courses
Holiday schemes
Productions
To hold theatre skills courses for teachers and

community workers
Introductory talks
Day workshops
Week-long courses
To work with schools on integrating drama into the

school curriculum
Meetings with teaching and support staff

Curriculum discussions
Classroom assistance
19
basic
planning
charities
evaluation
services
20
Outcomes
The outcomes of the Community Theatre Project
are the changes or benefits that take place as a
result of project activities. These relate to the
specific aims. It is important to assess only factors
which the project can reasonably control or
make happen.
Aims
Outcomes
To change young peoples

attitudes about social
issues through drama
Greater understanding of social issues raised

Greater awareness of own attitudes
Greater tolerance of other lifestyles and viewpoints
Changed views on social issues
Greater use of drama with young people by other
voluntary sector services
To enable young people to

express themselves through
dance and drama
Increased confidence of young people

Increased skills in dance and drama
Increased sense of commitment and responsibility
Development of team-working skills
Young people continue to develop theatre skills through
further opportunities
To enable teachers and

community workers to use
drama in teaching and
play activities
Ability to link drama to curriculum content

Acquisition of basic theatre skills and their application
to teaching
Greater use of drama in the school curriculum
Impacts
Impacts are the broader and longer-term changes
relating to your overall aim, or mission. The
Community Theatre Project would have to
demonstrate that it is meeting the overall aim
of changing the lives of children through
the use of drama.
Inputs
Outputs
Outcomes
Impacts
charities
evaluation
services
Performance indicators
There is one more key step in laying the
foundation for monitoring and evaluating
your project. To check how your
organisation is performing, you will need
to set performance indicators. They are
the criteria or clues on which to base
judgements about the progress and success
of the project.
Performance indicators:
let stakeholders know what they can
expect from the organisation
provide a focus for managers
help you to focus on what you need to
monitor and evaluate
help a project to judge its achievement
and effectiveness
help comparison between projects.
Who defines the performance indicators
for your project is important, as different
stakeholders will have different views about
what counts as success. If you involve them
you will have a wider and richer perspective.
Brainstorming can be a useful way to find
out how different people understand success
or failure. It may be helpful to look at
indicators that similar projects are using.
Funders and commissioners may want to
have performance indicators that are easily
measurable or that link to their own
strategies. This could mean measuring things
that are not at the heart of the project. So
discuss your key activities and intended
benefits with funders early on. Evaluation
priorities should not take over project
activities and divert you from your planned
priorities.
You may want to involve users in developing
indicators. Users have a unique and valuable
perspective on success. But be clear
yourselves, and with users in advance, what
weight will be given to their views when final
decisions are made about indicators.
It should take about a day to work with a small

user group. Here are some tips to help you:
21
basic
planning
make sure everyone has a clear list of

project aims and objectives
develop some user definitions of what
change or success will look like
discuss any indicators staff have already
developed.
Users and others may not be familiar with
monitoring and evaluation, so try to choose
your language with care.
Some projects, for example environmental
groups, may not provide a direct service to
users or may have other important aspects to
their work. In this case, you still need to identify
your stakeholders and to think carefully about
what will indicate success.
Types of indicator
There are a number of different types of
indicator. The most common are output and
outcome indicators. These are often confused
with impact indicators, but there is a difference.
Output indicators these demonstrate the
work the organisation does and show
progress towards meeting objectives.
Outcome indicators these demonstrate
changes which take place as a result of the
organisations work, and show progress
towards meeting specific aims.
Impact indicators these demonstrate
broader, longer-term change, often relating to
the overall aim or mission of the organisation.
Output indicators
Output indicators can be set for:
quantity (how many services or products)
take-up (used by how many people)
accessibility (used by what type of people)
quality (including user satisfaction)
cost.
Performance
indicators
charities
evaluation
services
22
There will be a great number of possible output

indicators, so you will need to prioritise and
limit them.
Example for the Community Theatre Project
Output
Indicators
Meetings with
agencies
Number and type of agency contacted and met; nature and

length of relationship
Publicity leaflets
Number of leaflets printed and distributed; locations distributed;

quality of leaflets; number and type of contact generated
from publicity
Performances
Number of performances; subject matter; profile of schools and

community venues; cost; source and level of sponsorship;
satisfaction levels
Weekend workshops
for young people
Attendance levels; profile of young people; level of participation;

drop-out rate; proportion of cost covered by users and sponsorship
Curriculum
discussions
Number of preliminary and follow-up discussions; numbers and

types of teaching and support staff involved
Outcome indicators
Outcomes and their indicators are often
confused, but are quite distinct from one
another. Agree your expected outcomes by
thinking about what success would look like for
each aim, that is, what change would happen.
Develop indicators by agreeing what would
demonstrate that this change had taken place.
Example for the Community Theatre Project
Outcome
Indicators
Level and extent of discussion generated after the performance

Young people have
greater understanding Level of increased understanding expressed by young people
of the social issues
Extent of changed attitudes and behaviours reported by teachers
raised
and play workers
Increased sense of
commitment and
responsibility
Individual attendance levels at workshops and rehearsals

Time keeping
Level of support and encouragement given to each other
Numbers of young people following classes through to performance
charities
evaluation
services
Limit the number of performance indicators

for each specific aim. Think about the
resources you have available and collect
a focused range of information against
your evaluation questions.
Attendance (for example, at a training
course, a clinic session or on a programme)
is an output indicator, indicating access to
a service. It is rarely an outcome indicator
because no change has happened. The
following example, for a health education
project, illustrates an exception to this:
Aim: to encourage users to take more
responsibility for their health
Outcome: the increased use of services
Indicator: attendance levels at clinics.
However, if the aim was, for instance,
to improve the health of Bangladeshi
communities, outcomes should relate
to those health improvements, not just
attendance at health centres.
Many aims may be abstract or intangible.
For example, to change young peoples
attitudes about social issues is an aim that
could be difficult to measure, as it does not
lend itself to being counted. In areas that
cannot be measured directly, you may need
to use indicators that assess the change
approximately. These are called proxy
indicators. For example, time keeping and
attendance levels, taken together with
other evidence, may be an indication of
an increased sense of commitment and
responsibility, the expected outcome.
When indicators are difficult to assess, it
could be helpful to get advice from
practitioners in the field. There are also
developments in the creation and validation
of different outcome assessment tools and
measures, for example, of well-being. You
may also be able to find some examples of
indicators contained within indicator banks
relevant to your work area. Other agencies
may also be able to offer examples of
indicators, or of self-report questionnaires
or other assessment tools

You can change indicators if you find them
too difficult to assess or if they are not
meaningful. However, there should be
some stability if you want to be able to
demonstrate patterns, trends or changes
over time.
Setting targets
Often it will be useful to compare
monitoring data against targets that you
have set in advance. Targets are expected
levels of achievement in relation to inputs,
outputs, outcomes and impacts against
which you can measure performance.
Indicators on their own have no specific
value. For example, if an output indicator
for the Community Theatre Project is
number of performances, you do not know
whether the project would be successful
if it gave five performances or 50. To help
define success, you may need to qualify
the indicators by setting targets.
Targets specify the level and quality of
activities and achievements that you
expect or hope for. They may be needed
for funding applications, but there are a
number of reasons for setting targets. They
can help with operational and individual
work planning, help the project to improve
performance, and help to demonstrate and
improve value for money. They will provide
a benchmark or standard to monitor
and evaluate performance or progress
against, and to help document progress
towards achieving aims and objectives.
At the same time, projects need to be wary
of allowing targets to drive activity in a
way that might limit the time given to other
areas of work or less tangible achievements.
Targets are often set for outputs. The
following examples demonstrate targets
for a number of different projects.
23
basic
planning
Setting
targets
charities
evaluation
services
24
Project
Output indicators
Output targets
Community
development agency
Number of new projects

set up
Three new projects during the year
Drop-in service
Number and profile

of users
Increase in number of women

attending by 20% in the next year
Training project
Level of satisfaction
with courses
90% of users record overall

satisfaction as high or very high
Targets may also be set for outcomes. For example:

Project
Outcome indicators
Outcome targets
Employment training
programme
Percentage of trainees
getting employment
65% of trainees placed in full-time

jobs within six months of finishing the
programme
Alcohol advisory
project
Amount of weekly
units drunk
75% of clients reducing the number

of weekly units drunk during the first
six months of counselling
Young peoples
counselling project
Numbers of young
people remaining in the
family home
60% of young people remaining in the

family home
Objectives need to be Specific, Measurable, Achievable, Realistic and Time-based.

Developing targets for specific objectives will make them SMART.
Developing a year plan

Once you have stated clearly what your
project is, a year plan, setting out what you
will achieve over the year, is an important
tool for monitoring and managing your
project. Some funders require you to submit
one, particularly in your first year.
Setting milestones and targets

Milestones are particular planned
achievements or key events marking a clear
stage in completing a main phase of the
project. You may agree these with your
funder and set them to a timetable. The
following are examples for a new project:
trustees appointed
workers in post
promotional activity underway
partnership agreements in place
conference held
first trainees enrolled or
completing training.
Many organisations will have an
agreement with their funder to give a
certain level of service, expressed as
targets (see above). High targets can help
to provide vision and ambition, and to
obtain funding. Low targets might be
charities
evaluation
services
more realistic and achievable. Finding the

right level is important for maintaining
credibility and for staff and volunteer
morale. They should not add up to more
than your capacity, so take into account:
work of individual staff members fits into

overall work plans. The simple plan below
shows the key events and timetable for the
first six months of a counselling project.
A more complex year plan will link
proposed activities with a budget and set
out a clear plan of action against each
objective, showing the following:
your resources
what you have achieved before
needs assessments
what activities will take place
what other organisations

have achieved.
how they will be carried out
When setting targets, make sure that they

cover your key activities. Allow for start-up
time, staff holidays and other things that
limit your activity.
It is a good idea to set your milestones
and targets with the staff responsible for
achieving them. There will be greater
commitment to reaching the targets set,
and a better understanding of how the
who will be responsible for what

and when
what resources are needed.
By collecting quarterly figures against your
targets, you will have some warning if your
targets are not likely to be met. If you give
good reasons for this, preferably early on in
the financial year, many funders will be
happy to renegotiate targets.
Counselling project year plan (first six months)

Activity
April
May
June
July
Aug
Sept
Project coordinator in place

Project administrator in place
Database installed
Counsellors appointed
Strategic planning day
Away day to discuss monitoring
and evaluation systems
Promotional activity to schools
First users seen
Review of project start-up
Planned activity
25
basic
planning
Developing
a year plan
charities
evaluation
services
26
Planning for monitoring

and evaluation
activities. If funding is uncertain, it may be

difficult to plan your evaluation realistically.
Once you have a year plan for your work,

you can set out a monitoring and evaluation
plan. In this, you may well be prompted by
what a funder wants you to monitor. Think
also about other information that you need
to help you manage the project efficiently
and effectively, and about who else might
be interested in learning about what you
are doing.
The timing of the various stages of an

evaluation is important. Build in time to
process the data, that is, to organise it, and
possibly enter it onto the computer or in
paper files. Time the presentation of the
evaluation report carefully so that it is
available when decisions have to be made
about the project, otherwise opportunities
for action may be delayed or lost. Plan how
to spread your evaluation findings to a
wider audience.
In planning for your monitoring and

evaluation activities, consider questions that
will help you prioritise. For example:
What is most important to evaluate at
this stage of the project?
both evaluation and the project itself

have good management support
What are the most significant outputs

and outcomes?
staff and volunteers understand and

value monitoring and evaluation tasks
Do your IT systems and resources help

you manage your information sufficiently?
enough time is set aside for monitoring

and evaluation activities
How can you best build self-evaluation

activities into your routine work?
How will the information be collated and
analysed? What skills are needed?
Further reading:
Davey, S et al (2008)
Using IT to Improve
your Monitoring and
Evaluation, Performance
Hub, Charities
Evaluation Services,
London
monitoring and evaluation plans

are realistic
What evaluation questions do you

want answered so you can plan next
years work?
Think about how you will carry out

monitoring and evaluation, who will do it,
and what methods you will use. You also
need to consider management, support and
supervision issues. For example:
Cupitt, S and Ellis, J

(2007) Your Project and
its Outcomes, Charities
London
Your evaluation is more likely to

succeed if:
How can you make sure that staff will

collect the information and collect
it consistently?
When is the information needed?
Even though funders may not recognise the
need for funding purely for evaluation, try
to build costs for evaluation into your
budget right from the start. This will avoid
having to divert scarce resources from other
evaluation is built into planning cycles,

job descriptions and budgets.
Basic planning looked first at how to set

out clearly your project aims and
objectives. It then moved on to describe
tools that will help you to assess how
well you are doing, including the use of
performance indicators and targets.
Finally, but most importantly, we raised
the need to ask questions about the
information you and your stakeholders
need and the importance of working out
how that information will be collected
and communicated.
further planning
charities
evaluation
services
Basic planning demonstrated the

importance of defining project aims and
objectives, clarifying the project inputs,
outputs and outcomes, and understanding
the links between them. It also highlighted
the use of performance indicators and
of monitoring and evaluation planning
procedures. Further planning looks at
more techniques both for project planning
and for planning monitoring and evaluation.
27
further
planning
Linking evaluation to
project planning
Your first planning stage is a good needs
assessment, which will make clear
recommendations based on the needs,
problems and solutions it has identified. A
needs assessment can also provide baseline
data, against which you can measure
changes during the lifetime of your project.
period. Operational plans flow from

strategic and business planning and set out
a programme of work for a one- or two-year
period.
It is helpful to link your planning for
monitoring and evaluation into your project
planning.
Logical framework analysis

One useful planning tool which makes the
link between project design and evaluation
is the logical framework analysis. This is
used for planning and managing projects in
international development work. It is a tool
that describes the project and its
component parts in a logical and linked way.
The logical framework, or logframe, is a
matrix showing:
the projects aims and objectives
the means required to achieve them
how their achievement can be verified.
Once you have identified needs, project

planning is carried out through two different
planning levels: strategic or business
planning and operational planning. Strategic
planning and business planning set out the
main directions of the project and its
financial health for a three-to-five-year
The logical framework matrix

Project structure
Indicators
Means of
verification
Broad aim
Project purpose
Outputs
Activities/inputs
Adapted from: Gosling, L and Edwards, M (2003) Toolkits: A Practical Guide to Monitoring,
Evaluation and Impact Assessment, Save the Children, London
Assumptions and
critical factors
Planning for
monitoring and
evaluation
Linking
evaluation
to project
planning
charities
evaluation
services
Cracknell identifies the vertical logic that

traces cause and effect from project inputs
through outputs to achieving project aims,
and a horizontal logic which builds in how
achievement will be verified through
indicators and evidence. One danger is that
it becomes too ritualistic or mechanistic.
Another danger is that there may be too
much emphasis on easily quantifiable
indicators, leaving little room for flexibility
and ways of measuring unintended effects.
28
The logframe may use some terms (output,

purpose and aim) differently from the way
we use them in this guide, but this does not
need to be confusing. The main advantage
of the logframe is that it links monitoring
and evaluation directly with project

management. Another useful element in
the logframe is that important assumptions,
without which progress or success would
be compromised, are identified and built
into project design. These might include
government policy or resources, or the
actions of other agencies or, within the
project, the recruitment of qualified staff.
Outside factors often have an important
influence on the success of a project and
should be identified early on. Periodic
checks are made to see whether
assumptions have been realised.
There are three levels of planning for
evaluation:
Level 1
Evaluation strategy
Evaluation priorities over a period of time
Level 2
Evaluation planning
Making evaluation operational
Level 3
Evaluation design
Approaches
and methods
Cracknell, BE (2000) Evaluating Development
Aid: Issues, Problems and Solutions, SAGE, London
charities
evaluation
services
Developing an evaluation strategy

Once your strategic planning is in place, it
is useful to develop a planned approach to
evaluation activities. This is your evaluation
strategy. It presents agreement on what will
be evaluated within the organisation over
a period of time, taking into account time,
money and effort.
You will need to evaluate at strategic points
in the life of a project. You will also have
different information needs at different
times, for example:
to examine new or innovative projects
to explore something that is not
working well
to further investigate user or other
stakeholder feedback
a project may be coming to the end
of its current funding and you need
good evidence of its worth to ask
for continued funding.
An evaluation strategy needs to consider

the following questions:
What is the purpose of monitoring
and evaluation?
Who are the stakeholders and audiences?
What are the key questions over a
period of time?
What types of evaluation will be
needed at different times?
What resources are available?
It is helpful to think ahead and map out the
evaluation activities you might need over the
period of your strategic plan, or longer. But
there will be some you will not be able to
plan in advance. Remember that if you
identify your evaluation needs in advance,
you will be able to plan monitoring systems
to systematically collect the data you need,
and in a form that will address the evaluation
questions. This will help to avoid collecting
unnecessary data.
Example: Mapping evaluation activities over time

When
Type of evaluation
Who
Year 1
Needs assessment
Baseline data
End of year report
Student researcher
Student researcher
Internal
Year 2
Implementation (process)
report
Internal
Year 3
End of year progress report
Internal
Year 4
End of year progress report

Good practice research
Internal
Internal plus external evaluator
Year 5
Outcome evaluation
Internal plus external evaluator
29
further
planning
charities
evaluation
services
30
Developing an evaluation strategy has a

number of benefits. It:
links evaluation activities to other
management activities, such as year
plans and strategic plans, so that the
project has the right information at
the right time
helps to plan for the resources
needed to cover the costs of the
evaluation activities
encourages thinking about evaluation
as a longer-term process, with a
different focus for evaluation activities
in different years.
Developing a monitoring and

evaluation plan
Your monitoring and evaluation strategy
will provide an overall framework to guide
monitoring and evaluation activities within
a broad timeframe, in line with your
strategic planning or even the lifetime of
the project. Basic planning discussed how,
when you draw up your projects year plan,
you also need to think about how you will
carry out monitoring and evaluation over
the year.
For this next level of planning you need to
ask yourself:
When will different data be collected?
Who will carry out and manage
the activities?
How will data be collected, stored,
analysed and presented?
How can routine monitoring activity
support evaluation?
How will evaluation findings be used
in short-term planning and decisionmaking, quality reviews and other
key processes?
Focusing your evaluation activities is vital.
This focus, in turn, shapes the questions to
which you want answers, and these should
now be clearer. The following questions are
quite different from each other:
How are resources being used?

How appropriate is the management
structure?
How well are we meeting identified needs?
How do we fit within a network
of services?
How well have we met our expected
outcomes?
Designing an evaluation
Your evaluation strategy will describe the
evaluation activities needed at different points
in your project management cycle and over
the lifetime of the project. You will then need
to begin to implement your monitoring and
evaluation strategy by planning activities over
a shorter time. Each separate evaluation will
also need designing. This will focus on the
approaches and methods to be used, decide
who the respondents will be and set out a
detailed timetable for activities.
Steps in evaluation design

When you design your evaluation, think about
the technical skill level of the evaluator or
evaluation team and how much data you will
be able to analyse and use. List the evaluation
questions and then look at possible data
collection methods, both quantitative and
qualitative. Your choice of method should
take account of the likely quality of the
results and the practical issues involved in
collecting the data.
Design matrix
It is helpful to summarise the design of your
evaluation activities in a design matrix. This
usually includes the following elements:
key evaluation questions
indicators the things you will assess
to help you to make judgements
data collection methods
data sources
data collection timetable.
charities
evaluation
services
The example matrix below shows part of

the design options for the evaluation of
St Bernards Homes, which provide hostels
and shared accommodation for older people.
Once the options have been considered, it
may be necessary to make choices about
what to include in the final design, so that
the evaluation is manageable.
Project aims
and objectives
Key question
Aim:
To enable older
people living in
St Bernards
accommodation
to make informed
choices about their
housing, social,
health and care
needs
To what extent
did older people
acquire benefits
not previously
received and use
health and care
services not
previously
received?
The mission, or overall aim, of St Bernards

Homes is as follows:
31
further
planning
To improve the health, social and housing

status of older people living in St Bernards
hostels and shared accommodation.
Designing an
evaluation
The matrix is completed here for only one of

the aims, and the same approach should be
used for each additional aim.
Indicators
Number of older
people with
Housing Benefit
Total amount
of Housing
Benefit received
Data sources
Users
Data
collection
methods
Timetable
Analysis of
case records
April/May
Financial
analysis
of records
April/May
Interviews
June
Questionnaire
May/June
(analysis
early July)
Users
Desk research
April/May/June
Staff
User feedback
forms
Continuous
Interviews
May/June
Staff
Other agency
personnel
Case records
Numbers receiving
health care
Types of
services received
Numbers
receiving care
packages
Types of care
packages
Objective:
To provide
information and
advice to older
people in
St Bernards
accommodation on
available housing
benefits, personal
care services, health
services and life
skills training
How well was

the service
used?
How appropriate
was the service
to users needs?
How did
statutory and
independent
agencies
participate, and
to what extent?
Number of older
people who have
had needs
assessments
Level of user
satisfaction
Number of people
contacting statutory
and independent
agencies
Extent and type
of working
relationships with
other agencies
Other agency
personnel
Needs
assessments
Service
participation
monitoring
Case reviews
Meeting
minutes
charities
evaluation
services
32
It is better to complete an evaluation

successfully with a very simple design than
to only get part way through a complex one.
If you put down on paper all your possible
evaluation activities, this makes it easier to
assess whether you have enough resources
and whether your design is realistic.
Think about the timetable of monitoring
and evaluation activities and whether they
can be brought together. Your management
information system may well contain a lot
of information, for example client progress
reports, which may be useful for your
evaluation. You may be able to set up a
single, common database, which can be
used for both monitoring and evaluation.
Too much emphasis on achieving aims and

objectives may mean that you overlook
unintended effects of the project or the effect
of external events, such as local or national
government policy or closures in other local
services. Your evaluation design should be
flexible enough for these issues to be
considered in the analysis and interpretation
of the findings.
This section has covered the complex
planning activities for evaluating your
project. It has shown how the different
levels of planning for evaluation start
from, and can be built into, strategic and
operational planning. Long-term planning
for evaluation activities, how you turn
your evaluation plans into action, and
the careful design of approaches and
methods, are all critical to success.
Needs assessment
Project strategic plans
Evaluation strategy
Project operational plans
Evaluation plans
Further reading
Cracknell, BE (2000)
Evaluating
Development Aid:
Issues, Problems and
Solutions, SAGE,
London
Robson, C (2000)
Small-scale Evaluation:
Principles and
Practice, SAGE, London
section 2 monitoring
Basic monitoring
35
Setting up your record keeping5
35
Feedback from users5
40
Outcome monitoring5
41
Analysing monitoring data5
43
Reporting monitoring data5
43
Monitoring your monitoring5
44
Further monitoring5
45
Developing more complex monitoring systems5
45
Process monitoring5
45
Monitoring for quality5
46
Impact monitoring5
47
Monitoring context issues5
47
Data management5
48
charities
evaluation
services
33
charities
evaluation
services
basic monitoring
Basic monitoring discusses how to set up
your record keeping. It looks at the different
elements of your project that you might
monitor. These include inputs (the resources
you put into your project), and your outputs,
including the scale and pattern of use, and
financial monitoring. It discusses using an
outcomes approach and developing outcome
monitoring systems. It looks at good practice
in monitoring user profile information,
introduces feedback from users, and suggests
a framework for analysing and reporting
your monitoring information.
Setting up your
record keeping
Monitoring is the routine and systematic
collection of information so that you can
check regularly on your progress. Beware
of collecting information randomly without
thinking through what information you
really need. Planning is vital in helping you
to monitor in a focused and systematic way.
It is not realistic to collect information for
all the possible performance indicators you
identified. This is why it is important to
discuss with your stakeholders, particularly
trustees, users and funders, the key
indicators for which you are going to collect
information.
Monitoring should give you enough
information to tell you what is working,
identify problems, tell you who is using
your services and how, and help you plan
to meet changing needs. It also means you
can provide relevant information to other
agencies, develop useful publicity, and tell
funders and other stakeholders about
progress. Write monitoring tasks into job
descriptions and planning cycles to make
sure that responsibilities are spelt out and
that everyone is clear about the importance
of monitoring.
Setting up record keeping, whether this is
paper based, computerised, or both, is a key
part of project start-up activities and should
be done early on. It is helpful to think

through at an early stage how you will
collect the data and how it will be organised.
Records might include data on project
activities and users, staff and management
meetings, project funding and finances, and
supportive agencies and individuals. Develop
some interesting ways of collecting data. For
example, photographs can capture activities
and events, or show changes in physical
environments.
charities
evaluation
services
35
basic
monitoring
Setting up
your record
keeping
Whenever possible, input information such

as membership data, publications sales and
advice records directly onto a computer
database. Make sure someone is responsible
for dating, filing, storing and collating
monitoring data collected on paper. You
also need to make someone responsible
for inputting and analysing the data held
electronically.
Get good advice on how your database can
best serve your information needs. Think
about the links you want to create between
different types of information. For example,
you may want to be able to link user profile
data with attendance at different activities.
It may be useful to talk to another project
with a good management information
system and to a database expert, if possible.
There are a number of IT systems available
that are designed to facilitate monitoring
and evaluation, and an increasing number
that will help you to manage outcomes
information.
The following questions are important
to consider when you start monitoring:
What depth and type of information
do you want?
How can you check the reliability
of your information?
How much time can you afford to spend?
How much will it cost?
How will you analyse the information?
Further reading:
Using IT to Improve
your Monitoring and
Evaluation,
Performance Hub,
Charities Evaluation
Services, London
charities
evaluation
services
36
It is important to be familiar with the Data

1
Protection Act. Make sure data is used for its
intended purpose. If personal information is
kept about individual service users, make
sure that they know exactly what the
evaluation is for, what data exists, that they
can have access to it to check its accuracy,
and that the project will preserve their
confidentiality.
Here are some basic points for successful
monitoring:
build simple, user-friendly monitoring
systems into everyday activities,
collecting data at the most natural point
get commitment from those collecting
the information, by explaining why
they are doing it
make sure that everyone responsible
for monitoring has clear and consistent
guidelines
make sure that monitoring records
are completed fully and accurately
people may not regard it as a
high-priority activity
give people collecting the information
feedback on the results of their
monitoring, and how it is being
used to make the organisation
more effective
check that the project is not collecting
the same piece of information more
than once.
Very often, the information you need already
exists. So design project forms, such as
planning diaries, referral forms, assessment
sheets and case records, so that they can
also be used for monitoring. This will avoid
duplicating effort.
On membership and access forms, referral

and case records, think about using a
range of possible answers with tick boxes,
rather than providing space for narrative
or comment. This information can be
entered more easily on a database and
analysed. You can give extra space for
more information if necessary.
Transfer information from case records
onto monitoring summary sheets to
give easy access to information.
Information will then be accessible when
you need it, and you can check from
time to time that the same information
is being collected in the same way by
everyone, within specified categories.
Once you know that your system is working
well, write down your monitoring system and
build it into your organisational procedures.
Your first question before collecting
information was: What information do we
need? From the early stages of your project
you are likely to need different types of
information on your activities, your users
and your finances. We now look at these
in more detail.
Monitoring inputs
Think about the resources you put into your
services or activities. These include staff
and volunteers, management and training,
and financial resources. Dont forget the
contribution made by partner agencies.
You may want to monitor against a number
of input indicators. For example:
the amount of management or
administration time spent on certain
activities
the level and use of staff and volunteer
resources
staff and volunteer profile and skills
You can get information on the Data Protection Act at

www.informationcommissioner.gov.uk and on an
information line: 01625 545745. A publication, The
Data Protection Act 1998: Legal Guidance, is also
available from this information line.
the amount and quality of staff training

the level of trustee attendance
at meetings
charities
evaluation
services
the amount of time spent on fundraising

and promotional work
catering, travel or other costs for specific
activities.
Think about the information you need so
that you can understand the full costs of the
project, or the details needed if someone
else wanted to carry out the project from
scratch. What processes are an essential part
of what makes your project work?
Monitoring outputs
When you set output indicators, focus
these on the information that will be
useful in planning activities and services.
A pressure group might want to monitor
its level of media coverage, whereas a
service provider might want to monitor
the scale and pattern of use. This could be:
the level of use at each session,
for example, at a luncheon club
the total level of service over a given
period of time, for example, the number
of advice sessions each month
the total number of users, number of
new and repeat users, and number of
users who have left the service, over
a given period of time
frequency or extent of use by individual
people
use of premises, transport or catering.
Remember that output information will be
useful for explaining to others what you do,
what is involved in providing your service,
and what resources you need. It will also be
important in helping to understand how and
why you have achieved your outcomes.
Monitoring the profile of

your users
When you set up the project you identified a
target group or groups. So it is important to
check if the users are the ones you intended
to reach. If you are examining the profile of
users to assess patterns of service use, it may

be more meaningful if you compare it with
local population statistics, or against the
user profile in other relevant services.
Your immediate users may not be the target
group whose profile you are most interested
in. For example, if you are training other
agency staff, you may need not only to
monitor which agencies they work for,
but to get information on the target group
of those agencies, so that you can identify
the end user.
You can collect data on organisations that
use your services when you first have contact
with them, by systematically asking for the
information you need for your database.
Collect individual profile information either
through a simple questionnaire that users fill
in themselves, or one that you fill in by
asking them questions. You may wish to
collect information such as where users live,
their age, gender, sexual orientation,
ethnicity or disability. Remember that
questions should be easy to answer and
to analyse. Develop specific closed
questions with a range of categories
to tick. Open-ended questions may be
interpreted differently by different users,
need more time to answer, and will get
responses that are more difficult to analyse.
There are a number of good practice points
to think about when you monitor the profile
of your users. Information should be given
voluntarily. Wherever possible, allow a
relationship of trust to develop first and
avoid guessing profile information. Using
pre-set categories for ages, or for ethnicity,
such as those used by the Commission for
Racial Equality, will make it easier to analyse
responses for a large group of people. By
using these categories, you will be able to
make comparisons with census data. People
often wish to describe their characteristics
themselves, so for a small group of users, or
where you are working with a range of
smaller minority groups, this may be
preferable.
37
basic
monitoring
charities
evaluation
services
38
The following categories are suggested
by the Equality and Human Rights
Commission. In Scotland it will be
helpful to use questions similar to the
Scottish census questions. The Equality
Commission for Northern Ireland
recommends particular ethnic
categories in its code of practice.
White:
British
Irish
any other white background
(please write in)
Mixed:
white and black Caribbean
white and black African
white and Asian
any other mixed background
(please write in)
Asian or Asian British:

Indian
Pakistani
Bangladeshi
any other Asian background
(please write in)
Black or black British:

Caribbean
African
any other black background
(please write in)
Chinese or other ethnic group:

Chinese
any other (please write in)
If you ask profile questions in a consistent

way across different activities and services,
you will be able to make comparisons and
look at the profile of service users for the
project as a whole.
If staff and volunteers are reluctant to ask
profile questions, it might help to find out
how another similar project has collected
sensitive information and how this
information has benefited the project.
Explain to users why you are collecting
the information and how it will improve
services. You may want to reassure your
users that profile information is anonymous
and kept separate from other information,
such as case records. On the other hand,
if you want to relate the information about
benefits or changes for users to the type of
user, you will need to keep this information
linked. You can do this by giving users an
identity number and linking this to profile
information that is kept anonymously.
Some information about users is relatively
easy to get, such as addresses. You can use
postal codes to monitor against targeted
geographical areas. It may be more difficult
to do profile monitoring for certain types
of activities, for example, projects working
informally, such as detached youth work or
those with large numbers of users, such as
telephone advice lines. Think about ways
of getting snapshot information over a
short period of time. This can be explained
to users and staff as a one-off or occasional
exercise, and may be accepted as less
demanding and intrusive.
Funders and commissioners often want
forms filled in at the end of each quarter or
year. Information is sometimes needed for a
formal service level agreement. If the funder
has different categories from your project
for describing users ages or other
circumstances, you may be able to reach a
compromise if you discuss it together.
charities
evaluation
services
Financial monitoring
Financial monitoring has three main
purposes:
accountability
control
evaluation.
These relate to the different needs of a wide
range of stakeholders to know what is going
on in your organisation or project. Financial
monitoring systems should be built in at
the beginning of a project. This allows
appropriate information to be collated as
an integral part of the project, and allows
reports to be produced at regular intervals.
Integrating financial information needs
Use a coding system to show the different
types of income and expenditure. This allows
summary information to be broken down
into categories, for example, to highlight
income and expenditure for a particular
project or fundraising activity. At the same
time it will allow income and expenditure
to be categorised for the statutory accounts.
External accountability
By law, your project will have to keep
accurate accounting records of income,
expenditure and assets. As well as the
legal requirements, a range of external
stakeholders such as funders, regulatory
bodies and users will have an interest in
particular aspects of the projects finances.
For example:
a funder giving money for a particular
purpose will expect to see financial
reports showing that the money has
been spent effectively according to
their wishes
a regulatory body, such as the Inland
Revenue, will expect to see a financial
monitoring system that allows them to
check that you have kept to the tax laws
users may be interested in various
aspects of how effectively you use your
resources.
Control
Trustees, staff and volunteers need financial
information to make sure that legal
responsibilities are fulfilled, and to
understand what is going on in the project.
Regular reporting of progress against an
agreed annual plan and budget is a key
means of control. However, a budget is
only an estimate, and what actually happens
may not be what was planned.
You will normally report monthly for staff on
actual income and expenditure compared
with budgeted income and expenditure.
Trustees might see reports less often, say
quarterly, to fit in with the trustee meeting
schedule. Major differences between
actual and budgeted expenditure should
be highlighted and explained to make sure
that the project does not spend more than
it can afford. It is also helpful for staff to
produce a forecast of what the year-end
position is likely to be. This can be done
quarterly for trustees and lets them take
action, if appropriate, to make sure that
an acceptable year-end position is reached.
Monthly cash flow projections are also
useful.
Evaluation
Reviewing income and expenditure for
particular activities, and monitoring trends
over time, can help to evaluate the financial
implications of the different activities of
the project. This is important for planning
service delivery and fundraising. For
example:
the cash flow will show whether there
are enough funds at all times both to
implement and run the project
evaluating the cost of providing different
levels of service, compared with income
received, will help with decision-making
when planning services
the relative cost-effectiveness of
different fundraising methods can be
compared, and this can be used to plan
how resources should be allocated to
future fundraising.
39
basic
monitoring
charities
evaluation
services
40
Feedback from users

Even with a small organisation or project, it
is important to monitor whether people are
satisfied with what you are doing. You can
get a range of different information on views
and experiences of the service, including the
changes or new services people would like.
People can also tell you about how convenient
and accessible they find the service and
whether project publicity is helpful.
This level of feedback will not tell you what
has changed as a result of your activities. For
that, you will need information that relates
to outcome indicators. Feedback sheets after
a training course, for example, are a basic
tool to gauge user satisfaction. They are best
viewed as a monitoring tool, and are most
useful in providing information that will help
you to adjust your services. They will not
assess how well participants were able to use
the training you provided. You will need to
ask further questions at an appropriate time
to find this out.
Try to collect user feedback as an integral
part of your service delivery or other activities.
For example:
In a day centre you may ask children
informally what they like most about the
play session, but record the information
systematically.
Put a simple questionnaire in your
waiting area and ask people to fill it
in before leaving. You might ask the
following questions:
How did you hear about us?
What were your first impressions of
the project?
How long did you have to wait?
Is there any way we could improve
what we are doing?
Is there anything else we should
be doing?
How convenient is the location, times
available or access to the service?
Ask workshop or training participants to
fill in a questionnaire showing how
satisfied they were. Ask them to grade

aspects of the event using a scale. This
gives a range of possible replies to
questions, to help analyse information
and to collate it over a number of events.
Here are some questions you could ask:
How well did the workshop meet your
expectations?
How did you rate each session?
How appropriate were the training
methods used?
How good was the handout material?
How satisfactory were the venue
and the food?
During your planned contacts with users,
take the opportunity to ask consistent
questions about the quality of your activities.
Have a complaints and suggestions box
and keep a record of them as part of your
feedback data. Discuss complaints and
suggestions regularly and take action on
them.
Think about what methods you can use that
will be consistent with your values and that
will help your organisation to develop further.
You can get feedback from training, workshops
and other sessions by involving people more.
For example:
put up wallpaper or flip chart paper as
a graffiti wall for individual comments
throughout an event
ask people to write comments on post-it
notes and invite others to tick the
comments if they agree or to add their
own further comments
draw an evaluation wheel on which
participants tick different sections
representing different activities to show
what was most or least useful or enjoyed.
There is more information about scales

and about participatory methods in the
Practical toolkit.
charities
evaluation
services
Outcome monitoring
It helps accountability, and allows you to

describe achievements clearly.
Focusing on outcomes
Outcomes are the changes, benefits, learning or
other effects that happen as a result of your
activities.
Desired outcomes often describe positive
change. They can also be:
about maintenance of the current situation.
This involves ensuring things stay the same
and dont get worse.
about reduction in something, for
example, criminal behaviour or vandalism.
Outcomes can be:
By monitoring outcomes, you can gather and

report information regularly on the proportion
of your users or target group who changed or
benefited in some way. It could be, for example,
the proportion who were housed, accessed
education, stopped misusing drugs or gained in
confidence. Funders are increasingly asking
voluntary and community organisations to
provide this as well as information on their
outputs. Other benefits are that:
seeing changes over time increases staff
motivation and team work
identifying changes in themselves provides
encouragement for clients
welcome or unwelcome
expected or unexpected.
You may be able to anticipate most of your
outcomes. However, some things may happen
that you did not plan for.
it helps to identify strengths and areas for

improvement
it can help to build the organisations
reputation.
First steps in monitoring outcomes

Individual client outcomes usually describe
change in one of seven outcome areas:
circumstances
physical or psychological health
behaviour
attitude
self-perception
knowledge or skills
relationships.
Some organisations, for example second tier
organisations, may have target groups other
than individual clients. Some do not have
immediate users, for example, organisations
lobbying for environmental change or other
campaigning organisations. Organisations may
also intend to achieve change at a number of
levels, such as in other agencies, or at the
wider community or policy level. Community
outcomes might include the development of
residents groups, increased levels of recycling
or community arts activity.
Outcome monitoring has a number of benefits.
There are potentially a large number of

outcomes you could assess. Keep the number
of outcomes you will monitor to a minimum
by prioritising key outcomes and combine
outcome monitoring with existing systems
wherever possible. You have identified
outcome indicators as the clues that will tell
you if outcomes are achieved. Look at your
list of outcome indicators and identify those
for which you:
are already collecting information. You may
already be collecting outcomes information
in an informal or unsystematic way.
could easily collect the information within an
existing monitoring system, for example, by
rewording or adding questions
need to find a new way of collecting
information.
You may be able to combine questions relating
to outcome indicators in a questionnaire asking
users about how satisfied they are. Think about
formalising feedback from volunteers or from
partner agencies working with your users (for
example, a placement agency) so that you
address specific outcome questions. Think about
41
basic
monitoring
Feedback
from users
Outcome
monitoring
charities
evaluation
services
42
using the information you collect when you first

work with your target group as a formal baseline
and a template for information you will collect
again at a later time.
Large organisations may wish to set up an
outcome monitoring system that covers all their
services. A useful approach is to identify areas,
or categories of outcomes, common to the
whole organisation. Single outcomes that relate
to those outcomes can then be identified by
particular services.
Collecting outcome information

To assess change over time, you need
information on your outcome indicators at
two or more points in time. For client
outcomes, for example, this should ideally be:
as early as possible, usually when clients first
present at your service. This will provide you
with a starting point against which you can
assess change.
as late as possible, usually when clients leave
your service.
How frequently you monitor will be governed by
the nature of your work and your target group.
client diaries kept as part of their

engagement with the project
records kept by other agencies working
with the clients.
Case records of individual clients development
can be a useful tool for outcome monitoring,
because they provide both descriptive and
evaluative information. They offer an
opportunity to assess individual progress, for
example through the frequency of specific
behaviours or their progress in a care plan.
Be careful to respect confidentiality and
maintain anonymity if you use case records.
Also, unless you have thought about the use of
case records for monitoring and evaluation,
they may have limitations as an evaluation tool.
Different teams or individuals may keep
records differently or they may be incomplete
or otherwise insufficient. It may be helpful to
have a standardised case monitoring form. For
example, when you are recording data from
progress reviews and case conferences, it may
be possible to do this on a separate summary
sheet against specific outcome indicators so
that the data can be easily retrieved.
Processing outcome information

You can collect information on the same
outcome in more than one way, for example
through a combination of client self-assessment
and worker observation.1 However, questions at
initial monitoring must be repeated in the same
way at later monitoring stages, so that results
can be compared. It will very often be useful to
ensure that questionnaires and other monitoring
sheets providing individual outcome information
have an identity code, so that changes can be
tracked for individual users over time.
You may find that your organisation is already
collecting information relevant to your outcome
indicators. For example, regular attendance at an
employment project may be one indicator of
increased motivation to find work. You may have
various other sources of outcome information,
such as:
centrally compiled and filed anecdotal
information on client outcomes after
leaving the service
1
There is more information on these methods in the

section Practical toolkit, pages 116-123.
Small amounts of information may be stored

on paper and analysed manually. However,
many voluntary organisations now use
computers to manage outcome information.
You can type or scan information from
completed outcome monitoring questionnaires
or other forms into a computer and store it in a
database. If each form has a client code or
name, two or more forms completed by each
client can be placed in the same section and
compared by the analysis system used. With
information presented numerically, that is,
quantitative information, you can ask the
database to calculate the change from one
form to the next. It is then possible to produce
a total number of clients showing change in
each of the different outcome areas.
Qualitative information can also be stored in
the computer. Either type in clients answers in
full, or summarise them first. This can be done
using specialist computer packages, but many
people use Excel or Word.
charities
evaluation
services
Analysing monitoring data

Having decided what data to collect, and set up
monitoring systems to collect it, you now need
to make sense of the data, that is, to analyse it.
The questions you wanted answering dictated
the type and amount of data you collected.
These questions provide the framework for your
analysis.
Before starting to analyse your data, you need to
refer to the questions again, so that you relate
data to the questions. At this point you may
need to break them down into sub-questions,
as in the examples below.
Framework for analysing monitoring

information
Who is using our services?
Has the user profile changed?
Are there any under-represented or
over-represented groups?
How does the profile of our users
compare to those using other relevant
services?
It is useful to look for patterns. For example,

where has any increase in user numbers come
from? Is it from young people, or older people?
Does the increase include people from a
particular community or group? Which
organisations are you reaching, or which parts of
the city or country?
If you have numerical data, and want to make
comparisons between two groups of different
sizes, it may be helpful to show the information
as percentages. However, be careful of using
percentages when you are dealing with low
numbers, as this may give you a distorted
picture of your data.
Analysing outcome information

To help you analyse your outcome information,
try to think in terms of asking questions of the
information you have collected. What do you
want to find out about your work? For example:
What percentage of clients over the past
year stayed more than six months in their
tenancy?
What proportion showed an improvement
in independent living skills?
What is working well?

Which services or activities are
used most?
What do users value most?
What changes or benefits have been
achieved? Were there any unexpected
outcomes?
What factors have affected positive
changes?
How many clients have benefited and who
are they?
What problems have we encountered?
Which services or activities have
been delayed or not run as planned?
Which services are over used or
under used?
What are our waiting times?
What complaints and suggestions do we
have?
It is also possible to look at smaller groups of

clients by asking questions such as:
What percentage of those with substance
misuse issues at assessment then showed
some improvement by the review?
Of those with diagnosed mental health
problems, how many improved their ability
to manage their illness?
Reporting monitoring data

Monitoring data is usually reported to your
trustees or to your funder at regular points. This
is usually quarterly, but for funders may be
yearly. It will:
keep trustees and others fully informed
about how well the organisation is doing
against targets
prompt useful questions about how
far activities are achieving expected
benefits
43
basic
monitoring
Analysing
monitoring
data
Reporting
monitoring
data
charities
evaluation
services
44
prompt corrective action before it is
too late.
Keep your life simple by collating information
regularly, for example, weekly or monthly, to
avoid too much work at one time. Design
simple reporting forms so that you can easily
demonstrate progress against targets. Your
basic point of reference will be the year plan.
If you have set out milestones, or key events
or achievements for the year, it will be
useful to report progress against these.
For example, you may have set out a
timetable for sending in a major funding
application, moving premises or holding
a conference.
From the performance indicators you
identified for your outputs, select key
indicators to report on, for example, number
of sessions, meetings, publications, profile of
users or other target group. Set these out in
a way that makes it easy to compare against
the targets you set for your key indicators.
Some of these may be performance levels or
standards agreed with your funder.
You may report on satisfaction feedback data
during your quarterly reporting. Some
funders also ask for some basic outcome
monitoring data every quarter, such as
numbers of people housed.
Further reading
Parkinson, D and
Wadia, A (2008)
Keeping on Track: A
Guide to Setting
and Using
Indicators,
Performance Hub,
Services, London
There are a number of points to remember

when presenting your data. For example,
make sure you distinguish between new and
repeat users when you collect and report on
take-up of services. Break down the data by
age, gender, ethnic group or other relevant
category to ensure that different levels and
types of participation in the project and
differences in benefits can be seen clearly.
Remember also to look at monitoring data
against your original plans and timetable.
Keep monitoring data over a period of time,

so that you can report on any trends that
emerge. For example, you may have made
changes to the times your service opens
and want to see whether this had the result
you expected. Outside factors, such as a
changed policy or a new service in the area,
may also affect trends. Using a consistent
reporting format over time will help you to
make comparisons. However, remember
that the user profile may have changed over
time, and this may affect trends or your
ability to compare.
Monitoring your
monitoring
Monitoring arrangements themselves need
to be monitored. Make sure they give you
the information you need and are not too
time consuming:
Keep a note of any problems as they
arise and the main ways monitoring
has been helpful.
From time to time do a more systematic
review. To start with, this should be
every six months. Later, it could be once
a year.
Make sure that you are not collecting
information that you do not use.
This section has concentrated on how
to set up basic monitoring systems on
different aspects of your projects
activities, including monitoring against
targets. It has discussed the monitoring
implications of focusing on project and
organisational outcomes. It has also
looked at the need for a framework to
analyse and report monitoring
information. You now need to think
about how monitoring links to
evaluation activities.
further monitoring
Basic monitoring examined how to set up
basic systems to monitor project activities
and outcomes and gather data about users. It
also looked at how to report monitoring
information within an analytical framework.
Further monitoring looks at more complex
monitoring. This includes monitoring project
impacts, and how to monitor against a
quality system. It also looks at collecting
routine data about factors within your
project, or in the outside world, that may
affect how well the project is doing.
Developing more complex

monitoring systems
You may need to extend your basic
monitoring system to collect qualitative
information to help you manage your
services routinely, or to collect information
needed later on for evaluation purposes.
Setting up a more complex system will
depend on what other monitoring questions
you might have:
Is service delivery consistent with the
project design and plans?
What are the projects cumulative or
longer-term effects?
Do services meet user needs?
What resources have been used and
how much does service delivery cost?
Is service delivery consistent with
quality standards?
As well as staff from your own project, staff
from other agencies will be a useful source of
monitoring information. It may be difficult to
get commitment to this, so discuss with your
project partners from the beginning:
what information each of you will collect
why you need it and how it will be used
the format for collecting the information.
If you receive face-to-face or informal feedback,

for example from busy teachers or social
workers, make sure that you record their
comments during the discussion or interview.
Process monitoring
To understand why your project is achieving
the changes it hopes to bring about, you need
to identify the factors that really matter in
the way you do things that is, the projects
processes. This may include looking at the
way you deliver certain outputs, such as client
assessments. But there will be other things you
want information about, such as the way staff
and volunteers work together, recruitment
processes, public relations and relationships
with other agencies. Process monitoring is also
important to assess the quality of what you do.
It can be helpful to set specific indicators for
processes. Examples might be:
the extent of involvement of group members
the amount of team working
the extent and type of meetings with
other agencies
the level and quality of contact with
the media.
Also allow for more wide-ranging and
unexpected data to be collected, for example,
through records, minutes and diaries.
Appreciative enquiry is an approach to get

information from participants about their
personal experience of projects by asking
specifically about:
what they like or value most
what pleased them most about this
aspect of the project and their part in it
what else could be done to develop
these positive features.
The key is to find out as much as possible
about the positive aspects of the project,
and to build on these.
charities
evaluation
services
45
further
monitoring
Monitoring your
monitoring
Developing
more complex
monitoring
systems
Process
monitoring
charities
evaluation
services
46
Monitoring for quality

It is not always easy to make judgements
about service quality. Projects sometimes use
simple quality indicators, for example:
waiting times for appointments
number and type of complaints
response time to information queries.
There are a number of other ways to
monitor quality. For example:
using accepted performance indicators,
such as client-practitioner ratios
observing interaction with users, such as
training sessions or group work
checking case recording methods,
for example, against agreed practice
standards
using someone who is not known to staff
to experience the service first hand.
If you work within the framework of a

quality system, such as PQASSO (Practical
Quality Assurance System for Small
Organisations), you will have created
standards. That is, you will have agreed
levels or expectations, across a range of
organisational activities, including
management and service delivery. Most
systems will suggest what evidence you will
need to show how well you are doing. Your
monitoring system should provide
information that allows you to check
whether you are meeting those standards.
In the extract below, the quality system
suggests the monitoring information that
will allow you to make an assessment
against the quality criterion.
A quality system will have its own cycle of
review. Monitoring and evaluation activities
should be planned so that they produce the
information you need for your quality
reviews.
Quality area: User-centred service

Criterion: Users are more involved
in the organisation. User feedback
leads to changes in service delivery
Monitoring information required
A range of methods is used to seek

user views, and the views are used to
influence decisions on services
User satisfaction surveys, minutes of user

forums, focus group reports
Active steps are taken to understand the

issues that affect users and potential users,
particularly under-represented groups
Research reports, government strategy and

relevant policy documents
Complaints and suggestions are discussed

openly and dealt with quickly
Records of complaints, suggestions and

actions taken
Adapted from: Charities Evaluation Services (2000) PQASSO (Practical Quality Assurance
System for Small Organisations), Second edition, London
charities
evaluation
services
The basic monitoring and evaluation cycle

can be adapted to show how monitoring
feeds back into quality improvements.
lity im
Qua
provem
e
ity review
Qual
nts
Monitoring
Sta
n
dard
develop
n
me
Impact monitoring
Impact monitoring examines the effect of
your project over a longer timeframe and in
a wider context. It asks about the broader,
cumulative consequences of your work.
You will need to take measurements over
several years and use outside information to
assess the effect of your project from a
longer-term perspective. Examples of
statistical information relevant to a local
project might be:
the number of school exclusions
the number of arrests in a particular
neighbourhood
example, a sexual health education project

may result in more people having a sexually
transmitted infection diagnosed. This could
demonstrate success, that is, increased
awareness of the importance of sexual health
check-ups, rather than indicate increased
rates of infection.
It is worth remembering that a number of
factors ranging from socio-economic
trends, government policy and other local
initiatives may all influence impact
measurements. It will be difficult to gauge
the part played by a small project within the
larger context.
Monitoring context issues

Decide what the important issues are that
will affect your projects performance.
These may be at different levels, for example:
central government policy, such as
regulations affecting asylum seekers
local government policy, such as
housing or environmental policies
services provided by other agencies,
including other voluntary organisations
factors within the project, including
staffing and financial resource levels.
Be realistic about what you can monitor.
Collect information that you need to run your
project more effectively anyway.
You can do this through:
newspaper cuttings and other articles
seminars and conferences
noise pollution levels.

meetings with policy-makers
A positive impact from certain services or
activities may result in an increase or
decrease in the statistical measures. For
networking and inter-agency meetings

and discussions.
47
further
monitoring
Monitoring
for quality
Impact
monitoring
Monitoring
context issues
charities
evaluation
services
48
The key is to record this information

systematically at the time, and in a way that
will be accessible and useful to people
reporting on the project and conducting
evaluations.
Data management
It is important to resolve practical data
management difficulties, as these can hold
back how effectively you can use monitoring
information to improve your work and the
quality of your reports to trustees and
funders. It may be helpful to involve an
external consultant in identifying internal
information needs as well as external
requirements. Research has shown that many
organisations struggle with IT resources that
are inadequate to meet reporting
requirements; this lack may be in IT
infrastructure and systems and in staff
training and capacity.
There is an increasing development of IT
software specifically for third sector
monitoring and evaluation; investment in
improved IT systems can have a number of
advantages, including:
considerable time savings
increased storage capacity
easier data search and access
easier checking for errors and missing data
Further reading
Using IT to Improve
Your Monitoring and
Evaluation,
Performance Hub,
Services, London
The Charities
Evaluation Services
website has
information on
monitoring and
evaluation software:
www.ces-vol.org.uk
more consistent monitoring information

delivery of well-structured reports
ability to meet a range of reporting needs.
Take time early on to decide what you need
the system to do. Be clear about whether you
want a stand-alone monitoring and reporting
system or whether you want it to support
your work with clients or to support other
functions, such as fundraising. It may be wise
to start with a simple system and to add
more complex features as you go along.
The introduction of a system may have
unexpected costs and implications. To

minimise these, it may be helpful to bear a
number of points in mind:
Understand your current difficulties with
storing, managing and analysing
information and assess how any given
system will address those problems.
Put the development of a monitoring
system within the framework of an
overall IT strategy.
Plan to involve time and effort from all
sections of the organisation.
Get good management support and,
ideally, a team of people with the
knowledge, skills and motivation to lead
on the work.
Involve people in the process who may
be reluctant to move to new ways of
managing data or be IT resistant.
Provide a regular flow of information
about the progress of the project and
keep everyone on board.
Remember that you may be able to get
funding support, so write a proposal that
describes how your IT system will make you
more effective and efficient and how it will
help more beneficiaries. And once you have
an effective IT system in place, make it a
feature in your applications for funding,
identifying how it will help you report on
your achievements
You can gather a range of monitoring
data during the lifetime of your project,
including data on the quality of your
management or services, and on the
changes resulting from your activities.
However, you cannot monitor everything.
Think carefully first about what you and
your stakeholders most need to know and
carry out a systematic review of the data
you collect. An important part of this is
thinking through your evaluation
questions, because your monitoring data
will feed into your evaluation. This is
discussed in the next section on Evaluation.
section 3 evaluation
Basic evaluation
51
Making judgements
51
The politics of evaluation
52
Understanding stakeholders needs
53
Outcome evaluation
53
Impact evaluation
56
Finding the evidence
56
Involving users in evaluation
59
Evaluation standards and code of ethics
59
Writing the evaluation report
60
Further evaluation
65
Focusing your evaluation
65
Evaluation approaches
67
Designing an outcome evaluation
70
Extending your evaluation activities
71
Managing evaluation
74
charities
evaluation
services
49
charities
evaluation
services
basic evaluation
Basic evaluation considers what makes
evaluation different from monitoring,
and how the two link together. It discusses
the need to take into account different
expectations and agendas and to work within
ethical standards. The section introduces the
process of data collection and looks at the
important step of interpreting evaluation
findings and at writing an evaluation report.
Making judgements
During the year you produce monthly or
quarterly monitoring reports for colleagues
and trustees, and use the data for funding
applications and publicity.
Your quarterly reports may lead you to
think about how you are delivering services,
for example, how you can best respond
to advice needs, or how you can improve the
accessibility of your services given
your user profile information. However,
an evaluation will allow you to examine
specific issues in greater depth. It gives
you the opportunity to bring together,
analyse and make judgements and
recommendations from the monitoring
data collected during the year.
Your monitoring information is likely to
contain:
profile information on your users
basic project record keeping, such as the
minutes of meetings and case records
statistical information on take-up of
services
feedback sheets from training courses
and workshops
diaries and other records of events
complaints and compliments from users.
As part of your evaluation, you may decide
you need to know more than this. Your
monitoring information will probably suggest
further questions that need an answer, such
as why certain activities were well attended

and others not, or why an inter-agency
network collapsed. You need to think clearly
about where the focus of the evaluation will
be and who and where you want to obtain
information from.
charities
evaluation
services
51
basic
evaluation
Making
judgements
Your focus may be on how well you are

meeting your objectives (that is, your
intended activities), and whether your
objectives are in fact helping you to achieve
your aims (the changes you intend to bring
about). These are often the key issues that
funders want projects to address in an end-ofyear report. By putting together your
monitoring data you should see an overall
picture and patterns for particular time
periods or activities. This allows you to make
comparisons with plans and targets and with
previous years. The data you collected will
also give you information about how users
and partners value your project.
Remember the importance of planning for
your evaluation. Think carefully about what
you want to find out from your respondents.
You should also decide whether you are able
to, or wish to, collect information from all
your members or users, or only a selected
number, that is, a sample. How you draw your
sample is important, because gender, age,
ethnicity, economic status and many other
factors can have an effect on outcomes for
individuals. Be aware that participants who
volunteer to be involved in the evaluation may
feel more positive about the project than
those who do not, and this may affect your
findings. Think also about how you can get
information from people who no longer use
the project, as they may have different and
valuable perspectives.
Make sure you set enough time aside for
this additional information gathering.
Questionnaires take time to develop, and
should be tested with a small sample from
your target group to see if they will capture
the information you want. Interviews take
time to organise and even longer to write
up and analyse.
charities
evaluation
services
52
Evaluation is a set of interlinked activities.

Each of these is an important part of the
overall process and needs adequate time built
in to protect the quality of the evaluation.
Conducting an evaluation
Planning
Design
Data collection
funding decision or a policy decision is

decided before findings are reported.
The project may be experienced differently by
different sections of its user groups. This may
be along the lines of gender or ethnicity, and
external agencies who are users often have
their own considerations. It is the evaluators
job to understand and bring out these
differences from the beginning: the evaluator
will want to make the differences clear in
order to take account of them properly.
Data analysis
Interpretation
Reporting
The politics of evaluation

Different stakeholders come into an
evaluation with different purposes,
perspectives and values. For example,
decisions to evaluate may be linked to
funding decisions or a policy agenda.
This may apply to self-evaluation as
well as to an external evaluation.
Evaluation may encounter a number of
difficulties, particularly if the reasons for it
are not clear to everyone. Here are some
examples:
Managers may want to carry out an
evaluation simply to make the project
look good. A project monitoring and
evaluation officer might find their
position undermined by a project
director if an evaluation brings up
negative findings.
Changes in senior management may
lead to different priorities, which
might challenge the agenda set for
either self-evaluation or for an external
evaluation already underway.
The evaluation may lose its relevance if a
The role of the evaluator or the

evaluation team is vital. You should:
make sure that stakeholders feel a sense
of ownership by consulting with them
and, where appropriate, sharing the
process of the evaluation as well as
its results
use the evaluation to encourage a process
of reflection throughout the project
recognise the pressures of daily project
activities while monitoring and evaluating
be receptive to other peoples ideas and
responses while remaining independent
keep in mind the evaluation aim of
learning.
Managers should talk through with staff
any fears they have about evaluation. It is
particularly important for evaluators to
work closely with project management, so
managers do not become defensive about
any negative findings, and reject or shelve the
report. You can help to reduce any concerns
among project staff about evaluation by
including staff on a steering committee
or evaluation task group. The steering
committee can help set the terms of the
evaluation and keep staff in close touch
with the process and the emerging findings
throughout. This may also increase the
likelihood that recommendations will be
accepted and implemented.
charities
evaluation
services
Understanding
stakeholders needs
Just as you consulted your stakeholders in
setting aims, objectives and indicators of
success, make sure you plan enough time
for consultation with managers, users,
funders and staff about their monitoring
and evaluation needs.
You do not always need to involve everyone,
but try to consult with representatives from
different stakeholder groups as they will have
different, sometimes contradictory, interests
and views. Although the performance
indicators you set will guide some of your
evaluation activities, there may be other
areas of your work that need exploring, and
other questions to be answered. Find out:
What are the main questions they want
monitoring and evaluation to answer?
What information do they lack about
the project?
How do they want to be involved?
How do they want the findings
presented?
Involving funders and partners at this stage
can also help to build relationships and can
lead to a greater likelihood of the evaluation
report being understood and accepted.
Consulting users can also encourage user
participation in the evaluation itself and in
developing data collection tools, such as
questionnaires.
Deciding what questions need to be
answered is the first important step. For
example, they might be:
Are we reaching the right target group?
Do we have the right sort of project,

or staffing, to do the work?
Collect only the information you need to
answer your evaluation questions, information
that you can use, and that is not already
collected elsewhere. If your monitoring has
focused on what you are doing the project
processes and outputs evaluation may
be the point at which you need to collect
information on the differences this has made
that is, project outcomes.
If you have been monitoring outcome
information as well, evaluation will allow you
to review it in context. You may also find it
useful to carry out some additional one-off
data collection, such as interviewing users or
partner agengies.
Outcome evaluation
We have defined outcomes as the changes,
benefits, learning and other effects that you can
attribute to your organisational or project
activities. They are different from the longer-term
change or impact of the project or programme.
Impacts relate to the overall aim or mission. They
are the broader or cumulative effects that may
be seen in local or national statistics about the
area of intervention, such as health,
homelessness, school attendance and youth
unemployment.
If you set indicators for outcomes, you might
have been able to monitor some of these
outcomes regularly. During the evaluation stage
you should be able to draw this information
together. Useful outcome information can also be
obtained during the evaluation itself through
one-off data collection. The task then is to put
outcome information in the context of:
Are we doing what we set out to do?

your services and working methods
Are we meeting users needs?
what you know about your users
Are our services effective?
external factors.
What has worked well, and what
has worked less well?
You can then make judgements about what has
53
basic
evaluation
The politics
of evaluation
Understanding
stakeholders
needs
Outcome
evaluation
charities
evaluation
services
54
been achieved. This will involve comparison

against:
your own previous experience,
expectations, standards, or defined targets
results from similar activity or previous
local results
expected outcomes and outcome levels
defined by other stakeholders. You may
have defined these standards with your
funders and users, for example.
This is also a good opportunity to pick up on
the unintended effects of activities. Projects
often produce outcomes that were not
expected, as well as planned changes, so it is
important that you collect information in a
way that will tell you about these.
Outcomes that are achieved in the short term,
but link to or lead to a final longer-term
outcome, are called intermediate outcomes.
Be clear, particularly with funders, about what
can be measured at any given stage. At an
early point, or a pilot phase, you may be able
to measure only intermediate outcomes, for
example increased knowledge or skills,
changed levels of confidence, rather than
longer-term outcomes, such as take-up of jobs,
which might fulfil the project aims more
completely. Also be clear about the change
your project is able to achieve and what might
lie with other agencies. Once you are clear
about what change is realistic, you will be in a
better position to negotiate on this with
stakeholders.
Longer-term outcomes may often be less easy
to evaluate than intermediate ones. For
example, in a family centre working with
young men, it will be difficult to assess their
ability to sustain responsible relationships in
the short term. However, it may be possible to
evaluate whether the young mens self-esteem
increased, or their awareness of responsibilities
or ability to manage anger. Some projects
provide a service which is a link in a chain of
services, or part of a multi-agency
intervention. For these projects, intermediate
outcomes may be the final outcomes assessed
as these are the only ones the project is involved

with.
Recognising intermediate outcomes can help
identify and acknowledge the different
contributions of separate agencies. For example,
a homelessness organisation may need to
consider the context in which they are working: a
contact and assessment team may have engaged
the client; a day centre may have built up their
skills and confidence; a resettlement team may
find them a home; and a tenancy sustainment
team may help them to keep the tenancy.1
Understanding outcomes
You can look for outcomes at a number of levels.
When you set indicators remember to set them
at different levels, if appropriate. For example,
these may be:
individual
family
community
environment
organisational or institutional
systems and policy.
You may be focusing your monitoring at one
level, but it could provide a more complete
picture of your project if you collect information
on outcomes at other levels during your
evaluation. For example, a project developing a
green space might monitor the changes in the
environment on a regular basis, and during an
evaluation collect information on the resulting
outcomes for the community as well. A project
working in schools might collect regular data on
school attendance and test results, and as part
of an evaluation also collect data from families
and teachers on the difference the project has
made to them. Examples of these outcomes
might include fewer conflict episodes or
increased contact between school and family.
For further discussion on outcomes for homelessness

organisations, see Burns, S and Cupitt, S (2003)
Managing outcomes: a guide for homelessness
organisations, Charities Evaluation Services, London.
charities
evaluation
services
Outcomes at an organisational level are also

important. Some of the questions to ask are:
How does the presence of the project
create changes in the agency which
houses it, or with which it is working?
Is collaboration among institutions
strengthened?
It is helpful to be clear how individual outcomes

are related to and lead on from outcomes at a
policy or systems level. For example, a
programme aimed at improving the welfare of
older people in residential care may need to
focus on changing procedures and bringing
about more participation by older people.
Have staff become more skilled?

Individual projects may have outcomes at a
policy level, although this is more usual with
larger programmes.
An external evaluation was carried out on a programme
whose overall aim was to prevent homelessness in
London. The projects funded within the programme
were intended to have effects on at least four different
levels: with the individuals at risk, with their families,
on the local community and on government policy.
Levels of intervention
The programme was able to define the expected

outcomes for each level. Clarifying expected outcomes
at the early stages of projects can reveal unrealistic,
vague or conflicting aims and is vital to effective
planning.
Expected outcomes
Policies to prevent
homelessness
Government policy
Increased acceptance
and co-operation
Local community
Change in attitude
More young people
reunited with their families
Families
Increased self-esteem
Confidence and
change in behaviour
Individuals
Be clear about the type of change you are intending, and for whom, as shown in the table below.
Change
In what
For whom
Develop
Expand
Maintain
Modify
Improve
Attitude
Knowledge
Condition
Perceptions
Policy
Individual
Family
Neighbourhood
Other agencies
Local government
Adapted from: Evaluation Forum, The (1999) Outcomes for Success,

Organizational Research Services Inc and Clegg and Associates Inc, Seattle, WA
55
basic
evaluation
charities
evaluation
services
56
Impact evaluation
The term impact is used in a number of
ways, but is usually taken to mean the effect
of a project or programme at a higher or
broader level, cumulative effects, or changes
that affect a wider group than the original
target. Impact often relates to the overall aim
or mission of your project. Impact is
frequently longer term and therefore the
timescale needed for impact evaluation may
well be longer than that needed for outcome
evaluation.
One area of overlap between outcomes
and impact arises when an organisation is
directly involved in policy work, education
or awareness-raising at community level, or
other broader work. Then:
Policy change or community awareness
would be defined as intended outcomes of
the organisation or work taken.
Impact could be the effect on a wider target
group of any policy change or increased
awareness.
Example of outcomes and impact:

homelessness agency
A homelessness agency works with homeless
people in a number of ways. It also
campaigns or lobbies to change the local
authoritys approach to who it defines as
homeless and their responsibility to house.
The agency would have outcomes for:
the individuals they work with directly
possibly those individuals families
the local authoritys policy on
homelessness.
They could also have an impact by:
increasing the number of people housed
by the local authority (the effect of the
policy change).
Impact is generally achieved through the

efforts or effects of a number of
interventions. So you need to think carefully
about the role of your project in relation to
these long-term changes and place your
projects activities within a broader picture.

Even if you are able to follow up individual
users later, to assess longer-term changes, it
may be difficult to analyse clearly the role
of the project in their successfully finding
and keeping jobs or housing, reducing their
alcohol dependency or maintaining family
relationships. Many other factors within
their personal circumstances and the wider
environment influence their behaviour and
potential.
Similarly, wider statistical information about
an overall aim for example on crime,
homelessness, teenage pregnancy, recycled
material could reflect the impact of a
programme or group of projects. But it will
also reflect other social and economic
factors and interventions, either negative or
positive. It may be relevant to get
information about the wider policy change
that relates to the area in which your
project is working and your target group, or
the area in which you are campaigning. This
will be useful data in which to place the
more immediate outcomes from your
project.
Finding the evidence

Data collection methods
How you collect evaluation data, and what
you choose to collect, is crucial to your
findings. There are many methods that you
can use. These are some of them.
Using documentary sources, such as:
existing project information sources
public documents
personal files or case notes
existing databases.
Collecting data directly from individuals,
such as:
diaries and anecdotal accounts
questionnaires
interviews
charities
evaluation
services
written responses to requests for

information
tests
samples of work
participatory data collection methods,
with people expressing themselves
creatively, often visually.
You may be able to test whether there is a

statistical relationship between an activity
and some measured change.
Qualitative data tells you about personal

reactions, feelings, impressions and processes.
It may be based on verbal or written records,
and extracts from letters, questionnaires and
minutes of meetings.
Using an independent observer

Data may be collected through:
written accounts
observation forms
mystery shopping, where the evaluator
experiences the service as a user would.
Audio-visual data collection methods
audiotape
videotape
photographs
In some cases data from open-ended

questions can be transformed into
quantitative data. This can be done by
grouping statements or themes into larger
broad categories and giving them a numerical
value. For example: 26% of users made some
negative comment about the waiting time,
while 15% were unhappy about the adequacy
of the waiting area, referring to the space
and seating available, and inadequate
provision of childrens toys. Be aware that
such quantification of qualitative data may
lose the essence of the personal views and
experiences recorded.
digital media.
Adapted from: Worthen, BR and Sanders, JR (1987)
Education Evaluation: Alternative Approaches and
Practical Guidelines, Longman.
It is useful to make a distinction between

primary and secondary data collection.
Primary data is data collected, for example,

through questionnaires or interviews
designed specifically for the evaluation study.
Distinctions between quantitative and

qualitative approaches are not absolute and
most evaluations collect both quantitative
and qualitative data. Be careful that a focus
on quantity does not obscure an
understanding of how people experience
project activities and benefits. Different
approaches, methods and types of data are
not in opposition but often complement each
other.
Planning data collection

Secondary data is data collected by other
organisations or researchers for their own
purposes that can provide a useful source
of information for the evaluation. This may
be, for example, local population statistics, or
a report on user experiences in other, similar
projects.
Make sure you plan enough time for

preparation before you collect your data.
You will need to:
decide which methods to use
prepare the data-collecting instruments
pilot them
Quantitative data deals with numbers, asking

questions such as: how much?, how many?
and how often? Things are either measured
or counted. It allows you to compare sets of
numbers and look at their distribution. It is
useful when you want accurate, precise data.
train data collectors

discuss evaluation methods with staff to
reassure them that the evaluation will not
unduly disrupt their work or project users.
57
basic
evaluation
Impact
evaluation
Finding the
evidence
charities
evaluation
services
58
Piloting
Before starting any data collection, it is a
good investment of time to test the datacollecting instruments. For example, ask
some project participants to fill in a
questionnaire or carry out interviews with a
small number of respondents. Then look at
the results to see if the questions have been
understood consistently by respondents and
if you have captured the information you
wanted.
Piloting is particularly important, for example,
with a survey questionnaire or other selfcompletion instruments, where a researcher
is not on hand to clarify questions.
Training
It is important that people carrying out
evaluation research are properly briefed and
trained. Data collection should be supervised
to ensure quality, for example, by comparing
data entry records for different researchers
and reviewing work regularly.
What venue will be most suitable?

Deal with any issues of distrust. Many
individuals and groups may be reluctant to
criticise a service, particularly if there are no
alternatives open to them. This may mean you
do not get a true representation of user
views. You may wish to involve community
leaders and other community members when
planning evaluation. Be sensitive to cultural
norms when choosing data collection
methods.
Consider having a focus group of a specific
ethnic minority community to allow
particular views to emerge and be heard.
Remember to consider access for people with
physical disabilities and be prepared to visit
people in their homes. You may need
someone to sign, or to prepare an audio
cassette of your questionnaire. For people
with learning disabilities, it can help to use
pictures instead of, or as well as, written
1
questions.
Diversity and access

The evaluation design should take diversity
into account. When designing the evaluation,
be clear about differences between groups
and recognise the potentially different
viewpoints of minority groups, and how these
differences might affect the way they
experience the project. Think about how
gender, age, disability, language and ethnicity
may affect the ability of different groups to
take part in the evaluation:
Which tools are most suitable for
which groups?
When would it be most convenient
for people to meet?
There is more guidance on focus groups in

the Practical toolkit.
Incentives
Evaluators sometimes offer a small token to
participants if they are asked to give up a
substantial amount of time. Ask people
working directly with your informants what
they would consider appropriate: a voucher
can be given as a token payment. You also
need to budget for travel costs, refreshments
for a focus or discussion group, and possibly
for child care costs.
Would people prefer to meet with

you alone or with someone else?
Is an interpreter necessary?
What type of evaluator or interviewer will
put your respondent group most at ease?
In some situations, for example, you may
wish to consider a same-sex evaluator.
People Firsts Access2Pictures (at 020 7820 6655)

and the CHANGE picture bank
(www.changepeople.co.uk) are designed
to make information easier to understand. Both are
available on CD-Rom.
charities
evaluation
services
Involving users in evaluation

It is increasingly expected that users will be
involved in evaluation, and this is sometimes
requested by funders. Evaluation findings that
users have contributed to could have a
greater impact on policy-makers. Indeed,
there are a number of good reasons to
involve users in evaluation:
Users may have been involved in the
design of services.
Involvement will make use of the
expertise that users may have in
relation to their own needs.
Users could gain a greater understanding
of the project as a whole.
It could help to meet the projects aims
of building capacity and confidence in
the users and lead to greater user
involvement in the project.
It will create a sense of ownership in
the evaluation.
On the other hand, it is important not to
assume that users want to be involved.
There are potential difficulties or limitations.
For example, there are issues of power. Users
may find it difficult to talk with confidence
to professionals or staff members who they
see as having more authority or influence.
In practice, only a small group of users may
become involved, usually the most vocal and
least vulnerable. Finally, there are practical
factors to consider, such as confidentiality
and how to select the users to be involved.
You should also weigh up the advantage of
user involvement with organisational needs
and the focus of the evaluation, and the
need to convince other stakeholders about
its credibility.
Be clear with users about what commitment
is involved. It is also important to be aware of
the diversity within groups: there is no single
user view, no representative user.
Remember, if you are working with users
you need to:
plan in enough time it may take much

more time than any other approach
provide proper training and support
59
basic
evaluation
encourage teamwork.
Evaluation standards and

code of ethics
In a self-evaluation, as for an external evaluation,
get agreement within the project about its
purpose, remit and value. You should also:
make sure that everyone involved in the
evaluation is in agreement with the process
from the start
choose methods that are suitable for the
projects resources and working practices
agree whether and how findings will be
made public, and when data should be
kept confidential to the project.
At the outset, think about the ethical issues
raised by the evaluation. Give evaluation
respondents a written introduction to the
evaluation setting out its purpose, the time
it will take to be involved, and how it might
affect them. They should understand that they
do not have to take part if they do not want to,
and know what risks, if any, are involved in
taking part in the study. You should explain
how information will be used, give assurances
about confidentiality, confirm that no personal
consequences will result from the evaluation,
and that individuals and their replies will not be
associated in the report without consent.
It may be helpful for staff or volunteers to
ask users if they would be happy for their
contact details to be passed to an evaluator
for a telephone or one-to-one interview.
This will be an opportunity to reassure them
and to answer questions about the evaluation
process and how findings will be used.
Where appropriate, check your interpretation
of data with informants. Offer to give them
feedback on what is learnt. If personal
feedback or a workshop is too time
consuming, you could let them have a copy of
Involving
users in
evaluation
Evaluation
standards and
code of ethics
charities
evaluation
services
60
the report or executive summary instead.

Think about:
face-to-face explanations for participants
with limited literacy skills
explanations in their first language for
participants with limited English skills
working together with advocates,
supporters or carers to make sure that
people with learning or other difficulties
have enough information about the
evaluation.
Decide how long you will keep data and
make sure it is held securely. You should
be fully aware of, and take into account,
the projects confidentiality policy. When
1
collecting information from a minor:
remember to get consent from the
parent or guardian
find out the consent policies of agencies
involved, such as schools and social
services departments.
If you do not pay enough attention to these

key stages, you risk wasting the time you have
already invested. A good report is vital.
Interpreting findings
So far you have gathered your monitoring
data over the year and collected some
additional data. You can find information on
analysing data in the Practical toolkit. The
next stage is possibly at the heart of your
evaluation activities. You must think about
what lies behind your data, that is, interpret it.
Interpretation means looking beyond the data
itself and asking what the results mean in
relation to your evaluation questions. Be wary
of assuming that there are links of cause and
effect between your project activities and
results. Involve other people in this level of
interpretation and, where appropriate,
acknowledge in your report the possibility of
other interpretations. Remember to put
outcome information into context. To do this,
it will be useful to ask a number of questions:
Did the level of resources, for example, the
level of money or staffing, affect the
outcome?
There is more guidance on working with

children in the Practical toolkit.
Writing the evaluation

report
You should have the basis of a useful evaluation
if you planned and focused your evaluation
carefully, if you have the right skills, if you chose
an appropriate mix of methods to collect the
data, and if you analysed it carefully.
However, there are two further important
stages:
interpreting your findings
reporting.
1
You will need permission from the local authority before

interviewing children under 18 in care and from a parent
or guardian for children under 16 not in care.
Was the outcome affected by the way you

delivered your service?
Did external factors, such as local policy,
prevent you from achieving your outcome?
Example of alternative interpretations
Children at a play centre are becoming
more sociable and creative in their play.
Is this down to the skills of the play worker
and the activities? Or is it partly because
many of the children are beginning to play
with each other in their family homes as
their parents get to know each other?
Consider whether enough time has been
allowed in the project for various things
to happen. Were your assumptions, for
example about co-operation with other
agencies, borne out? Is there a difference
in results that relates to the type of user,
or to different ways of working with users?
charities
evaluation
services
When you ask different people the same

question, do you get different answers?
If so, why is that?
It is not always easy to get a sense of the
significance of different findings when you
first look at the data you may lose the
most important point in studying and
presenting the detail, or overemphasise a
minor finding. So try to stand back from
your findings and look at the broader picture
before finalising your report.
Think about how your own biases and
interests may affect the data you collect and
how you analyse it. For example, have
competitive or other reasons led to failure to
report appropriately on services provided by
another agency?
Drawing conclusions
You are now ready to tell your stakeholders
what you have found, and must draw
conclusions from your findings. Conclusions do
not just repeat the data, but should link clearly
to the evidence presented. Avoid generalising
from a particular finding and make clear in
your report the difference between facts,
respondents views and the evaluators
interpretation.
If your report is short, you might summarise
your conclusions at the end of the findings
section. In a longer report, summarise your
conclusions on one section of your findings
before reporting on the next topic.
One approach to finalising the report is to
organise discussion groups on the draft report.
Think of this as your final stage of data
gathering, as you may be given new insights
and different interpretations. You may also use
these discussions to suggest recommendations
arising from the findings.
This process will improve the report and allow
others to feel involved in the evaluation
findings. It is more likely that
recommendations developed in this way will
be acted on. Make the distinction between
data and interpretation clear. If differences of

opinion arise, make sure that the evidence is
reported clearly in the report, and offer
alternative explanations.
61
basic
evaluation
Writing your recommendations

Recommendations are based on the conclusions.
An evaluation need not always lead to
recommendations, and be wary of making
recommendations if you do not have enough
evidence. It is helpful to be clear about where
you are able to propose a course of action and
where further information or consideration may
be necessary. Be careful not to mix findings or
conclusions with recommendations, although
recommendations should follow clearly from the
findings.
Make recommendations in a way that will
increase the chances of action being followed
through. Be clear about the most important
recommendations and those that are less
significant. If possible, give a timescale and say
how the recommendations should be
implemented.
Group recommendations together in a logical
way and make it clear who should act on the
recommendation, for example, your trustees or a
partner agency. Recommendations may be about
policy, staffing, the target group, service quality,
activities, and monitoring and evaluation; you can
use sub-headings to separate them. It may also
be sensible to group together recommendations
that are for immediate action and those that are
longer term. Recommendations should be
specific, realistic and achievable.
Presenting findings
How you present the report will have an
important bearing on your credibility, so consider
what your evaluation report will look like before
you start putting together your findings. Think
first about the purpose of the evaluation and its
audiences. The style of reporting and level of
detail will vary according to your audience. Will
your audience prefer evidence in the form of
tables, charts and graphs, or case examples?
Writing the
evaluation
report
charities
evaluation
services
Most reports will cover the following:
62
an introduction, which may include

the projects aims and objectives
the aims and objectives of the
evaluation, including the main
evaluation questions
how the evaluation was carried out
findings
conclusions and recommendations.
It is also useful to produce a brief summary,
placed at the front of the document, for
people who will not read the whole report.
You can find a more detailed example of a

report outline in the Practical toolkit.
First, prepare an outline of the report and

the points you want to cover in the findings
section. Discuss with others the outline for
the findings section and the data you will
draw on. You will find it easier to report your
findings if you write a structure for this
section.
Then, gather all your data together, including
previous evaluation reports, and do any
analysis that is needed. Remember that you
may need to bring data from several sources
together. If it is contradictory, you can make
this clear in the report.
Put together a framework based on your

evaluation questions and other themes as they
emerge, rather than presenting findings drawn
from different data collection methods. You
may want to include findings that are
interesting but do not answer the original
evaluation questions. But do not feel you have
to report on everything. If the information
adds nothing to your findings, leave it out. If
the findings were inconclusive, say so.
Keep the information needs of your audience
in mind. You may need to present the
information in a different, possibly more
concise, format for your trustees or your
funders. If so, a stand-alone summary
section is helpful. Your funders may want
to see the full report, but may also want some
of the information presented in their own
standard report format.
The way a final report is presented is
important. It should look easy to read, with
clear headings and sub-headings. Consider
using lists and bullet points, boxes and shaded
areas to highlight or separate content, and
footnotes for extra information. Make sure
your numbering system is as simple as
possible. Have a clear and comprehensive
contents page.
Think about sharing the writing of the report.
To make sure it reads consistently, edit and
discuss each others drafts.
Findings
Data
Monitor
routinely
Analyse
Collect
additional data
Assemble
Interpret
Report
charities
evaluation
services
63
basic
evaluation
When writing your report:

remember your audience and the key questions covered by the evaluation
keep it short, including only what the reader really needs to know
use careful wording for negative points and avoid reflecting on individuals
use plain English, and avoid jargon
make sure the content follows a logical sequence
use a standard, readable sans serif font, of at least 12 points in size.
There are guidelines on good practice in presenting information.
Tables, diagrams and charts

Tables, diagrams and charts get information
across effectively and concisely. Look at
other reports to see how charts and tables
are used and set out. Bar charts are good for
comparing two sets of information about
the same things, so they can be used to
show changes over time. Pie charts give a
visual representation of proportions. If you
want to show how proportions have

changed over time, you need a separate
chart for each year.
The bar chart below shows how a sexual
health project has distributed information
leaflets using a number of different channels
over a period of time.
Chart 1 Distribution of leaflets years 1, 2 and 3

percentage 100
of total 90
distribution 80
70
60
50
40
30
20
10
year 1
0
year 2
drop-in
year 3
centre
outreach
For example: Royal National Institute for the Blind

(1998) See it Right: Clear Print Guidelines and The
Word Centre (1999) Plain English Writing Guide.
schools
community
centres
advice
centres
social
services
other
charities
evaluation
services
64
Chart 2 Distribution of leaflets in year 3
drop in centre
outreach work
schools
community centres
advice centres
social services
others
The pie chart above shows the proportion

of leaflets distributed through different
channels in a single year.
You can use simple tables to break up and
illustrate the text, and further clarify the
data.
Remember to give charts and tables a chart
number and a full descriptive title so the
reader does not have to read the text to
understand them.
Case examples
Case examples describe a particular event
or set of events, activity or personal history.
They are intended to demonstrate and explain
evaluation findings. They can be very useful in
demonstrating good and bad practice,
illustrating complex services and bringing an
evaluation to life. But it is helpful to explain
the point you are illustrating in the main
narrative.
Kumar, R (1999)
Research
Methodology: A stepby-Step Guide for
Beginners, SAGE,
London
Quote what people have said in their

interviews, or on questionnaires or feedback
sheets to illustrate your findings. Make sure
you include negative as well as positive
comments if they are given. Do not use too
many quotations and be careful that you do
not use quotations instead of interpreting
your data.
Robson, C (2000)
Small-scale Evaluation:
Principles and Practice,
SAGE, London
Remember that individual respondents and

their views must not be identified within a
Further reading
case example, or elsewhere in the report,

unless they have given written permission.
Be careful to remove or change identifying
material from your findings, and be
particularly careful when you have a small
number of respondents from a particular
group, where it may be easy to identify them.
Where it is difficult to ensure anonymity, you
must get clearance from the people involved
for the information you are presenting.
Editing and proof reading

You need to edit your report, to check it
for content, meaning and flow. Book time
with a colleague so you can review what
you have written and get feedback before
the report is finalised. After editing, you will
need to proof read your report to make sure
that there are no typing errors or spelling
mistakes, and that paragraph, page and
other numbering are correct. These final
stages take longer than you think, so leave
plenty of time in your timetable.
This section has looked at the key
components of self-evaluation. If you
have collected monitoring data you will
be able to bring this together across a
whole year. Once you have thought
clearly about your evaluation questions,
you can decide what other data you need
and how you will collect it. Your report
will suggest answers to your evaluation
questions, and recommend ways forward.
further evaluation
Basic evaluation concentrated on
bringing together monitoring data and
supplementing it within a structured selfevaluation exercise. Further evaluation
examines evaluation approaches within
a basic theoretical context and considers
different types of evaluation activity.
These relate to the focus of the evaluation
enquiry. This section also looks at the
relationship between evaluation, quality
audits and other audits. Finally, it considers
good practice points for managing and
reporting your evaluation, whether this is
carried out internally or by a consultant.
weaknesses of the project, such as:

whether the project met its overall aims
Formative and summative

evaluation
Evaluation can be carried out with different
intentions. These are usually described as:
Formative evaluation intended
to improve and develop the project.
Summative evaluation intended
to make judgements when a project
is completed.
Formative evaluation may begin at the start
of the project. It continues through the life of
the project and includes evaluation of both
project implementation and progress. It will
rely heavily on good internal monitoring.
Summative evaluation assesses a projects
success. Sometimes, this is called an end-ofterm evaluation. It takes place once the
project is stabilised and has had a chance to
bring about changes. It collects information
about processes and outcomes, and answers
questions about the successes, strengths and
65
further
evaluation
the extent to which users benefited

the most effective activities
whether the project could work
in another setting.
Focusing the level

Project level evaluation should ideally be
an integral part of the project. Monitoring
should start soon after the project is
funded, and provide information for periodic
evaluation to improve the project as it
develops and progresses.
Focusing your evaluation

The Further planning section raised the
importance of focusing the issues or
questions for evaluation. These questions
may be influenced by changes which are
inside or outside the project, or by the
stage reached in the projects life cycle.
charities
evaluation
services
Programme level evaluation examines the

overall activities and the resulting changes
from a number of funded projects. These
projects will have a common, wider aim,
such as improving sexual health among
young people, or developing the involvement
of older people in care services. Information
is gathered and put together from project
level evaluation, usually once a year. It can:
identify common themes across a
group of projects
evaluate particular issues across projects,
such as partnership working
help make decisions about project
funding and support
show how well a collection of projects
fulfils the broad aims of a programme,
and how it brings about policy or
institutional changes.
Focusing the subject matter

Evaluations also need to identify particular
aspects of the project to study. This may
be the way the project is working, what it
is doing, or what it is achieving. Basic
evaluation looked at focusing on outcomes
and impact. This section introduces context
and process evaluation. Focusing in this way
does not necessarily mean that you research
and report on these things in separate
evaluations, although you might do so.
Focusing your
evaluation
charities
evaluation
services
66
Context evaluation
Evaluation can look at how the environment,
or context, a project operates in affects it,
and this will help you understand how and
why the project works. A needs analysis
will have established some of the basic
contextual factors. Later on, you may
need further context information when
you modify or develop your project. What
other services are available from other
agencies? What is the political climate
and how has it affected the work?
Contextual information is also essential
when attempting to reproduce projects
and services elsewhere. How does an urban
or rural environment, for instance, affect
implementation?
It is also important to understand how
organisational contextual factors might
hinder or support project success.
Questions might include:
How do the organisational structure
or staffing decisions influence project
development or success?
How effective are decision-making
processes?
You will evaluate implementation at an early

stage in the project. Later on, a progress
evaluation will show how far you are meeting
a projects aims and objectives. It will also
point out unexpected developments, and
help you to fine tune the project. Evaluating
the implementation and progress of your
activities will provide an important back
up and explanation for the findings of any
outcome evaluation. You will be able to
put those findings in the context of what
was implemented and why. Without that
information it will be very difficult to show
the relationship between project activities
and outcomes.
Implementation questions could include:

Is the project staffing in place,
and is it appropriate?
Have activities followed the
proposed timetable and plan?
How is the project promoted
and publicised?
How well are partnership
relationships working?
Progress questions could include:
Process evaluation
A process evaluation will help you to
assess the planning, setting up and
implementation of a project, and decide
how to improve or modify current
activities. It will focus on processes
how the project works and also provide
valuable information on progress. Process
evaluation is particularly important for
pilot projects so you can learn what needs
to be improved. Specific purposes might
include:
finding out what is working well
finding out where there are problems
and why
assessing how users and other
stakeholders experience the project,
and their use of project services.
How are services carried out to

allow access for particular
user groups?
How do different project activities
relate to stated objectives?
How satisfied are users and what
do they most value?
What are the effects of project
activities on users?
charities
evaluation
services
Evaluation approaches
Once you have clarified your evaluation
focus, it will be helpful to consider your
evaluation approach.
While there are many variations, monitoring
and evaluation approaches that focus on
accountability needs have largely followed
the tradition of scientific investigation.
These approaches assume the power and
value of measurement and objectivity.
However, there has been a shift towards
approaches that can be used more
easily in decision-making, and an
acknowledgement that quantitative data
is itself an approximation or interpretation.
This links to the increasing emphasis on
participation by stakeholders throughout
evaluation evaluation that can be used,
and evaluation for learning.
Such approaches are labelled naturalistic.
Naturalistic approaches assume that there
will not always be a definite answer. They
focus on qualitative data and description,
and they value subjective understandings
and different perspectives. Naturalistic
approaches, such as case study evaluation
and participatory evaluation, acknowledge
that all enquiry is value-driven and is
more likely to be sensitive to the different
values of programme participants. This
fits particularly well with a voluntary
sector ethos.
In practice, the difference between the two
approaches to evaluation is not as clear cut
as in theory. Qualitative and quantitative
methods may be used in both approaches.
Case study evaluation

Case study evaluation allows you to examine
a particular individual, event, activity, time
period and so on in greater detail. It
emphasises the complexity of situations
rather than trying to generalise. This
approach may be particularly appropriate
when evaluating certain types of projects or
for certain audiences. In programme

evaluation you may choose to report on
certain projects as case studies. Case studies
are used to illustrate findings and offer
particular learning.
You can use case studies to illustrate specific
successes or difficulties, and to try to
identify the circumstances in which your
services are effective and where they are
not. It will also allow you to describe in
greater detail the processes of the project.
In describing processes, case studies can tell
you something about the constraints under
which you operate.
For example, if your case study is about
an individual, you will be able to research
what was happening to them in their
contact with the project, how they
experienced your services or activities,
and how the project affected their life.
It will allow you to take into account the
circumstances that were unique to them,
while at the same time identifying the value
they experienced from the project. This
approach recognises that outcomes are
often specific to individuals and will not
necessarily be common across a group of
people. So each case study does not
attempt to generalise the findings, but to
look at individual stories.
Each case study should therefore stand
alone, so make sure that the data you have
for case studies is as complete as possible.
At the same time, if you have a number of
case studies or stories to tell, you will then
be able to compare and contrast them and
pull out any common themes. This may
make your findings more compelling.
You may want to let participants tell
their story in their own words. If so, you will
need to tape an interview or to get
permission to quote a diary record or
other form of self-report.
67
further
evaluation
Evaluation
approaches
charities
evaluation
services
Participatory evaluation
68
We have discussed how stakeholders can

take part in evaluation in a number of
ways. For example, they can work with the
project to:
formulate the aims and objectives
of the evaluation and the key questions
design and test different data collection
tools, such as questionnaires
gather and analyse information.
Participatory evaluation actively involves all
stakeholders in the evaluation process, gives
value to different perspectives and provides
information of practical value to the project.
It aims to help stakeholders identify and
solve problems themselves: in other words,
building peoples organisational capacities.
It can also strengthen partnerships and
encourage an inter-agency approach
through joint reflection. This gives project
partners a chance to assess their own
performance and draw lessons for their own
future planning.
In practice, there may be considerable variation

in the level and degree of participation. It
will be more, or less, appropriate in different
circumstances. The level and degree of
participation may also depend on whether the
evaluation is internally, externally or jointly
led, and whose perspectives are dominant
those of the funder, the users or the staff team.
Participatory evaluation can be externally led,
by people with no direct involvement with the
project. An external evaluator may give technical
help, or act as a facilitator or supporter, helping
stakeholders decide on and carry out their
own monitoring and evaluation activities.
Some people assume that participatory
evaluation is less scientifically rigorous. But
participatory approaches have their own
systematic rigour, and will often include
quantitative measurement. Users can enhance
a project by their involvement in setting
performance indicators, collecting monitoring
data, completing self-assessment reports and
other tasks without necessarily putting aside
standards of validity.
A model of participation in evaluation

One-way
information
Consultation
Dialogue
Basic evaluation
tasks
Complex
Collaboration
evaluation tasks
Stakeholders
know the
results
Views asked
about
predetermined
issues
Involvement
in the design
and input into
decision-making
on some of
the issues
Stakeholders
carry out parts
of the data
collection
Stakeholders
Joint
involved in
responsibility for
processing
the evaluation
findings and
making
recommendations
No participation
Full participation
charities
evaluation
services
User-led evaluation
Theory-based evaluation
A participatory evaluation aims to involve

stakeholders to varying degrees in the
evaluation process. In a user-led evaluation
the control passes to the users themselves or
to user-representatives.
Theory-based evaluation is useful for larger

and complex social programmes. It aims to
identify the theory, which underlies any social
programme, about what will make it work.
This theory is called a programme logic model
or a theory of change. It can then be used to
decide what must be done to achieve aims
and the key intermediate outcomes that will
lead to ultimate long-term outcomes.
A good start is to hold a consultation

meeting with users to decide the areas
of interest and the evaluation questions.
Train the user-evaluators in data gathering
techniques, including interview skills
and data analysis. Users can design
questionnaires and other data collection
tools themselves, and decide how to
collect the information.
Tracking these intermediate outcomes can

be a good option in the first year or two.
Stakeholders are learning how and why a
programme works, and what activities need
to come before others. This allows the
programme to be adjusted, or the theory
to be modified, as learning occurs.
User-evaluation needs adequate support. You

could use a consultant to:
A programme logic model helps focus the

evaluation on measuring each set of events in
the model to see what happens, what works,
what does not work, and for whom. This may
in turn lead to a better understanding of the
effect of the programme, and help to find
ways of increasing its effectiveness.
help users think through the technical

aspects of the exercise
discuss research methods
observe the interviewers
discuss analysis of the data
comment on the draft report.
The attempt to link activities and outcomes

throughout a programme initiative means
theory-based evaluation can also guide
those who wish to apply lessons to similar
programmes in other settings. It fits well
with the current concern by government and
policy-makers with what works and why.
Programme logic
Inputs.....lead on to
Outputs.....which lead on to
Short-term outcomes.....which lead on to
Outcomes.....which lead on to
Impact
69
further
evaluation
charities
evaluation
services
70
Designing an outcome
evaluation
unlikely to have happened purely by chance.

However, remember that the presence of a
relationship does not prove cause and effect.
When you design an outcome evaluation, the

following models are useful.
Comparative designs
After project assessment
If you have enough resources, you may be

able to look at a comparison group. This
may be:
This option collects information against

outcome indicators only after the project is
completed. You will not be able to measure
the difference from before the project
started. You are therefore very reliant on
participants own memories, and feedback on
how they feel that things have changed. One
way of making this information more valid is
to also use evidence from a number of
sources, such as observation, or interviews
with staff or others who had contact with
project participants.
Before and after project assessment

With this option, you take the same
measurements before and after the project.
This means you may be able to get a more
convincing demonstration of changes for
participants over the life of the project. If you
think during the planning stage about how
you will evaluate your project, you may be
able to collect baseline data about your users,
about services, policies or the environment
before your project starts. You can then
make later measurements against these.
By assessing the statistical significance of the
change you can test whether the result is
a group of people which has not received

project benefits a control group
a similar scheme run elsewhere.
Comparative studies are sometimes thought
to offer more conclusive evidence that
outcomes are attributable to the project.
However, many projects are unique and
cannot be safely compared. For most
voluntary organisations, running a comparison
group will be a difficult and more costly
option, and it is often hard to recruit and keep
group members. It is important to consider
carefully how participants for the project
group, the experimental group and the
control group are recruited. The groups must
be exactly the same in all respects in age,
sex and socio-economic factors other than
the fact that one group receives the projects
help. Working with two groups, one of which
does not receive a benefit, may raise ethical
considerations.
The distance that needs to be measured is
that between B and D as well as that
between A and B.
B
Positive change
C
Before project
Negative change
After project
project group
comparison group
charities
evaluation
services
The group not benefiting from the project

may have experienced positive or negative
change, and the reasons for this need to be
understood.
It will be difficult to make sure that there
were no differences between the groups
before the project, when measurements are
only taken after the project. With before and
after assessments it is difficult, if not
impossible, to isolate all the factors that may
have caused change in the comparison
group, even if the groups were similar at
first. Most studies show relationships and not
cause and effect because of the difficulties
of eliminating other possible factors.
Emergent designs
There may be situations when all your
evaluation questions, and even the methods
for collecting evidence, are not clear at the
beginning of the evaluation. In this case, you
have flexibility to allow important issues and
questions to emerge as you do the work, and
you design your data collection
in response.
Extending your evaluation

activities
We have discussed some of the main areas of
concern for evaluation focusing on the
extent to which projects are achieving the
project aims and objectives and the benefits
to users. Yet there are other important issues
on which you need to make judgements. If
you want to know whether your project
offers value for money, or whether you are
working according to values of equal
opportunity and user empowerment, for
example, you will need different sorts of
evaluation. However, this does not
necessarily mean that you need separate
evaluation studies.
Economic evaluation
Funders often want to know whether a
project provides value for money or is cost
effective. Economic evaluation recognises the
choices involved in spending limited resources
and focuses on the non-monetary and
monetary costs as well as benefits.
Costs are the various inputs, both direct and
indirect, needed to set up and run a project.
The most important cost is often the time
spent by staff and volunteers on different
activities, and this needs to be multiplied
by the value of each persons time. Other
resources to be costed include physical
facilities, equipment, supplies and time.
In a cost-benefit analysis you measure costs
and outcomes in monetary terms. The cash
flow can be used to work out the percentage
annual rate of return. Or, if you divide the
project costs by project benefits, this will give
the cost-benefit ratio. This approach needs
considerable economic expertise, and extensive
work that is expensive and problematic
because it is imprecise in its measurements. It
is rarely suitable for voluntary sector activity.
Cost-effectiveness analysis can be less

demanding than a full cost-benefit analysis.
It measures the outcomes and impacts of
projects and programmes in non-monetary
terms against the costs, for example, cost per
job created or client served.
Cost-effectiveness is about using the resources
you have in a way that produces the maximum
outcomes. The questions asked by costeffectiveness evaluations are useful ones for
voluntary sector projects. They ask whether
the project has achieved economy, efficiency
and effectiveness or, in more simple terms,
whether project costs, design and
achievements are reasonable. Questions
include:
Economy are the resources (people
and supplies) used in the project the
most appropriate ones, and are they
obtained at a reasonable price?
Could the costs be lower without
damaging the project?
71
further
evaluation
Designing an
outcome
evaluation
Extending your
evaluation
activities
charities
evaluation
services
72
Efficiency is the design and operation

of the project the best way of
producing the outputs? Is there no
other approach that would be better?
Effectiveness what outputs and
outcomes is the project producing?
Are they sufficient to achieve project
aims? Are the benefits reasonable in
comparison to the cost?
Benefits are usually compared with planned
benefits, and costs are usually compared with
similar initiatives and against planned costs.
A cost-effective project should yield benefits
to the target group greater than the overall
costs of running the project and should
compare well to the costs of similar projects
achieving comparable results.
While the focus on costs and efficiency issues
is useful, a full-scale value for money exercise,
involving comparison with other schemes or
with the cost of not undertaking the project,
requires economic expertise and robust
evidence. Also:
There are considerable problems in
calculating potential costs and benefits
for longer-term outcomes, where costs
and savings may be too far in the future.
It is difficult to put a value on social
benefits that do not have a monetary
value, and where benefits are for
communities rather than individuals.
Finding out the costs of alternative
ways of delivery could be timeconsuming, and it may be difficult
making comparisons between them.
As with other outcome and impact
evaluation, it is difficult to isolate the
effects of a single intervention from
other influences.
a monetary value on the social and

environmental benefits of an organisation
relative to a given amount of investment.
The process involves an analysis of inputs,
outputs, outcomes and impacts leading to
the calculation of a monetary value for those
impacts, and finally to an SROI ratio or
rating. For example, an organisation might
have a ratio of 4 of social value created for
every 1 spent on its activities. SROI studies
work with judgements at various points in
calculating their ratio, but these should be
well documented so as to be transparent.
SROI studies have now been applied to
organisations producing social returns, such
as helping ex-offenders into employment,
where benefits can be seen as clients cease
to receive benefits and start to pay taxes, all
of which result in savings to the criminal
justice system. It is recognised that the
methodology may not be suitable or
appropriate for some organisations, but the
SROI methodology could help make a good
case for providing certain types of services
and be especially useful if an organisations
funders require outcomes information in
financial terms.
Like other outcomes approaches, SROI can
help you manage and improve services and
make your organisation more sustainable. It
can also improve your relationships with
stakeholders; stakeholder engagement is an
important part of the SROI process and is
one of its strengths.
Evaluating sustainability
A key evaluation question is about how
sustainable the project is. Will the project be
able to flourish after a particular source of
funding has ended? You need to ask:
What ongoing funding is secured?
Social Return on Investment
How sound is the structure of the

project or organisation?
The social return on investment methodology

(SROI) was originally developed for social
enterprises, and developed from traditional
cost-benefit analysis. The methodology puts
How has the project linked with other

agencies and the wider environment?
What are the social, political, cultural
charities
evaluation
services
and economic conditions which support

or hinder a projects growth and
sustainability?
How sustainable are the benefits
both the outputs and the outcomes?
Can the benefits be continued for
later groups?
If funding and staff are withdrawn,
do project users have the skills and
resources to continue?
You will be able to answer some of these
questions during the life of a project. You
may also want to revisit these questions
after several years, to check how the project
developed and how it achieved sustainability.
important aspect of your project, but may

not be reflected in the project aims and
objectives. If so, you may wish to cover
this aspect of your work in your evaluation
activities. What important questions do you
have about the partnership? For example:
How useful and effective were
meetings and other joint exercises?
What strengths did each agency
bring with them?
What were the difficulties and how
were they overcome?
How was information and understanding
shared between partners?
What changes has joint working
brought to each agency?
Evaluation against values

As well as asking whether your project
achieved the changes and benefits it
intended, a key question is whether it
worked in a way that is consistent with
its values. For example, your quality system
or other policy documents may state
that the project should be user-centred,
empowering, environmentally friendly
or respect equal opportunities.
This is where evaluating your processes
is important. You may need data on your
consultation and decision-making processes,
or to observe how staff and volunteers
interact with users. Another important
source of data might be feedback from
users, partners and other agencies, so
make sure you ask them questions that
will give you relevant information.
Outcomes will also reflect on your values,
so set outcome indicators that relate,
for example, to empowerment. Break down
your data, as appropriate, according to
gender, ethnicity, age and sexual orientation
so you can assess whether project
benefits were affected by user profile.
Evaluating joint initiatives and

partnerships
Working with other agencies may be an
What resources, including skills and

expertise, have been shared or
increased?
Should joint or partnership working
be continued? How can it be improved?
It is important to be clear about the precise
nature of the partnership. Is it a contractual
partnership, or simply a collaborative
relationship?
If you include the evaluation of joint
initiatives and partnerships in your evaluation
plan it will prompt you to keep good
monitoring records of this part of your work.
Interviews with key individuals in partner
agencies during your evaluation may shed
light on their commitment to joint working
and help to resolve difficulties. Sharing the
results with your partners may help you to
find better ways to work together.
Management audits, social audits

and other assessments
Evaluation is not the same as audit. Auditing
has traditionally been concerned with
financial honesty and good practice and
checking against agreed procedures, controls
or standards. Evaluation, on the other hand,
aims to make or aid judgements against a set
of questions or criteria. However, as audits
73
further
evaluation
charities
evaluation
services
74
have become more participative, and seek to

discover how an organisation can become
more effective as well as more efficient, the
distinction has become increasingly blurred.
Audits ask whether a programme is operating
as originally intended. Evaluation may do this
too, but may also try to discover whether or
not a programme has worked, what has
made it work and what effects it has on
whom.
Management auditing systematically

examines the organisation and its
structure, its management practices
and methods, and is carried out
externally and independently.
Value for money audit or performance
audit is mainly linked to reviews of
publicly-funded programmes. It usually
relates to the three Es economy,
efficiency and effectiveness important
in economic evaluation. A fourth E can
be added equity, or doing things fairly
which links performance auditing to
social auditing.
Social auditing has grown out of the
business world. It links a systematic
reporting of organisational activities to
the issue of social impact and the
ethical behaviour of an organisation.
Managing evaluation
Whether you are carrying out selfevaluation, or using an external evaluator,
or combining both approaches, evaluation
requires careful management to avoid or
overcome some potential problems.
For example:
one or two people often carry much of
the responsibility of the evaluation and
it may be difficult to involve others
the time and resources involved may
prove too much of a burden, resulting
in an incomplete or poor quality process
evaluation staff may become
demoralised
there may be a hostile response to
negative findings or to the process itself.
Self-evaluation may not always be appropriate
for your specific evaluation needs. External
evaluation can bring a fresh perspective,
specific skills and new insights. However,
make sure you build in enough consultation
time so that the evaluation, while
independent, works to relevant criteria and
meets your immediate priorities, timetable
and resources.
Using an external evaluator

A social audit should be done regularly,
as with a financial audit. It should try to
get the views of all stakeholders,
possibly using questionnaires and focus
groups. While there are strong similarities
to evaluation, there is a greater
emphasis in this type of audit on public
accountability. A social audit report
should be published and made available
to all stakeholders and the general
public, covering all aspects of an
organisations activities.
Organisational self-assessment identifies

and analyses an organisations strengths,
weaknesses and potential, and can
be carried out against a set of capacity
elements or quality standards and
indicators.
If you decide to work with an external

evaluator, help the evaluation to run smoothly
by doing the following:
name someone to act as a key contact
be clear about budgets
make sure that you and the evaluator
have agreed on a realistic timetable
and deadlines
make sure that all the relevant people
and documents are readily available
make it clear what length and style
of report you want, and whether you
want a summary.
charities
evaluation
services
It is important that the evaluation continues

to belong to the project. If you use an
external evaluator, make it clear in the
contract that you will expect to see and
comment on the draft report. As long as
your suggested amendments do not distort
the findings, it is reasonable to ask for
changes, to allow for factual adjustments or
differences in interpretation. However, this
will be subject to discussion and negotiation
with the evaluator. If you cannot agree, both
possible interpretations might be included in
the report, or the evaluator might refer
to the different view in a footnote.
In self-evaluation, an external consultant
may help with the technical aspects of
the evaluation, help gather specialised
information and provide an external
viewpoint. You may want to use a social
sciences student to help with data input
or analysis. You may also want some help
to carry out interviews. This is skilled work,
so you need someone with experience.
Setting up a steering committee is one way
of encouraging participation, learning and
sharing. The steering committee should,
if possible, include some people with
evaluation or research skills and experience.
Think carefully about the relationship
between the evaluator and the steering
committee. Where does control of the
process lie? In a self-evaluation, this should
remain with the project. With an external
evaluation commissioned by a funder, this
question of control may need to be clarified.
Other good practice points, whether the
project is managing an internal or external
evaluation, include the following:
have realistic expectations of what
the evaluation can deliver
communicate openly and have respect
for the evaluators, telling them about
any changes or information that might
affect the evaluation
collect data and work to established
procedures and with appropriate
permission
be sensitive to the needs of the

respondents
recognise the need for evaluation to
maintain confidentiality
be aware of possible disruption to,
or implications for, ongoing work
preserve the integrity of the evaluation
by not using findings out of context.
Contractual arrangements
Terms of reference
The terms of reference, or brief, for an
external evaluation should set out all the
principal issues that will guide evaluation
activity. These should be dealt with by the
proposal from the evaluator. The contract
should refer to these documents and should
spell out the responsibilities of each party.
The contract you make with any external
consultant is important and should be
signed and in force before the evaluation
begins. It will include or have attached the
terms of reference and the evaluators
proposal, and state clearly the amount and
method of payment. It should also clarify
issues to do with ownership and copyright.
It will be useful for the contract to detail
administrative requirements, and where
these responsibilities lie. For example, you
may need to:
make databases available, or collate and
analyse existing monitoring data
make available any previous reports or
relevant literature
contact stakeholders to brief them about
the evaluation and introduce the
consultant
send out questionnaires, in order to keep
costs down.
Ownership and copyright
Be clear from the start about rights and
responsibilities. The following questions
should be answered:
75
further
evaluation
Managing
evaluation
charities
evaluation
services
76
Who owns the report?

Does an external evaluator have the
right to publish the report in his or
her own name, without first consulting
the project or evaluation sponsors?
Who needs to approve the final
document?
Can the project or evaluation sponsor
edit certain sections of the report
before publishing it?
Can the project reject an independent
external evaluation or prevent it from
being published?
comparison and prevent bias, and that

word-for-word responses are essential.
Check that the length of responses from
the interpreter matches what the informant
has said, and make sure that responses are
not edited, distorted or embellished.
It may be appropriate to use advocates if
you work with people with learning
difficulties. Remember that advocates are
there to help only when necessary. Find out
about the respondents preferred means of
communication and ask questions in an
accessible way.
Quality control
Unless you have agreed who has authorship
and copyright, neither party may be free to
use the report widely; this includes website
publishing. Joint copyright means that both
the author, that is, the evaluator, and the
project, have rights to the final manuscript.
Neither party should publish as opposed to
circulating to an immediate audience
without the agreement of the other.
These issues of authorship, copyright and
publication should be settled early on and
precise procedures put in place.
Practical details
Think through in advance any practical
details. For example, are there any
constraints on data collection that the
researcher should be aware of? Are there any
IT compatibility issues or difficulties about
access to paper files? Are there any issues
concerning confidentiality for staff and
users?
Pilot testing will show if questions are likely

to be understood by respondents and if they
will capture the information that the
evaluator wants. Do not skip this stage,
because it will save time in the long run.
Improve the quality of data collection and
analysis by training and supervising data
collectors carefully, so that they ask the
same questions in the same way, and use
the same prompts.
A second person should check from time
to time that agreed procedures are being
followed. This may include procedures
for contacting respondents, obtaining
consent, and collecting and recording
data. Researchers can get bored or tired
by a large number of repetitive tasks,
and this can sometimes lead to uneven or
missing data or transcription errors. Examine
the data file for accuracy, completeness
and consistency, and check for coding
and keying errors.
Using interpreters and advocates

Sometimes it may be essential to work with
interpreters in order to access people whose
first language is not English. However,
working with interpreters is not easy. Allow
enough time to explain the aims and
objectives of the evaluation, and how the
part they are involved with fits into the
whole. Make sure they understand the need
for questions to be asked precisely to allow
Data analysis, including coding, is discussed

further in the Practical toolkit.
Evaluation budget
Evaluation needs resources, including time
and money. Self-evaluation will minimise
costs, although you should still set a
charities
evaluation
services
budget. Include evaluation plans and costs

as an essential part of your initial project
proposals. This will show funders that
you are keen to learn from the project and
your ability to demonstrate effectiveness.
It is difficult to set a standard for
appropriate evaluation cost ratios. However,
the cost should reflect some proportional
relationship. Some argue that it should
be as high as 5 to10% of the overall project
or programme budget. This may not be
realistic for many small projects, or indeed
acceptable to funders, but you should
take into account monitoring activities,
management time and other routine
evaluation commitments.
This section has examined different types

of evaluation and evaluation approaches.
Understanding these will help you
to be clear in your discussions with
stakeholders about the possible focus for
your evaluation. It has considered some
practical aspects of managing evaluation.
All your efforts have been focused on
producing a clear, illuminating and
persuasive report. This is the beginning
of the next phase of the evaluation cycle
utilising the findings.
77
further
evaluation
You can minimise costs by designing systems

that collect information from
the start. Costs can also be reduced by
conducting telephone or group interviews
rather than one-to-one interviews, or
by using a questionnaire rather than
interviewing. An evaluation budget
should consider the following:
staff time
consultants costs
travel and other expenses for
respondents
data analysis and other research
assistance
travel and communications, for
example, postage and telephone costs
printing and duplicating
other services, such as interpreting
or signing services.
Make sure you allow enough time for your
evaluation. Qualitative evaluation studies can
take longer than you expect if you plan to
interview and observe individuals and groups
of people. Your timetable should allow
enough time to carry out an online survey or
postal questionnaire and analyse the data.
Allow twice as much time as you think you
will need to write your report.
Further reading
Clarke, A and Dawson,
R (1999) Evaluation
Research: An
Introduction to
Principles, Methods and
Practice, SAGE, London
Gosling, L and Edwards,
M (2003) Toolkits: A
Practical Guide to
Monitoring, Evaluation
and Impact Assessment,
Save the Children,
London
Van Der Eyken, W (1999)
Managing Evaluation,
Second edition, Charities
London
charities
evaluation
services
section 4 utilisation
Basic utilisation
81
Disseminating evaluation findings
81
Using evaluation findings
82
Further utilisation
84
Valuing the findings
84
Using evaluation for management
84
Using evaluation for policy change
85
Using evaluation for strategic planning
86
charities
evaluation
services
79
charities
evaluation
services
basic utilisation
Once you have presented your report, it may
be tempting to file the evaluation away. This
will be a waste of all the time, energy and
resources spent on the evaluation and risks
losing the goodwill of everyone involved.
Importantly, it will also be a lost opportunity
to improve what you do. This section
suggests ways that you can disseminate, or
pass on, your findings, and how you can use
the findings to demonstrate your projects
progress, to make adjustments and to plan
the future direction of the project.
Within your project

at management committee functions
You can use your findings in many ways for

different audiences. When you are planning
your evaluation, think about who would be
interested in it, or who might be influenced
by the results. Other projects can learn from
your experience as well.
How the evaluation is to be disseminated
and used should be part of the evaluation
design. If you have involved stakeholders
from the beginning of your evaluation, you
will have planned together how the lessons
learnt will be processed. It is also more
likely that there will be agreement about
the findings.
Oral presentation of the findings as well as
the written report is very useful and should
be built in. In a larger organisation formal
feedback systems will most likely be needed.
You can summarise your monitoring and
evaluation results in your annual report, and
therefore publicise information to a wide
audience. However, this medium is not
appropriate for too much detail or critical
analysis. There are a number of ways you
can disseminate your findings:
81
basic
utilisation
staff meetings
annual general meeting
in the newsletter.
With other stakeholders
distribute copies of summary sheets
of your findings to your partners
write brief articles in other
organisations newsletters
Disseminating evaluation
findings
charities
evaluation
services
write a press release

include key findings in your
publicity and promotion
present key findings at forums
you attend.
Think also about using the following:
training events and conferences
special items in newsletters
local radio interviews
the internet
displays
videos.
It is encouraging to give feedback to users
and others involved in the evaluation, to
show the results of their efforts. Sharing
the information with other agencies makes
sure that they can learn from what you have
done. Your evaluation can also provide good
material for publicity and public relations.
People will see that your organisation
is actively providing interesting or useful
information for the partnerships and
networks in which your project is involved.
Disseminating
evaluation
findings
charities
evaluation
services
82
Using evaluation findings

Providing information to funders
and other stakeholders
Monitoring and evaluation will provide useful
information for funders about the level of
activities and benefits for your users. You will
be able to give examples of what has worked
well, and what your users most value. You
will also be in a better position to make a
good case to funders for continuing or
developing activities. Make it clear to them
that you have effective monitoring and
evaluation systems in place, and show how
you have learnt about and adjusted services
in the light of your findings.
Use information from the evaluation to build
up credibility with other agencies and
increase the likelihood of referrals. You can
also use your evaluation findings for
publicity, for lobbying or campaigning, for
advocacy and to highlight gaps in services.
Think about how you can use the media,
such as talking on local radio or inviting a
journalist to meet project workers and users.
Learning from evaluation

Sharing the information within your
project will help you to become a learning
organisation. Management can improve its
decision-making, and staff and volunteers
will appreciate the value of the work that
they do and understand how they can make
further improvements. Information-sharing
can help you assess whether the project is
still meeting needs, and whether staff have
the right skills and are working to quality
standards. Once the immediate reporting
back has taken place, make sure that dates
are set for action so that impetus and
enthusiasm are not lost.
It may be possible to make small-scale
adjustments to your project after a reportback meeting. For example, you might:
adjust workloads
change your monitoring systems
change promotional material
introduce new training
increase your quality control.
Using evaluation for organisational

planning
Managers may commission an evaluation
because they need more information to
decide between different courses of action.
The evaluation is therefore designed to
provide decision-makers with knowledge and
information to make informed choices.
Your evaluation should show which parts
of the project are working, for what people
and in what circumstances, and provide a
warning if something is going wrong. These
are key findings and you need to decide
what action to take. Is extra funding needed?
Are new activities required? Do staff need extra
training or skills?
You may need to discuss some recommendations
at other organisational planning meetings and on
development days, with the aim of improving
project delivery and services. Make sure you draw
up action plans and make sure the project
supports managers and staff in carrying out
recommendations. You may, for example, need
to change job descriptions and working
methods.
The evaluation will also provide information for
your next year plan. It will help you to review
your objectives. Are your services or activities the
right ones to achieve the intended change or
benefits? If the project has brought about some
unexpected results, how will you take those into
account in future planning? You may need to
gather more information about the outside
world, for example local strategies and other
service provision, before making decisions about
changing services.
charities
evaluation
services
The evaluation may give you clearer

information about who is using your services,
about your members, or who you are reaching
with your information or publicity. This will
help you to think more carefully about who
you are not reaching. If the findings point out
areas where need is greatest or least served,
you may need to consider redefining your
target group. You may need to carry out more
publicity or establish new contacts and
networks. It may be that you need to follow
up your evaluation with a more in-depth
needs analysis.
Your evaluation will also allow you to review
your targets for outputs and outcomes. If you
have not met certain targets, or if you have
exceeded them, then you should be able to
set this against what you now know about the
capacity of the project and the performance
of other agencies. Your evidence should be
strong enough to show if there were
good reasons for a lower than expected
performance, whether targets were set
realistically and whether you should
adjust them.
Give managers, staff and partners an

opportunity to comment on the way the
evaluation was carried out, so that lessons
can be learnt for future evaluations. File
the evaluation data and report for future
reference. Use the experience of selfevaluation to examine how you might
improve the quality and usefulness of
the information you are gathering.
83
basic
utilisation
Using
evaluation
findings
This section has covered the final stage

of the self-evaluation cycle. The starting
point was planning for the project itself,
and defining the evaluation questions.
By systematically collecting data, and
presenting your evaluation findings
clearly, you will be able to demonstrate
your progress to your stakeholders.
You will also be able to make any
adjustments needed to keep your
project on track to achieve your
aims and objectives.
Further reading
Lyons-Morris, L and
Taylor Fitz-Gibbon,
C (1987) How to
Communicate
Evaluation Findings,
SAGE, London
charities
evaluation
services
84
further utilisation
Basic utilisation discussed the need to
disseminate findings and to use them to
demonstrate progress and to feed back
into, and guide, the management of the
project. This section examines further how
you might use evaluation findings internally
and externally for three main purposes:
management
- reviewing targets and target groups
- incorporating quality improvements
into planning
- reviewing key resources
policy change
strategic planning and development.
Valuing the findings

To make it more likely that evaluation
findings will actually be followed up,
decision-makers must be receptive to
evaluation and believe that it can make a
difference. In organisations which have an
established culture of monitoring and
evaluation, evaluation findings are more
likely to influence the decision-making
process. Negative findings may well be
ignored by project staff if people receive
them as a personal criticism or see them as
threatening. Whatever the findings of the
evaluation, they may be welcomed by some
groups and rejected by others.
There are a number of factors that influence
whether findings will be used. It is important:
to provide information that is really
useful
to deliver the information when it is
needed
that stakeholders consider the
evaluation to be of a high standard
to communicate evaluation findings
appropriately evaluation stands little
chance of making a difference if decisionmakers do not read the final evaluation
reports because they are badly written or
presented.
Using evaluation for

management
Evaluation findings are most likely to be used
when people recognise their immediate
relevance to the project. Staff and volunteers
are under pressure in their daily work routines
and will need motivation to use evaluation
findings and make changes. Work towards
changing the culture of the organisation, so
that people are receptive to new ideas and
challenging feedback.
Evaluation can be helpful to provide
information on pilot projects, or to test out
new service innovations. With routine
evaluation also, use the lessons learnt about
what you could do better, or differently, in
your operational planning. Evaluation can be
useful to identify areas of unmet need and
areas for service development. Do you need to:
change the way the project is managed?
reallocate resources?
expand or change direction?
Using evaluation to set service

delivery standards
Evaluation should give you some important
information about how you deliver your
services to users, how this affects user
satisfaction and how service delivery affects
the outcomes for users. For example, user
satisfaction and trainer observations should
allow you to set standards for your training.
These could be about such things as the
maximum number of participants, the quality
of training materials or accessibility for
disabled people.
Other examples of service standards that
evaluation might help you to set include:
the ratio of staff and volunteers to users
staff and volunteer training and
supervision
reception care and information
food quality and the observance of
religious and cultural norms.
charities
evaluation
services
Reviewing key resources

Your evaluation should not just look at
the results of your activities, but should
relate these to the projects inputs.
How have activities been affected by
the projects management structure,
its staffing or funding levels? Are the
projects physical facilities unsuitable or is
the project transport inadequate? Do you
need improved information technology to
support the project?
If the evaluation has not drawn clear
conclusions or made recommendations
about these issues, make sure that the
evaluation review looks at any implications
the findings may have for key resources.
Financial or human resources may need
to be reallocated, or you may need to
raise more income in order to develop
new services or activities. You may have
to do further feasibility work to make a
good case to funders.
Using evaluation for policy

change
Evaluation can play a key role in highlighting
the effect that wider social structures and
policies have on your own work and on the
lives of the people you work with. For
example, an evaluation of services for
disabled people might examine local
government policies and provision, and
explore how they influence or constrain
service provision. An evaluation of services
for people with learning difficulties might
look not only at service users views but also
at how wider community care policies affect
the experiences of service users.
Policy is influenced by a combination of
factors, including the assumptions, personal
ideology and interests of the policy-makers.
Although the processes involved in policymaking are complex, evaluation can be
designed not only to improve practice,
but also to improve and change policies
at many levels.
When you design your evaluation, think

about policy issues and your evaluation
aims relating to them. These policies could
range from operating policies in your
own organisation to the policies of larger
systems, for example of your local authority.
What levels and types of policies are you
trying to change? Who are you trying to
influence?
When you report on your evaluation, think
about who your audience is. If you intend
to publish the evaluation, this clarity about
your audience is essential. Communicate
the lessons you have learnt directly and
simply. Be direct about what types of policies
might be effective, based on findings.
Communicate what you know to community
members and project users. This can gain you
allies to help you influence policy-makers.
Organisations can use evaluation to develop
their advocacy and campaigning role, the
process of evaluating being an essential part
of working for social change. You may need
to work with others, using evaluation to
strengthen networking and collaboration.
85
further
utilisation
Valuing the
findings
Using
evaluation
for
management
Using
evaluation
for policy
change
charities
evaluation
services
86
Using evaluation for strategic

planning
There is an important role for evaluation in
strategic as well as operational planning. The
strategic planning process starts when you
have analysed monitoring and evaluation data.
Steps in strategic planning
Routine monitoring
Evaluation
Step 1
Analyse
data
Other data collection
Step 2
Assess
strategic
options
Step 3
Agree
strategy
Two useful tools in the strategic planning

process are:
strategic environment: SWOT analysis
(Strengths, Weaknesses, Opportunities
and Threats)
situational or PEST analysis (Political,
Economic, Sociological and Technical
issues affecting the project the need
for the project).
Framework for a SWOT analysis
Positive
Negative
Internal: for example, internal resources,

skills and organisational structure
Strengths
Weaknesses
External: for example, political, social and

economic environment; external stakeholders
Opportunities
Threats
Monitoring and evaluation data can provide

valuable information for developing both
SWOT and PEST analyses if you choose to use
these. You need time to draw out conclusions
before summarising the strategic position
of the organisation and identifying essential
questions.
charities
evaluation
services
You have now reached the final point in

the planning, monitoring, evaluation and
utilisation cycle as your review process
feeds back into project planning.
Aims and
objectives
Strategic
planning
87
further
utilisation
Operational
planning
Implementation
Evaluation
Using
evaluation
for strategic
planning
Process
Outcome
Voluntary organisations vary greatly in their

purpose and nature, size, management
structures, and in the resources available to
them. They will also differ in the range of
monitoring and evaluation activity they
choose, or are able to carry out. Whatever
the profile of your project, or the complexity
of the systems you adopt, you are likely to
increase effectiveness if you clarify and plan
project activities, obtain information
routinely and systematically about your
services and activities, and make periodic
judgements about how well you are doing.
health and sustainability of your project and,

more importantly, for your stakeholders and
the people you exist for.
Effective monitoring and evaluation
will help your organisation to provide
services of the highest possible quality,
and embody the highest standards of
integrity, credibility and accountability.
It will make sure that you are working
with the greatest possible effectiveness
and efficiency, that you provide value
for money and, above all, that the work
you do will make a real difference.
Finally, using your monitoring and

evaluation findings is what makes the time
and energy you have invested worthwhile.
You involved your stakeholders from the
start, you defined the important questions
together, and at the end of the process you
produced valid and reliable findings and
presented these clearly and persuasively.
This will all be wasted if you file your report
away in a drawer. Instead, you will have a
powerful tool to demonstrate the value of
what you are doing, plus practical lessons
to develop the project.
Whether your project is small, medium or
large, you will want to do everything in
your power to be and be seen to be
as effective as possible. This is vital for the
Further reading
Patton, MQ (1997)
Utilization-focused
Evaluation, Third
edition, SAGE, London
charities
evaluation
services
toolkit (89-133) new jp:toolkit 03/09/2009 13:35 Page 89
section 5 practical toolkit

Collecting and analysing data
Introduction
Establishing credibility
Documentary data sources
Collecting data directly from individuals
Data collected by an independent observer
Data collection matrix
Sampling
Data analysis
Data collection tools
91
91
91
92
93
102
104
105
108
Introduction
Tool 1 Quarterly monitoring report
Tool 2 Outline evaluation report
Tool 3 Interview topic guide
Tool 4 User satisfaction form
Tool 5 Intermediate outcome monitoring form
Tool 6 Individual assessment tool
Tool 7 Outcome assessment: participant self-report
Tool 8 Telephone interview guide
Tool 9 User interview schedule
Tool 10 Participatory learning and action (PLA) tools
111
111
112
114
115
116
118
120
122
124
126
128
Further reading
130
Glossary
132
charities
evaluation
services
89
charities
evaluation
services
collecting and
analysing data
charities
evaluation
services
Introduction
Establishing credibility
An important part of the early phases of

evaluation is to think through the evaluation
focus and the approach appropriate to your
evaluation questions. You also need to be
clear about data collection: where from,
how you will collect it and how you will
analyse it.
Whether you do a self-evaluation or an

external evaluation, it is very important that
the evaluation is seen to be credible. This
will rely very largely on what stakeholders think
of the perspectives and skills of the evaluator,
the techniques and methods used, and the
soundness and accuracy of the findings.
91
collecting
and
analysing
data
The Practical toolkit gives you an overview

of data sources and data collection methods,
and provides more detailed guidance on
specific methods, such as interviews, focus
groups, questionnaires and observation. It
introduces key aspects of data analysis and
has some checklists and examples of tools
that you can adapt.
It will be helpful in establishing credibility

to be clear about stakeholders expectations,
and to discuss in advance the methods
and approaches that are most appropriate
for the specific evaluation and project.
When choosing data collection methods,
first think about the skills and resources you
have available. Think also of what will be
most appropriate to the sensitivity and
complexity of the issues, and what will be
most appropriate to the project values and
your stakeholders, including your users.
Finally, you need to consider:
Reliability how consistent is the
measure? What is the likelihood of
getting the same results if procedures
are repeated?
Validity are you measuring what
you intend to measure?
Bias
Bias occurs when findings are unduly
influenced by the way data is collected,
analysed or interpreted.
When data is collected. For example,

the researchers themselves, or the way
questions are asked, may influence how
people respond. If questions are not
clear enough, they may be interpreted
differently by different people.
When data is analysed. For example,
if data is not broken down to show
differences between groups, it may
not reflect the true differences
within a sample.
Introduction
Establishing
credibility
charities
evaluation
services
92
The effects of bias can be limited through

careful training, cross-checking and
supervision. Analyse your own biases as
well and see how these might affect
data collection and analysis. Being aware
of and monitoring any possible bias may
enhance the validity of the results.
Documentary data sources

Documentary sources may raise questions that
it will be useful to tackle in other parts of the
evaluation, so it is best to do this work at the
beginning of the study.
Literature search
Some measures that can be taken include:
eliminating bias as far as possible by
ensuring questions are asked with the
same wording, and in the same manner
making sure that groups chosen
for comparative purposes share as
many characteristics as possible, except
for the variable you are studying
obtaining evidence to answer your
evaluation questions from different
types of sources. This is called
triangulation.
The term triangulation means getting
evidence or taking measurements from
different points. You can use triangulation
by looking for different sources of data
on the same area of enquiry or applying
different methods to the same data
sources. For example, you may wish to
ask a group about the benefits of working
together, obtain feedback from the group
facilitator, and observe the group yourself.
If you do this, you will have greater
confidence in your evaluation findings.
It may, of course, result in contradictory
findings, and it will be your task to
consider explanations for such differences.
The following section gives a detailed review

of data collection methods. There are three
major sources of data:
documentary
obtained directly from individuals
obtained from independent observers.
A review of other published or unpublished

material can be important to set the project
in a wider context. This secondary data may
be government reports or university and
voluntary sector research. The internet gives
access to wide-ranging information that
may be useful for your evaluation.
Desk research
This is a review of records, databases and
statistical monitoring data. You may need
to look at these over a period of years.
For example:
records about members and users
planning documents
policy and discussion papers and reports
publication sales and distribution figures
minutes of meetings
correspondence.
Desk research is valuable because it helps
you understand how a project has developed,
and how it is managed and resourced. It is
essential for implementation evaluation.
Client-based data, such as case records, may
not only tell you who you are seeing and
when, but may also provide some outcome
data.
charities
evaluation
services
Collecting data directly

from individuals
Questionnaires and surveys
A questionnaire, with a list of questions that
everyone is asked, is a popular method to
get information from users. Surveys collect
information in a standardised way from a
large number of people, very often using a
questionnaire. They are useful for collecting
a wide variety of data, particularly on
opinions and attitudes.
A questionnaire can be quite user-friendly,
and put fewer demands on the users and
project than other methods. One advantage
is that the same set of questions can be
used as often as needed, so that you can
make comparisons over time, or between
groups of people. Responses can be made
anonymously and questionnaires are a
relatively cheap and efficient way of
reaching large numbers of people or
covering a wide area. Disadvantages
are that response rates could be low and
they call for a level of literacy among
respondents. It is difficult to get complex
information, and to control and know
how people interpret questions.
Questionnaires are used in two ways:
A schedule of questions for face-to-face
interviews, with the form filled in by
the interviewer.
A self-completion questionnaire, filled
in by the respondent. These are often
mailed out, but short and simple
questionnaires can also be handed out.
You can ask people to fill them in on
the spot, which should increase
the response rate. Very complex
questionnaires are better completed
face-to-face or over the phone, to make
sure that questions are understood.
The design of the questionnaire is

important. To get the information you
want, it is important to ask questions in a
way that people will understand and to
minimise the possibility that people will
interpret their meaning differently. Try to:
learn from questionnaires that other
agencies have sent out
involve your users wherever possible
to make sure questions are phrased
in language they use
pilot test the questionnaire, that is, ask
at least two or three people to fill in
the questionnaire as a dummy run to
see if it works as planned.
Online surveys are increasingly used as an
efficient way of reaching a target audience;
they have the potential to reach large
numbers of respondents, provide instant
collation of quantitative data, and permit
statistical analysis when the data is
downloaded into a spreadsheet file. The
survey can also be printed in hard copy for
those respondents unable to access a
computer, and the additional data can be
inputted into the online survey. As with
other questionnaires, it is important to limit
the number of questions and to design the
questionnaire thoughtfully. If certain
information is essential to your survey, you
may need to make some questions
compulsory. Think also about the order of
the questions, placing the most important
ones early on.
Many elements of good questionnaire
design are common to both postal
questionnaires and online questionnaires.
The next page covers some of the important
points to bear in mind.
93
collecting
and
analysing
data
Documentary
data sources
Collecting data
directly from
individuals
charities
evaluation
services
94
Designing the questionnaire

Once you have decided to use a
questionnaire, spend time on its design, as
this will play a vital part in the quality of the
data you get back. If you can, get the help
of someone experienced in designing and
analysing questionnaires. They can help you
make sure that data can be easily processed,
and think through how to link different bits
of information together, such as profile data
and attendance, or length of membership
and satisfaction levels.
There may be benefits in having

questionnaires and surveys that can be
completed electronically. You will need to
consider this in the design.
Think about your target group and the sort of
format that might be needed to make sure
that all users can fill in the questionnaire. Do
you need a large type size for people who are
partially sighted? How will you include people
in your survey who are not literate, or who
cannot speak English, or who are physically
unable to write?
When you design your questionnaire:

Have a cover sheet or introductory paragraph which:
introduces the questionnaire and the evaluation
promises anonymity and confidentiality where possible
asks respondents for consent to any follow-up contact
or to clarify or explore answers.
Give simple, exact instructions on how to answer
each type of question, for example, whether to tick
a box or circle a number.
Keep the questionnaire as short as possible.
Include only questions you need.
Use tick boxes where appropriate.
Provide enough space for answers.
Double-check question numbering.
Design a clear and attractive layout with a large, clear font.
Include instructions for return of a postal questionnaire,
with a time deadline and a stamped return or freepost envelope.
charities
evaluation
services
How to write good questions
Make sure you get a good response
Questions ask the respondent for factual

information, opinions or preferences.
These can be designed to get two types
of answer:
You are more likely to get a good response

if you:
open-ended with free response

closed or pre-coded.
Pre-coded questions force choices from a
limited range of options but have the
advantage of being easier to translate into
statistical analysis. Many questionnaires
combine closed with open questions. Make
sure that if you ask respondents to tick
boxes you allow for negative as well as
positive responses or outcomes to be
shown. Allow also for some open-ended
questions on areas for improvement or for
wider-ranging comment.
pilot the questionnaire first to iron out

any potential problems
try to mail out at least three times
more questionnaires than you
need responses
send a reminder
use a freepost service or a stamped
addressed envelope for responses
offer it by email if people prefer.
When writing questions you should:

Keep questions short: no more than 20 words where possible.
Use clear and simple language that suits people filling in the
questionnaire. Avoid jargon, slang or unfamiliar terms.
Have questions that follow a logical sequence.
Make sure that all questions are relevant to the respondent and
can be answered by them without significant time or effort.
Make sure that questions do not have more than one possible meaning
and cannot be interpreted differently by different respondents.
Relate questions to a specific time period, as in: Over the past
week, how many times have you ? When asking respondents
about the frequency of an event or activity, specify a suitable
timeframe, such as less than once a week, once a week,
more than once a week.
Avoid asking two different things in the same question.
If you want a detailed answer, make sure the question will give you one.
For example: What three things, if any, would you change about the course?
Provide a category for answers such as: Dont know, or Did not attend.
Ensure responses are quantifiable by providing boxes to tick, or a choice
of numbers on a scale. You can leave space for additional qualitative responses
as well.
95
collecting
and
analysing
data
charities
evaluation
services
96
Face-to-face interviews
Types of interview
An interview is more than a one-to-one

conversation, because the interviewer
retains control, and gets responses to
specific questions. A record is kept of
responses. Advantages of interviews are
that they do not require literacy, and you
can check that your questions have been
interpreted correctly and that you have
understood responses properly. It is also
a more personal approach for sensitive
subjects. Disadvantages include the lack
of anonymity, that it is a time-consuming
method and it is difficult to achieve
consistency among interviewers.
Structured interview
In a structured interview the interviewer
only asks questions that have already been
prepared in an interview schedule, using a
set order and exact wording. Interviewers
are given detailed instructions on how to
ask the questions.
You can hold interviews with an individual

or with a group. Use group interviews
when you have limited resources or when
an individual response is less important.
Remember that in group interviews people
may be reluctant to disclose personal views
or sensitive information. Identify key people
with whom a one-to-one interview is most
important.
Interviewers need to be able to establish
rapport with the respondents. Interviews
should usually be held in a convenient
environment where the respondent feels
most comfortable. This might mean that
you have to travel to their home or office.
Sometimes all the questions are closed,

and the respondent chooses answers from
fixed responses. However, some open-ended
questions can be included; the instructions
to the interviewer might then include
some probes used to tease out further
information. It is important that the
wording of questions is clear and does
not vary, so that different interviewees
do not interpret questions differently.
The advantage of the structured interview
is that it is focused, and the answers will
be more consistent and easier to compare
and analyse. The disadvantages are that
the method is inflexible and the questions
may not be the most relevant for the
individual respondent. So you may get a
distorted picture of their views, impressions
and reactions.
Recording interviews
You will be able to take notes during most interviews.
This is important, because if you wait until afterwards
to write things down, you will forget much of what
you heard, and your own biases are more likely to
intervene. Type up your notes as soon as possible
after the interview.
You may find it useful to tape a lengthy interview.
This will allow a detailed analysis and you will be able
to use accurate quotations in your report as evidence.
Always get permission before taping. Transcribing a
taped interview can be time-consuming and costly.
One option is to use notes of the interview as the
basis of your analysis. You can then use your tapes
selectively to supplement the notes.
charities
evaluation
services
Semi-structured interview
A semi-structured interview follows a less
rigid format, with open-ended questions
designed to draw out more qualitative
information. The schedule is used as a
prompt to make sure that all the required
topics are covered. Questions do not have
to follow a predetermined sequence and
the interviewer can explore the answers in
greater depth. The interviewer is free to
vary the exact wording of the questions
and to probe and follow up answers. A
semi-structured approach can be particularly
useful for group discussions.
Unstructured interviews
These are sometimes called depth interviews,
as they encourage the interviewee to reveal
in depth any aspects they want to raise.
Unstructured interviews are flexible, can be
tailored to the individual and can lead to a
greater understanding of the respondent.
Their weakness is that information is likely
to be different for each interviewee, is less
likely to be systematic and comprehensive,
and data analysis can be difficult.
Combining interview styles
You might combine interview styles. There
may be a small set of specific questions that
you want to ask everyone, and a number of
topic areas which can be explored in a more
flexible way.
Telephone interviews
Telephone interviews are particularly useful
for getting feedback from busy people.
They can be used to obtain quite sensitive
or complex information, or where you need
to contact a large number of people in
their home or office. In some situations you
may be able to conduct lengthy telephone
interviews, but try not to make them longer
than 20 minutes.
It is important to have a well-thoughtthrough schedule of questions to ask. You

should put them in order of importance,
as you may only be able to ask the most
important ones if you are to keep the
interview brief.
Having a tight structure for the responses
makes conducting a telephone survey easier,
as many of the responses can be recorded
in tick boxes. However, as with face-to-face
interviews, you can get in-depth answers
to questions by using more open-ended
questions. It is hard to listen to what the
respondent is saying and write down at the
same time. It is a good idea to record the
interview as well; this means that you can
get accurate quotes and go back to the tape
where you have gaps in your notes.
These good practice points may help to get
more out of telephone interviews:
Get consent to the interview in advance.
Make an appointment for the call
just as you would for a face-to-face
interview.
Careful sampling is especially helpful
for keeping the task manageable. Use a
larger sample than you will need, to
allow for difficulties in making contact.
Allow enough time for making successful
contact with respondents.
Suggest in advance any information
that respondents might find it helpful
to refer to.
Think about how to keep your call
confidential, if appropriate.
It is good practice to send people an outline
of the questions or topics you will cover
in interviews. This may be reassuring and
allows them to think about your questions
in advance.
97
collecting
and
analysing
data
charities
evaluation
services
98
Group interviews
Focus groups
Advantages of group interviews are that they

can lead to the group identifying solutions
and agreeing action. The opportunity can
also be used for group members to raise
issues that may not have been on your
agenda. Disadvantages include the possible
domination of a group by one or more
individuals, and that group views tend
towards agreement, rather than allowing
for different views. Consider having a
separate note taker and remember to record
and use the whole discussion, not just the
points agreed by the group as a whole.
The essence of a focus group is that it is

focused on a limited number of questions
based on a core topic. As such, it is different
from a more open, flexible group interview,
which may be varied in size and scope.
Focus groups usually have eight to 12
people. They can be used to clarify specific
topics that can then be discussed by a larger
group, or to help set the questions for the
evaluation.
A group interview or discussion needs

careful planning and preparation. It may
be possible to use an existing group that
meets regularly for another purpose. At
other times a group may need to be put
together from scratch.
Group interviews are not good for revealing
sensitive information, and the facilitator
needs skills to:
encourage each person to speak
keep the discussion on track
recognise important issues and probe
if necessary
summarise and check back.
To help you work out your questions for
interviews or discussions, it can be helpful
to hold some unstructured interviews with
one or two of your respondent group first.
Ask open questions about what they feel are
the key achievements and issues. Although
their views will be individual, this will help
you develop your final list of questions.
There are a number of other key features:

a facilitator guides the discussion
sessions last for one and a half to
two hours
recording is usually word for word.
Choosing focus group participants
Whenever you can, try to make sure that
participants are as similar as possible in
terms of age, gender, social class and other
characteristics, to encourage a good
communication flow. Some people argue
that ideally participants should not have
met before. However, to be practical, it may
be easier to interview a ready-made group,
if one exists.
If your participants are unfamiliar with the
term focus group, it might be better to call
the group an informal discussion.
Running group interviews and focus groups
Group interviews and focus groups can be
quite lively and energising. Group members
support each other to communicate and
people spark ideas off one another.
However, they need considerable skill to run
and can be time-consuming and difficult to
analyse.
charities
evaluation
services
The facilitator guides and leads the

participants through a focused discussion
on the topic to be investigated. A topic list
should be prepared in advance to make
sure that all relevant issues are covered
during the meeting. In particular it is the
facilitators responsibility to make sure that
all participants have an opportunity to
speak and not to let one or two individuals
dominate the discussion. Make sure that
the venue is fully accessible and find out
if anyone in the group has specific
communication needs.
It is important that the note taker can write
quickly and accurately. If you decide to
tape the discussion, always ask participants
permission first and remember that
transcribing tapes takes a long time.
Participatory learning and action

Participatory learning and action (PLA)
methods combine a range of interviewing
and group work techniques. Facilitators work
with participants interactively, and encourage
them to express themselves freely, visually
and creatively, and then to discuss what they
have expressed.
PLA has been developed from work in
countries in the developing world, where it
has been called participatory rural appraisal
(PRA) or participatory appraisal (PA). The
term participatory learning and action
was adopted to emphasise the need for
the approach not just to generate data,
but also to involve participants in change.
So the methods are best used within an
overall approach to involving users.
The approach was designed to reach

and hear the experiences of the most
marginalised and isolated sections of
communities. It is intended to allow
communities themselves to analyse
their situation and find solutions, at
the same time recognising that people
have diverse views.
There are a number of PLA tools. Examples
of these are:
drawing, used to illustrate ideas

timelines, which show the sequence
of different events over time
ranking exercises, where things are
placed in order
participatory evaluation exercises,
such as evaluation wheels
matrices, which display information
such as preferences easily and visually
mapping exercises, used, for example,
to show the social and economic
features of an area, or changes
over time.
At first, try to work with people who are
familiar with PLA to increase your confidence
in using these methods. Remember that it is
important to record the discussion that takes
place about the visual descriptions. You can
increase the validity of the data by getting
information from a number of sources, that
is, triangulate the data.
99
collecting
and
analysing
data
charities
evaluation
services
100
Diaries and work schedules

Diaries can be used to collect information
from staff or users. They can provide an
individual record of staff members or users
day-to-day perceptions of a project. Diaries
or work schedules, which set out actual
daily work routines and the amount of time
spent on different tasks and activities, can
be useful to develop a profile of a project,
or of a particular task within a project. But
they need to be designed so that people
can keep them up to date easily.
A diary can take many forms, ranging from
a set of quite detailed questionnaires bound
together, to a series of key themes on which
to respond, to a blank page for each day.
A relatively flexible format, but with some
focusing prompts, is often used.
Using tests and scales

Any form of test or scale should be relevant
and sensitive to your target group, and
should use appropriate language.
Most achievement tests measure against
either:
a norm, that is, performance compared

to a previously tested group
or
criteria, for example, if a student has

gained specified skills.
A standardised test is one that has been
rigorously developed and tested for its
reliability and validity. Those measuring
attitudes, beliefs or other psychological
aspects are referred to as psychometric
tests. Standardised tests, mainly used in the
health care field, may not be transferable to
a voluntary sector project because of their
specialised and clinical nature. Although
you may construct your own test instrument,
be aware that there may be limits to its
reliability and validity.
There are several types of scales, which allow

measurement against a predetermined rating.
They can be used in a number of ways, often in
questionnaires and in observation, and are often
used to assess attitudes and satisfaction levels.
The most common scales require a response
to a number of statements. There may be
a simple choice of agreeing or disagreeing
with the statement. Or, as in the Likert-type
scale, one of the most popular methods
of measuring attitudes, there is a range
of possible responses as shown below.
Statements may be worded in a positive
or negative manner.
A scoring scheme is usually associated
with the responses. For example:
strongly agree = 1
agree = 2
neither agree nor disagree = 3
disagree = 4
strongly disagree = 5.
A written scale may be replaced by a
combination of written and numerical scores.
For example:
Strongly 1 2 3 4 ... 5 Strongly
agree
disagree.
Always make clear the association between
the scores and possible responses. If you
want to avoid a neutral or undecided middle
category, use an even number of choices,
for example, 4 or 6.
The two examples on the next page are asking
for levels of satisfaction from training course
participants and from recipients of a newsletter.
charities
evaluation
services
Please assess the following elements of the course by ticking the boxes 1 to 5, as follows:
1
2
3
4
5
Very poor
Poor
Average
Good
Very good
1
a Course content
b Tutorial groups
c Information materials
d Course administration
e Social activities
Please show your satisfaction with the newsletter by ticking the boxes 1 to 5,
where 5 shows that you are very satisfied.
Very dissatisfied
a Topics covered
b Design and layout
c Clarity of language
d Length
e General interest and relevance
To find out why respondents have given a

particular score, it is helpful to design your
questionnaire or other data collection
instrument to allow for explanations or
comments to be recorded.
You can also investigate attitudes by using
ranking. Here, respondents are invited to
show their preference by putting various
factors in order of importance. This could
be done by using a numerical ranking system
or, for example, by arranging statements or
pictures visually.
Very satisfied
101
collecting
and
analysing
data
charities
evaluation
services
102
Working with children

Dont be afraid to bring children into
your evaluation as they often have clear
ideas about their problems, priorities
and expectations. However, be aware that
if adults are present, such as teachers
or social workers, this may put them off
talking or they may not say what they
really feel.
There may be child safety issues if you
interview them alone, so make sure that you
are visible when interviewing children, or
work with another interviewer. You must
get permission from the local authority
to interview children under 18 who are in
care. You will need to get permission from
a parent or guardian for children under
16 who are not in care.
Here are some good practice points:
choose a small, friendly place for
the interview and allow time to gain
the childs confidence
if possible visit them beforehand to
explain what you are doing, and
tell them if you plan to use a video
camera or tape recorder
be careful with your language
and check that you are understood
explain that there is no right or
wrong answer
Think about the different techniques that will

be suited to different age groups. Childrens
drawings can be useful. Ask them to talk
about them. When you are asking children
questions, it can be helpful to use drawings
and symbols, such as sad and happy faces.1
Data collected by an
independent observer
Observation is useful for collecting information
on group behaviour and in situations where
people find it hard to talk. It is also useful for
assessing the quality and standards of delivery,
for example of training, in a residential setting,
or of the activities of a day centre.
The use of observation can involve project
participants in the evaluation, and encourage
their involvement in other data collection
exercises. On the other hand, the technique
runs the risk of bias and subjectivity. You will
need to think clearly beforehand about what
you are analysing. Reliability of observation
depends heavily on the training and skills of
those doing the observing. You can reduce
this risk if you devise a well-thought-out
observation checklist. Where you have more
than one observer:
give children a chance to ask you

questions at the end of the interview.
agree with observers in advance

how ratings should be applied
check that observers apply the
ratings correctly
You can ask children their views directly,

or informally, or you can use observation
methods. Relaxed, open-ended interviews
and group discussions work well. Around
four to six children is a good number
enough to get them talking but not so
many that they wont get a chance to
speak. Interview in the same age groups
and in same-sex groups where appropriate.
pilot and run some observations jointly to

make sure the approach is standardised.
Source: McCrum, S and Hughes, L (1998) Interviewing
Children: A Guide for Journalists and Others, Second

edition, Save the Children, London
charities
evaluation
services
Setting up the observation instruments can

therefore be time consuming. There are other
potential difficulties. Participants may feel
observation is intrusive and it may cause
anxiety. The observation itself may affect
the behaviour of those being observed.
You can use a structured or unstructured
approach to observation. Unstructured
observation is an open, exploratory approach
which provides descriptive data about
activities, the contexts in which they take
place and the people taking part. Even
unstructured observation will be led by basic
questions about who is involved and how,
what is being done, in what way, when and
where, and how things change and why. It is
also important to take note of things that do
not happen, as these may be significant.
In structured observation the researcher
conducts the observation within the
framework of a number of more precisely
predefined categories. These can include
the quality of group interaction, effectiveness
of certain techniques, or individual responses
or behaviours.
Techniques include the following:
using a checklist of items to note or
tick off
devising a way to count and categorise
specific aspects of behaviour and attitude
taking notes against a series of questions,
either open or fixed choice, such as:
Which topic areas cause the most/least
discussion?
Participant observation
This involves observing while participating
in activities, such as training courses and
workshops, club activities and activity
weekends. The advantage of participating
is that you will gain experience of the
project as a user and put users at their
ease. The disadvantage is that it is harder
to stay neutral.
Take into account that:
gathering, recording and analysing
observation data can be costly
there may be concern about the reliability
and validity of observational data
if there is more than one observer,
you need to check that you are using
the same criteria and applying them
consistently.
Always let people know when they are going
to be watched, and ask their permission.
However, be aware that saying this is likely
to affect behaviour.
Mystery shopping is an assessment of a

service carried out by people posing as users.
It is useful as a check on service delivery
and to check performance against standards.
Depending on the service provided, you may
need a specialist to do this. You can do it
with the knowledge and consent of those
providing the service or without it. When
deciding on this, bear in mind the type
of service and the likelihood of ill-feeling
being caused if people have not been told
beforehand and had the value of the method
explained to them.
103
collecting
and
analysing
data
Data
collected
by an
independent
observer
charities
evaluation
services
104
Number of older
people who have had
needs assessments
Level of user
satisfaction
Number of people
contacting statutory
and independent
agencies
Extent and type of
working relationships
with other agencies
Number of older
people with
Housing Benefit
Total amount of
Housing Benefit
Numbers receiving
health care
Types of services
received
Numbers receiving
care packages
Types of care
packages
Data collection matrix

A data collection matrix is a helpful tool to
plan the methods you will use for each area
of enquiry. You can assess which method
will give you the richest source of data, how
you can triangulate data by collecting it
from a number of sources, and how to

balance what you ask using each method
so as to keep the work manageable. The
matrix below shows the methods suggested
to collect data for an evaluation of St
Bernards Homes, which provide hostels and
shared accommodation for older people.
Interviews
with users
Staff
interviews
Telephone
interviews
with other
agencies
Monitoring
data
Document
review
Case
records
charities
evaluation
services
Sampling
Random sample
Whatever method you choose, you may wish

to get your data from a sample. By sampling
you can collect information on a smaller
number of people or a smaller number of
project activities rather than from the entire
group, or population.
Sampling may be necessary either for
qualitative or quantitative studies. In
drawing up the sample, you need to decide
who and what to gather information from,
for example:
categories or groups of people
specific individuals
105
collecting
and
analysing
data
For quantitative studies, some form of

random sampling will be a reliable and
valid method of sampling. First you need
a complete list of the whole population so
that every individual or organisation has
the same chance of being included in the
sample. This allows you to make statistical
generalisations about the wider population
from what is found in the sample. Try to
avoid taking a sample that you know is
not typical. The size has to be large enough
to cover the range of experiences and
characteristics that you want to find out
about.
Data collection
matrix
Sampling
For example:
project locations
every fourth name on your records
activities or events.
every tenth organisation on your register.

Find a random way of deciding where to
start the sample. In the following example,
a sample of six is chosen randomly from
a population of 18. Every fourth person is
selected. When you get to the end of the
grid you continue counting from the
beginning.
A
5
C
1
Sample starts here
G
2
K
3
O
4
charities
evaluation
services
106
Stratified sample
Sometimes a simple random sample is not
the best way of making sure that all the
types of people or organisations that you
work with are represented. For example, if
you only have a few clients who are disabled,
random selection may miss them entirely, just
by chance.
To avoid this, in a sample of 80 you may
decide that you want:
25 men aged 19 to 54
25 women aged 19 to 54
outcomes and five who have not been

successful. This allows you to make the most
of the information you get from a smaller
sample, but you cannot use the data to make
generalisations about the population or
to describe typical experiences.
A time sample
A time sample will include every case that
happens over a set period. For example, you
may monitor the number of people coming
to a busy service over a week or a fortnight.
Choosing the right sample size

as well as:
10 disabled people
10 people under 19
10 people over 54.
This would enable you to get the views of all
the important constituent groups, in large
enough numbers for the feedback to tell you
something about the way that individual
groups perceive the service, as well as how
the overall user group sees it.
Random stratified sample

In a random stratified sample, people or
organisations would be randomly selected
within sub-groups.
For example, if 70 of your users are women
and only 30 are men, you might want
to get feedback from women and men in
proportion. If you are interviewing 30 people,
you would interview 21 women and 9 men.
To choose them, divide up your participant
list into men and women, and choose, for
example, every fourth person on each list
until you have the number you want.
Purposeful sample
With a purposeful sample, you make a
deliberate choice of who or what to include,
according to specific characteristics. This
type of sample enables you to capture a wide
range of views, experiences or situations.
For example, you might choose to interview
five users who have had successful
Statisticians have worked out the recommended

sample sizes for various populations, for those
who wish to make generalisations from the
sample and apply them to the whole
population. However, for qualitative studies
there are no hard and fast rules about your
sample size. It does not necessarily have to
increase according to the number of users you
have. For practical purposes, sample size should
be as large as you can afford in terms of time
and money. Be guided by your available
resources, by how you plan to analyse the data,
and the need for credibility.
Select the number of individuals or
organisations that will give you a wide enough
range of experience, and a reasonable number
within each range to give you the depth
of useful information you need. If you plan
to analyse the data by sub-sets, for example
according to age or ethnic group, your
sample size should be large enough to provide
significant numbers in those sub-sets.
Longitudinal studies
A longitudinal study is data collected on
a number of occasions from the same
population over a period of time. If you are
unable to collect data from the same group,
you may interview another group from the
same population, as long as the samples are
selected randomly both times to represent the
entire population. Use identical data collection
instruments, with the same wording, or you
will invalidate the findings.
charities
evaluation
services
Samples have associated problems. These are

shown below:
Problem
Sample bias
This is mainly due to non-response or incomplete
answers. For example, completed responses to a
survey to older people may not reflect the true
age range of project users. This may be because
very old or more vulnerable users were less likely
to complete the questionnaire and their views
are therefore less likely to be represented.
Sampling error
Here you run the risk that if another sample was
drawn different results might be obtained.
Action
Make repeated attempts to reach nonrespondents. Describe any differences

between those participants who replied
and those who did not.
Select as large a sample size as is

practicable within your resources.
107
collecting
and
analysing
data
charities
evaluation
services
108
Data analysis
The quality of your evaluation will depend
to a large extent on the quality of your data
analysis. Data analysis is about organising your
evidence in a logical, well-argued way in order
to make sense of it. You may need advice
when you design your evaluation to make sure
that the type and amount of data you propose
to collect is capable of being analysed.
Data analysis should be done throughout
your evaluation process, not just at the end.
Where large amounts of quantitative data are

collected, the calculations need to be done by
people trained in statistical analysis.
Calculations aim to show you whether or not
your results are statistically significant rather
than just due to chance. A statistical analysis
can tell if the differences shown by a sample
can be repeated without having to repeat
the study or choose a different sample.
Correlation, ttest, chi-square, and variance
analysis are among the most frequently used
statistical tests.
Computer analysis
Quantitative data analysis
Data collected using quantitative methods
can be analysed statistically for patterns,
for example:
percentages
averages
ratios
range between lowest
and highest levels
trends
rates.
Before you collect large amounts of
quantitative data, think how it is going
to be stored and analysed. You will need
a computer database if you have a large
number of questionnaires asking for a lot
of information. Remember that monitoring
forms should have a client code or name if
you want to be able to compare two or more
forms relating to the same client. If you
cannot handle the data in your project, you
may be able to get the coding and analysis
done by a student or a research firm.
First check your data for responses that
may be out of line or unlikely, such as
percentages that are numerically incorrect.
You may need to exclude these from the data
to be analysed. The process of data analysis
can be a useful way to check the accuracy of
the data collected and can identify unusual
or unexpected results.
Most common spreadsheet software has a

range of useful statistical functions for
analysing data. There are also specialised
statistical programs. Understand the computer
program you use, because this will influence
the design of your questionnaire and coding.
When using a computer spreadsheet or
statistics package, you should regularly check
the accuracy of the data entry because of
potential user error.
Qualitative data can also be stored in the
computer. It is often helpful to analyse
qualitative data numerically. A simple analysis
of questionnaire or interview data should
suggest some main categories or types of
response. You can then add up the number of
responses you have in these different
categories.
Coding
Coding means using a number, or a few key
words, to summarise a lot of textual material.
Categories are used to classify the data so they
can be processed by the computer.
There are two main approaches to coding
responses:
Pre-planned respondents reply to fixed
choice questions with pre-set categories.
Additional coding is not necessary.
charities
evaluation
services
Post-analysed this will be necessary

for open-ended answers and comments.
It allows the researchers to set codes
from the full range of responses.
Codes should include dont know, no
response and not applicable responses.
Piloting evaluation tools can help to show
whether questionnaires cover all possible
responses to each question.
Many evaluation questions can be answered
through the use of descriptive statistics.
For example:
How many cases fall into a given category
their frequency.
What is the average, the mean or the
median the central tendency.
Frequency
Once coded, the frequency of each response
can be worked out. Frequency can be shown
in number totals or percentages. Set out the
findings clearly in a table or chart. It is good
practice in tables to show the actual numbers
(the frequencies) as well as the percentages.
Example of grouping data: if, for Thursbury
Community Theatre Project, one of the
evaluation questions was: What was the age
range of young people participating
in after-school and holiday schemes? you
could ask the computer to group data into
a table, for example:
Age range of young people
attending after-school and holiday
schemes over the year
Age range
Number of
young people
Percentage
of total %
Under 10
10-12
13-16
Over 16
21
16
56
44
15.3
11.7
40.9
32.1
Total number 137
100
Central tendency
The mean, or arithmetical average,
involves adding up the data and dividing
by the number of participants.
The median is the point at which half the
cases fall below and half above the
sample.
The mode is the category with the largest
number of cases.
The mean may not provide a good illustration
of responses where these have been widely
distributed. You may need to be more
specific. For example, 65% of participants
found the course good or very good, 15%
found it average, and 20% said that it was
poor or very poor will be more helpful than
presenting the average response. The mode
is useful for describing the most commonly
occurring non-numerical data.
Remember to make the response rate clear
in your analysis. Whenever you can, record
the basic characteristics of those declining to
participate in, for example, a questionnaire,
say by age or sex. Make it clear how the
profile of those responding relates to the
wider population you are researching.
Cross tabulation and sub-group analysis
Cross tabulation examines findings in greater
detail. If the Community Theatre Project
wanted to examine the length of
participation in a holiday scheme by age, the
computer could show this. The table on the
next page demonstrates this.
109
collecting
and
analysing
data
Data
analysis
charities
evaluation
services
110
Length of participation in the summer holiday scheme by age

Age range
2 days or under
3-5 days
6-10 days
11-15 days
Total
Under 10
10-12
13-16
Over 16
1
3
-
1
2
1
-
1
6
1
2
2
8
7
3
6
18
8
Total
19
35
In this case, while the cross tabulation allows

a more detailed description of the summer
holiday scheme, it is not possible to draw
any significant conclusions about the length
of participation according to age. You
need at least 20 cases in each sub-group
for differences to be judged significant, that
is, to judge that any difference shown is
unlikely to have occurred by chance. Even
then, you must recognise that there may
be several interpretations or reasons for the
apparent relationship.
Qualitative data analysis

Analysing by themes
If you have carried out interviews or used
survey questionnaires with open-ended
questions, or you have observation data,
you will need to establish what the data
is telling you. The most common way of
doing this is to analyse it by themes and
patterns in your respondents perceptions,
attitudes and experiences. You may:
already have established themes against
which you will analyse the data, that
is, you have a theory or theories which
you wish to check to see if there is
illustrative data
allow themes to emerge from your
data and then categorise your notes
into recurring themes that seem
relevant to your evaluation questions.
Use a highlighter pen, a simple coding,

or cut and paste to bring together data
that relates to particular themes. Then draw
out similarities or differences. What you
are doing is categorising your notes into
recurring topics that seem relevant to your
evaluation questions. Track recurring words
or comments. Go through responses to one
specific question in a questionnaire and
list all the phrases that say the same thing.
For a questionnaire:
for each question read through at
least 15% of the questionnaires, writing
down each different type of response
when no new responses are found,
finalise the code list against each
separate category of response
repeat this for each open-ended question.
Coding qualitative data also allows you to
quantify your qualitative results because,
once your questions are coded, you can
count how many respondents said the
same things. When you write your report,
you can illustrate the differences among
and within each category of response by
providing more detail of responses and
some quotations to illustrate them.
There are a number of software packages
that can help you carry out this coding and
grouping of qualitative data, and help you
save time in your analysis.
data collection tools

Introduction
This section gives some examples of data
collection tools and reporting formats.
However, the tools you use to collect your
data will be highly individual to your
particular project, to the information that
you need and your specific informants.
Here are the examples we are using.
Tool 1
Tool 2
Tool 3
Tool 4
Tool 5
Tool 6
Tool 7
Tool 8
Tool 9
Tool 10
Quarterly monitoring report

Outline evaluation report
Interview topic guide
User satisfaction form
Intermediate outcome monitoring form
Individual assessment tool
Outcome assessment: participant self-report
Telephone interview guide
User interview schedule
Participatory learning and action (PLA) tools
Most of these tools have already been used

by voluntary organisations. It is always useful
to learn and build from formats that other
projects have used. Most people will be
happy to share what they have used
successfully. It will be particularly helpful
to contact organisations that work with
specific groups, such as people with physical
disabilities or learning difficulties. They may
be able to advise you on points to consider,
or incorporate, in order to encourage
participation and feedback to the evaluation.
charities
evaluation
services
111
data
collection
tools
Introduction
charities
evaluation
services
112
tool 1
quarterly monitoring report
The following is an example of a summary monitoring
report for trustees and staff, providing information against targets.
Services delivered by Thursbury Community Theatre Project: 1 January to 30 September
1st quarter
2nd quarter
3rd quarter
This
quarter
Cumulative Annual
target
% of
target
met
1
7
1
9
16
2
15
20
5
60%
80%
40%
Total number
of organisations
15
25
60%
Total attendance
102
533
1200
44.4%
Number of classroom
workshops
Half-day
One-day
Two-day
2
1
-
12
5
2
20
10
5
60%
50%
40%
Total number
of schools
11
15
73.3%
Total attendance
43
321
700
45.9%
Number of
holiday schemes
66.7%
Total attendance
28
42
75
56%
Number of theatre
skills courses
50%
Total attendance
(teachers)
12
12
10
120%
Total attendance
(community workers)
10
0%
Number of
performances
Schools
Community centres
Other venues
Positive
4th quarter
On target
Negative
charities
evaluation
services
113
data
collection
tools
Other performance targets and indicators

To date, 67% of all performance costs have been met
through sponsorship and charges (target 60%).
28% of the attendance at holiday schemes has been
drawn from black and ethnic minority groups (target 30%).
Three performances were with special needs
groups (target two).
Eight of the performances in schools and community venues
have drawn attendance from the Eastleigh and Freemantle
Estates, identified as estates with a high level of social
problems and lying within our targeted wards. The holiday
schemes served young people mainly from those two estates.
Twenty five of the performances focused on the three
areas targeted: bullying, sexual health and drugs. We held
another two performances on the theme of family relationships.
tool 1
quarterly
monitoring
report
charities
evaluation
services
114
tool 2
outline evaluation report
The report would not necessarily include all sections.
Preliminaries
Front cover
Short title, project name, date, author

Contact details on inside front cover
List of contents
Also list appendices and tables
Abbreviations and glossary
Present in full any abbreviations or acronyms used in the text

Glossary of terms used
Executive summary
Provides an overview of the evaluation, its findings

and implications
The main document

Introduction to the evaluation
Brief description of the evaluation, the evaluation questions

and limits to the study
Methods, data collection tools, respondent groups
(for example, staff, trustees), sampling techniques, analysis
Report overview
What is covered in each section
Project description
Service aims, objectives, values, outputs and expected

outcomes, structure, resources, location, timescale
Target users, user profile, numbers and patterns of use
Results and findings
Usually organised in terms of the evaluation questions

Do not feel obliged to report on everything just what
is necessary
It is helpful to have a summary of the findings at the end
of this section
Conclusions
An assessment of the value of the project

Way in which the results shed light on the evaluation questions
Be explicit about any assumptions you have made
Consider alternative explanations to the conclusions drawn
Recommendations
Statement of key recommendations for future action that

would improve the project and the outcomes for users
Base any recommendations on findings
Ensure recommendations are realistic and within the control
of evaluation users
Ending sections
Notes
Supplementary information relating to the main text
References
All texts and publications referred to in the report

Make consistent use of a recognised system
Appendices
Anything which is relevant to the report, but not essential to

the main text, such as the evaluators brief, questionnaires,
graphs, charts and detailed statistics
tool 3
interview topic guide
charities
evaluation
services
The following example is for an interview with

a project officer in the course of an evaluation
of the implementation of a mentoring project.
A topic guide outlines areas for interview,
allowing a flexible questioning approach.
115
data
collection
tools
tool 2
outline
evaluation
report
tool 3
interview
topic guide
1 Introduction:
personal introductions and explanation
of the evaluation process
length of interview and outline of topics to be covered
assurances about confidentiality and anonymity.
2 Outline of roles and responsibilities in the project.
3 Start date in post, and overview of tasks and
workload since started.
4 Management support and working relationships
with other project staff.
5 First mentor recruitment: methods, successes and difficulties.
6 Recruitment of young people and liaison with schools:
process of building relationships; types of relationships
with different schools.
7 First mentor/mentee group: most successful aspects;
least successful aspects.
8 Drop out mentors: profile and reasons for drop out.
9 Drop out young people: profile and reasons for drop out.
10 Lessons learnt from first mentor/mentee group; things
that will be done differently.
11 Most important issues: project strengths and weaknesses
emerging during the start-up of the project.
12 End of interview:
contact details for any follow-up information
thanks.
charities
evaluation
services
116
tool 4
user satisfaction form
Training course evaluation form
We would be grateful if you would fill in this form so that we can monitor, evaluate and
improve our training.
Course
Date
Trainer
Venue
Please show whether you agree or disagree with these statements

about how the training was delivered (tick one box only for each statement)
Strongly
agree
Agree
Neither
agree nor
disagree
Disagree
Strongly
disagree
The course aims

were well met
The course met my
own expectations
The course was
pitched at the right
level for me
The presentations
were clear and easy
to understand
The group work
was useful
The trainer(s)
responded well
to questions
The handouts are a
good record of
course content
Overall, the course is
good value for money
Please write any comments explaining your responses to the above statements
Please continue on the next page
charities
evaluation
services
Please show whether you agree or disagree with these statements about
how the training may help you (tick one box only for each statement)
Strongly
agree
Agree
Neither
agree nor
disagree
Disagree
Strongly
disagree
As a result of this
course, I feel that I:
have further developed my
knowledge and understanding
of monitoring and evaluation
am more confident about
implementing monitoring
and evaluation within
my organisation
will be able to help
improve the quality
of my organisations
service delivery
will be able to help my
organisation be more
effective in meeting
users needs
Please write any comments explaining your responses to the above statements
Please make any other comments or suggestions about the course
Thank you very much for your time and co-operation
117
data
collection
tools
tool 4
user
satisfaction
form
charities
evaluation
services
118
tool 5
intermediate outcome monitoring form
Peer education training course:
trainers participant monitoring form
Trainers should complete three forms for each participant: one at the end
of the first day, one at the end of the first semester, and one at the
end of the course (before the placement).
Skills
Participant ID number
listen to others
communicate
with others
use appropriate
body language
summarise what
people say
discuss difficult/
sensitive topics
work with different
types of people
plan presentations
deliver presentations
facilitate a group
They really need

to work on this
Not very good
Satisfactory
Good
The participants
ability to:
Very good
Rate the participant on the following, ticking one box only for
each statement, and giving evidence to support your rating.
Please explain or give evidence for your answer
charities
evaluation
services
Confidence
Strongly
disagree
Disagree
Neither agree
nor disagree
Agree
The participant:
Strongly agree
Rate the participant on the following, ticking one box only for each statement,
and giving evidence to support your rating.
Please explain or give evidence for your answer
is confident of
their skills in
working with
people
is able to
describe all of
their skills
is aware of a
range of career
opportunities
available to them
is self-confident
feels good about
themselves/has
high self-esteem
Support
Dates of any one-to-one

support with participant
Comments
Any other comments

Where relevant, please note any other changes you have observed in the
participants skills or self-image.
119
data
collection
tools
tool 5
intermediate
outcome
monitoring
form
charities
evaluation
services
120
tool 6
individual assessment tool
The following horticultural therapy assessment tools
are used, together with an individual person plan,
to help clients identify their needs and how best
they can be met. If a clients aim is to get a job, the
type of job and skills required should be identified.
The initial assessment provides a baseline from
which progress can be assessed periodically. The
assessment tool therefore can also serve as an
individual outcome monitoring tool. It is important
to note, however, that the effectiveness of the
project does not necessarily relate to an increase in
individual skills or work habits. Maintenance of
mobility alone may be an indicator of success for
some clients. Individual outcome information
therefore needs careful interpretation.
Table 1
Continuous assessment: work habits
Date
a Punctuality
b Appearance
c Personal hygiene
d Manners
e Attitude
f Interest
g Effort
h Aptitude
i Communication
j Co-operation
k Integration
A five-point scale can be used to complete this

assessment as follows:
1
2
3
4
5
Unacceptable level
Shows some competence
Below average standard for open employment
Approaches average standards for open employment
Acceptable standard for open employment
charities
evaluation
services
Table 2
Continuous assessment: planting
Date
a
Selects correct plant material
Selects correct number of plants
Selects correct tools
Transports a), b) and c) to

planting area
Digs or dibs correct size of hole
Removes plant from container
Handles plant appropriately
Places plant in hole
Holds plant at correct level
Fills in soil
Firms in soil
Judges/measures planting
distance
m Waters plant
The following scoring can be used to complete

this assessment:
1
2
3
4
Has attempted the task

Can complete the task with physical assistance
Can complete the task with verbal assistance
Can complete the task unaided
Extracted and adapted from an assessment briefing

paper in Growth Point, Winter 1994, Number 208. Thrive,
The Geoffrey Udall Centre, Beech Hill, Reading RG7 2AT.
121
data
collection
tools
tool 6
individual
assessment
tool
charities
evaluation
services
122
tool 7
outcome assessment:
participant self-report
The who-are-you? quiz
We are all better at some things than others
tick the boxes and see how you score.
Score
Very Good
Good
OK
Could be
better
Need to work
on this
Smiley
Comment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Smiley
charities
evaluation
services
Cross reference personal and social skills to questions

This self-assessment tool can be used at different points in time. It is based on the personal
and social skills categories used by Fairbridge, a service working with disaffected young
people, in its staff training and course development. Two further areas Taking responsibility
and Facing up to consequences were introduced during an evaluation.
Questions
Fairbridge/CES personal and

social skills area
1 How good am I at letting other

people know what I mean?
Communication
2 How good am I at understanding

what other people are saying to me?
3 How good am I at getting on with people?
Establishing interpersonal
relationships
4 How good am I at making and

keeping friends?
5 How good am I at keeping my
feelings under control?
Managing feelings
6 How good am I at understanding why I

like some people and not others?
Understanding and identifying

with others
7 How good am I at understanding that different

people have different ways of thinking?
Understanding social values
8 How good am I at sorting problems out?
Problem solving
9 How good am I at understanding other

peoples point of view?
Negotiating
10 How good am I at give and take, or compromise?

11 How good am I at thinking and planning ahead?
Planning
12 How good am I at learning from my successes

and mistakes?
Reviewing
13 How good am I at accepting my share of the

blame when things go wrong?
Taking responsibility
14 How good am I at thinking through what will

happen to me and other people before I
do something?
Facing up to consequences
123
data
collection
tools
tool 7
outcome
assessment:
participant
self-report
charities
evaluation
services
124
tool 8
telephone interview guide
Note: If the named person is not available,
find a time to call back. Do not explain the
reason for the call to a third party, or ask
people to call back.
Once the named person is on the phone,
introduce the evaluation:
My name is Grace Kamwa. I am working
on behalf of the National Advice and
Information Service to evaluate its service.
I am calling because you kindly agreed to
allow us to contact you by telephone for
feedback on the service you received when
you contacted the advice line last month.
1 Firstly, was the advice line the first place

you contacted with this query?
Yes
No
Dont know/cant remember
2 If you did try other places first,
which were they?
Check that the time is convenient or offer

to call back at a more convenient time.
Reassure them that all information is
anonymous and confidential.
3 Did you contact the advice line on your

own behalf or for somebody else?
Own query
For a friend/relative
In a professional capacity
Other (what?)
4 What prompted you to contact the

advice line? (if not already given)
5 How do you feel the advice worker

dealt with your enquiry?
How polite were they?
Very polite
Quite polite
Not very polite
Rude
Dont know/not sure
charities
evaluation
services
6 How well did they understand

your situation?
Very well
Quite well
Not very well
Not well at all
Unsure
11 Overall, have you been able to act on any

of the information given? If so, what and
how? Has anything changed with regard
to [query] since you contacted the advice
line? Record details of response.
tool 8
telephone
interview
guide
7 How clear was the advice or information

the advice worker gave you?
Very clear
Quite clear
Not very clear
Very unclear
Unsure
8 How useful was the advice or information
given to you by the advice line?
Very useful
Quite useful
Not useful
Unsure
If not useful, go to Q 9.
If quite useful, go to Q 10.
If very useful or not sure, go to Q 11.
9 So you feel that the information given
to you was not useful. Why was this?
Tick the following from responses given:
Your questions were not answered
You did not understand the advice
or information
It was not relevant
The advice line was not the
appropriate organisation to help you
It was not accurate. If yes to this,
how do you know this?
8 So you feel that the information given
to you was quite useful. Is there any way
in which it could have been more useful?
125
data
collection
tools
Continue to probe for outcomes and

examples and whether these are
attributable to adviser input or written
information and complete the following
from responses:
Solved completely
Know where to go
Reassured
Other
12 Would you recommend the advice line
to a friend?
Yes
No
Unsure
If no or unsure, why not?
Thank you very much for all your
comments. They will be combined with
other responses and used by the advice
line to continue to improve its advice
and information services.
If the respondent specifically asks for

a summary of the findings, this can
be made available. Keep a record of
those requesting one.
charities
evaluation
services
126
tool 9
user interview schedule
This is an extract from an interview schedule used by
People First, an organisation run by and for people
with learning difficulties, as part of their user-led
evaluation. Cards were used with sad and happy faces,
which were helpful, but did not always work. A book
was also used with large prints of the pictures.
7
Getting on with staff/support workers
12
Did you meet the staff before they came to work here?
Yes
13
No
Dont know
Did you take part in interviewing the staff?

Yes
No
Dont know
charities
evaluation
services
14
15
How often would you like the staff to be here?
127
data
collection
tools
tool 9
user
interview
schedule
Do you get on with the staff?

Yes
No
Dont know
16
How do staff help you?

(eg: learning to cook; personal hygiene; talking
about problems)
17
If you think about the staff when you lived in hospital..

Is it the same here?
Is it better?
Is it worse?
charities
evaluation
services
128
tool 10
participatory learning
and action (PLA) tools
Card sorting
Things we liked
Things that were OK
Things we didnt like
Evaluation wheel
Evaluation of adventure weekend
Participants are asked to tick the things they liked.
Camp leaders
Cooking
Survival training
Canoeing
Making friends
Camping
charities
evaluation
services
Graffiti wall
Large sheets of paper are hung on the wall and participants are
invited to write comments on them during the activity or event.
129
data
collection
tools
tool 10
participatory
learning
and action
(PLA) tools
Timeline Kates Story

positive
change
charities
evaluation
services
130
further reading
Annabel Jackson Associates (2004), Evaluation
Toolkit for Voluntary and Community Arts in
Northern Ireland, Annabel Jackson Associates,
Bath.
Burns, S and Cupitt, S (2003) Managing
Outcomes: a Guide for Homelessness
Organisations, Charities Evaluation Services,
London.
Burns, S and MacKeith, J (2006) Explaining the
Difference Your Project Makes: A BIG Guide to
Using an Outcomes Approach, Big Lottery Fund,
London.
Charities Evaluation Services: Evaluation
Discussion Papers.
1 The Purpose of Evaluation (1998)
2 Different Ways of Seeing
Evaluation (1998)
3 Self-evaluation (1998)
4 Involving Users in Evaluation (1998)
5 Performance Indicators: Use and
Misuse (1998)
6 Using Evaluation to Explore Policy (1998)
7 Outcome Monitoring (2000)
8 Benchmarking in the Voluntary Sector
(2003)
9. Assessing Impact (2005)
Charities Evaluation Services (2008) PQASSO
(Practical Quality Assurance System for Small
Organisations), Third edition, London.
Clarke, A and Dawson, R (1999) Evaluation
Research: An Introduction to Principles, Methods
and Practice, SAGE, London.
Coe, J and Mayne, R (2008) Is Your Campaign
Making a Difference? NCVO, London.
Connell, JP and Kubisch, AC (1998) Applying a
Theory of Change Approach in Fulbright
Anderson, K, Kubisch, AP and Connell, JP (eds),
New Approaches to Evaluating Community
Initiatives, Vol 2: Theory, Measurement and
Analysis, The Aspen Institute, Washington DC.
Cracknell, BE (2000) Evaluating Development
Aid: Issues, Problems and Solutions, SAGE,
London.
Cupitt, S and Ellis, J (2007) Your Project and its
Outcomes, Charities Evaluation Services, London.
Davey, S, Parkinson, D and Wadia, A (2008)
Using IT to Improve your Monitoring and

Evaluation, Performance Hub, Charities
Evaluation Services, London.
Dewson, S, Eccles, J and Tackey, ND (2000) Guide
to Measuring Soft Outcomes and Distance
Travelled, Institute of Employment Studies,
Brighton.
Dixon, L and Aylward, N (2006) Youth Arts in
Practice: A Guide to Evaluation, National
Institute of Adult Continuing Education,
Leicester.
Ellis, J (2008) Accountability and Learning:
Developing Monitoring and Evaluation in the
Third Sector, Charities Evaluation Services,
London. Available to download at www.cesvol.org.uk
Ellis, J, Gregory, T and Wadia, A (2009)
Monitoring and Evaluation Resource Guide,
Charities Evaluation Services, London. Available
to download at www.ces-vol.org.uk
Feuerstein, M-T (1986) Partners in Evaluation:
Evaluating Development and Community
Programmes with Participants, Macmillan,
London.
Gosling, L and Edwards, M (2003) Toolkits: A
Practical Guide to Monitoring, Evaluation and
Impact Assessment, Save the Children, London
Guba, EC and Lincoln, YS (1981) Effective
Evaluation, Jossey Bass, San Francisco.
Hatry, HP, Cowan, J and Hendricks, M (2004)
Analysing Outcomes Information: Getting the
Most from Data, The Urban Institute,
Washington DC.
Hatry, HP and Lampkin, L (eds) (2001) Outcome
Management in Nonprofit Organisations, The
Urban Institute, Washington DC.
Kreuger, RA (1994) Focus Groups: A Practical
Guide for Applied Research, Second edition,
SAGE, London.
Kumar, R (1999) Research Methodology: A Stepby-Step Guide for Beginners, SAGE Publications,
London.
Latif, S (2008) Becoming More Effective: An
Introduction to Monitoring and Evaluation for
charities
evaluation
services
Refugee Organisations, Charities Evaluation

Services and the Refugee Council, London.
Evaluation: A Systematic Approach, Seventh

Edition, SAGE, California.
Marsden, D, and Oakley, P (1990) Evaluating

Social Development Projects, Development
Guidelines No 5, Oxfam UK, Oxford.
Sanfilippo, L (2005) Proving and Improving: A

Quality and Impact Toolkit for Social Enterprise,
new economics foundation, London.
McCrum, S and Hughes, L (1998) Interviewing

Children: A Guide for Journalists and Others,
Second edition, Save the Children, London.
Scriven, M (1991) Evaluation Thesaurus, Fourth

edition, SAGE, California.
Sudman, S and Bradburn, N (1982) Asking
Questions: A Practical Guide to Questionnaire
Design, Jossey-Bass, San Francisco.
McKie, L, Barlow, J and Gaunt-Richardson, P

(2002) The Evaluation Journey an Evaluation
Resource for Community Groups, Action on
Smoking and Health Scotland, Edinburgh.
Sue, VM and Ritter, LA (2007) Conducting Online

Surveys, SAGE Publications, California.
Mebrahtu, E, Pratt, B and Lnnquist, L (2007)

Rethinking Monitoring and Evaluation: Challenges
and Prospects in the Changing Global Aid
Environment, INTRAC, Oxford.
Taylor, D and Balloch, S (eds) (2005) The Politics

of Evaluation: Participation and Policy
Implementation, The Policy Press, University of
Bristol.
new economics foundation (2000) Prove It!

Measuring the Effect of Neighbourhood Renewal
on Local People, new economics foundation,
London.
Taylor, M, Purdue, D, Wilson, M and Wilde, P

(2005) Evaluating Community Projects: A Practical
Guide, Joseph Rowntree Foundation, York.
Oppenheim, AN (1992) Questionnaire Design and

Attitude Measurement, Pinter, London.
Valley of the Sun United Way (2006) The Logic

Model Handbook, Valley of the Sun United Way,
USA.
Palmer, D (1998) Monitoring and Evaluation: A

Practical Guide for Grant-making Trusts,
Association of Charitable Foundations, London.
Van Der Eyken, W (1999) Managing Evaluation,

Second edition, Charities Evaluation Services,
London.
Parkinson, D and Wadia, A (2008) Keeping on

Track: A Guide to Setting and Using Indicators,
Performance Hub, Charities Evaluation Services,
London.
York, P (2005) A Funders Guide to Evaluation:

Leveraging Evaluation to Improve Non-profit
Effectiveness, Fieldstone Alliance and
Grantmakers for Effective Organisations, USA.
Patton, MQ (2008) Utilization-focused Evaluation,

Fourth edition, SAGE, California.
Wilkinson, D (ed) (2000) The Researchers Toolkit:

The Complete Guide to Practitioner Research,
RoutledgeFalmer, New York
Paul Hamlyn Foundation (2007) Evaluation

Resource Pack, London. Available to download at
www.phf.org.uk
Phillips, C, Palfrey, C and Thomas, P (1994)
Evaluating Health and Social Care, Macmillan,
Basingstoke.
Robson, C (2000) Small-scale Evaluation:
Principles and Practice, SAGE, London.
Roche, C (1999) Impact Assessment for
Development Agencies, Oxfam Publishing,
Oxford.
Rossi, PH, Lipsey, MW and Freeman, HE (2004)
PLA contacts
Hull and East Yorkshire Participatory Learning and
Action Network
c/o Community Development Company
The Community Enterprise Centre
Cottingham Road
Hull HU5 2DH
01482 441002
Institute of Development Studies
University of Sussex
Brighton BNl 9RE
01723 606 261
131
further
reading
charities
evaluation
services
132
glossary
There are some technical
terms that are difficult to
avoid because of their wide
use in voluntary sector
management today and,
more particularly, within the
context of monitoring and
evaluation. These are
explained below.
A
Accountability: how much
individuals or groups are
held directly responsible for
something, such as spending
or activities.
Accuracy: the extent to
which data and an evaluation
is truthful or valid.
Achievement: performance
by a project or programme
demonstrated by some type
of assessment or testing.
Activities: this usually
means the main things your
organisation does, often
the services it provides.
Auditing: checking that

certain standards are met and
controls are in place; this may
be done internally or by an
outside agency.
B
Baseline data: facts about
the characteristics of a target
group, population and its
context, before the start of
a project or programme.
Benchmark: comparison of
activities, processes or results
with those already achieved
by your organisation or by
another organisation
Bias: undue influence causing
a particular leaning towards
one view.
more conclusively any change

observed in the project
participants due to the
project.
Conclusions (of an
evaluation): final
judgements based on analysis
and interpretation of data.
Control group: a group not
receiving the project benefits,
but matching in all other
respects the group being
treated, used for
comparison purposes (see
Comparison group).
Correlation: a statistical
measure of the degree of
relationship between
variables.
Budget: an estimate of future Cost-benefit analysis:

estimate of the overall cost
income and spending.
and benefit of each
Business plan: a detailed
alternative product or
plan showing how resources
programme.
will be managed to achieve
Cost-effectiveness analysis:
the strategic plan.
determines the results of
C
projects and programmes in
non-monetary terms against
Case study: an intensive,
the
cost.
detailed description and
Aim: an aim tells everyone

why the organisation exists
and the difference it wants
to make.
analysis of a single unit,

such as a person, project
or programme within a
particular context.
Criterion, criteria:
standard(s) against which
judgement is made.
Anonymity: action to make

sure that the subjects of the
study or report cannot be
identified.
Client: person or group of

people receiving a particular
service or collection of
services.
Assessment: judgements
about the organisations
performance.
Coding: translation of a
given set of data or items
into categories.
Attitude: settled way of

thinking or behaviour.
Audience(s): individuals,
groups or organisations that
will or should read or hear of
the evaluation.
Collate: gather together in

an ordered and useful way.
Comparison group: a group
not receiving project benefits
which is studied to establish
Data: information gathered

for a specific purpose and
linked to evaluation
questions.
Design (of evaluation): sets
out approaches, methods,
data sources and timetable.
Dissemination:
communication of
information to specific
audiences.
charities
evaluation
services
E
Effective: having the results
or effect you want; producing
the intended benefits.
Efficient: producing the
intended results with the
minimum necessary resources.
Evaluation: involves using
monitoring and other
information to make
judgements on how an
organisation, project or
programme is doing.
Evaluation can be done
externally or internally.
Executive summary: a nontechnical summary report
which provides a short
overview of the full-length
evaluation report.
F
Facilitator: someone who
brings together and focuses
a discussion, encouraging
participation by group
members.
Feedback: presenting
findings to people involved
in the subject in a way
that encourages further
discussion and use.
Focus group: interview with
a small group focusing on
a specific topic.
Formative evaluation:
evaluation designed and
used to improve a project,
especially when it is still being
developed.
I
Impact: the effect of a
service on a wider society
than its direct users. This
can include affecting policy
decisions at government level.
Impact evaluation:
evaluation of the longer-term
effects of the project, relating
to overall purpose.
Implementation
evaluation: assessment of
programme or project
delivery.
Indicators: see Performance
indicators.
Informant: person providing
information both directly and
indirectly.
Informed consent:
agreement given, before an
evaluation, to take part or
to the use of names and/or
confidential information, in
the light of known possible
consequences.
Inputs: resources and
activities which are used in
the organisation to create
the services offered.
Instrument: tool used for
assessment or measurement.
Integrated: built into and
part of a process or system.
Intermediate outcomes:
outcomes achieved in the
short term, but linking to
longer-term outcomes.
Internal evaluation:
evaluation carried out by
the staff of the project
being evaluated.
Interpretation:
understanding what the
data means in relation to
evaluation questions.
Intervention: service or
activity intended to change
the circumstances of an
individual, group, or physical
environment or structure
Interview: directed face-toface discussion between two

or more people, which aims
to find out the answers to
questions.
Interview schedule: list of
questions or topics to be
covered during an interview.
L
Longitudinal study: a study
over a substantial period of
time to discover changes.
M
Management: the people
responsible for the
organisation; the techniques
they use to run the
organisation.
Measurement: finding out
the extent or quantity of
something.
Median: the point in a
distribution which divides
the group into two, as nearly
as possible.
Matrix: an arrangement of
rows and columns used to
display information.
Mean: obtained by adding
all scores and dividing by
the total number.
Methodology: details of
how the evaluation is
carried out.
Mission statement: a short
statement of the overall
aim or purpose of the
organisation, usually
concentrating on the
difference it wants to make
and defining the values that
it will work by.
Mode: the value that occurs
more often than any other.
133
glossary
charities
evaluation
services
134
Monitoring: routine and

systematic collection and
recording of information.
N
Needs assessment:
identification of the extent
and types of existing
problems, services available
and unmet needs.
O
Objectives: the practical
steps the organisation will
take to accomplish its aims.
Observation: direct
examination and noting
of processes, events,
relationships and behaviours.
Pilot test: a brief and

simplified preliminary study to
try out evaluation methods.
Prompt: reminder used by an

interviewer to obtain complete
answers.
Plan: a written description

of the steps the organisation
will take to achieve certain
things.
Proxy indicators: things you

measure or assess that will
indicate changes that cannot be
measured directly.
Policy: a clear statement

of intent about how an
organisation will behave
over certain issues.
Purpose: the reason for the

existence of an organisation
or an activity; the changes it
hopes to achieve.
Population: all people in

a particular group.
Primary data: data collected

specifically for the evaluation
study.
Qualitative evaluation:
evaluation or part of an
evaluation that is primarily
descriptive and interpretative.
Operational plan: the same

as a year plan (see Year plan).
Procedure: a written, up-todate statement of how things

are done, easily available to
those who need to know.
Outcome evaluation:
evaluation of the intended
and unintended effects of
a project or programme.
Process: the method, or stepby-step description, of how a

task or activity is to be done.
Outcomes: the changes

or benefits resulting from
services and activities.
Outputs: what the
organisation does; the
services it delivers.
Process evaluation:
evaluation of how the project
works, that is, its processes
and its activities or outputs.
Profile: the characteristics
of a group of people or an
organisation.
P
Participatory evaluation:
actively involving stakeholders
in the evaluation process.
Partnership: an arrangement
between organisations for
joint action.
Performance indicators:
well-defined, easily
measurable information,
which shows how well the
organisation is performing.
PEST analysis: analysis of the
political, economic,
sociological and technical
issues affecting the project.
Programme: where a number

of projects are funded or
operate within the framework
of a common overall aim.
Progress evaluation:
evaluation of the
implementation of a project
or programme in relation
to aims and objectives.
Project: a major task
involving several activities and
resources, which may need
special attention and control
(note: some organisations are
called projects).
Qualitative information:
see Qualitative evaluation.
Quantitative evaluation:
an evaluation approach involving
the use and analysis of numerical
data and measurement.
Quantitative information:
see Quantitative evaluation.
Questionnaire: a series of
questions listed in a specific
order.
R
Random sampling: selection of
a smaller number of items from a
larger group so that each item
has the same chance of being
included.
Ranking: placing things in
order; used to identify
preferences or priorities.
Recommendations: suggestions
for specific appropriate actions
based on evaluation conclusions.
Reliability: likelihood of getting
the same results if procedures
are repeated; therefore
genuinely reflecting what you
are studying.
charities
evaluation
services
Respondent: individual
providing information directly.
Statistics: numerical facts

systematically collected.
Review: assessing an activity

against a target or standard
to see how far you have
achieved what you intended.
Strategic plan: covers the

vision for the future of the
organisation and outlines the
steps necessary to achieve
this over a three- to five-year
period.
S
Sample: selection for study
of a smaller number of items
from a larger group.
Sample bias: error largely
due to non-response or
incomplete responses from
selected sample subjects.
Sampling error: where the
probability is that different
results might be obtained
from another sample.
Scale: presents respondents
with a range of possible
responses to a number of
statements.
Secondary data: data
collected for a different
original purpose.
Self-evaluation: when an
organisation uses its internal
expertise to carry out its
own evaluation; evaluation is
integrated into project
management.
Service: all the goods and
information you supply, and
things you do for your users
(and indirectly for purchasers).
Stakeholders: the people
who have an interest in the
activities of an organisation.
This includes staff, volunteers,
users and their carers,
trustees, funders, purchasers,
donors, supporters and
members.
Standard: an agreed level on
which to base an assessment.
Strategy: a planned way of

achieving long-term aims.
Summary: a short
restatement of the main
points of a report.
Summative evaluation:
evaluation designed to
present conclusions about the
merit or worth of a project,
taking into account processes
and changes effected by the
project.
SWOT analysis: an analysis of
the strengths and weaknesses
of the organisation, and of
the opportunities and threats
which surround it.
System: the way things are
done to achieve a result;
usually written down and
made available to those who
need it.
T
Target: something to aim for;
it is a countable or
measurable result.
Terms of reference: detailed
plan for an evaluation.
Treatment: particular project
or programme activities or
services.
Trends: show changes over
time; can be used to plan
future services.
Triangulation: looking for
evidence or taking
measurements from different
points to increase reliability
and validity.
Trustees: the people

responsible for controlling
the management and
administration of an
organisation; another name
for management committee
members.
U
Unanticipated outcomes:
a result of an activity, project
or programme that was
unexpected.
Users: people who use the
organisations services.
Utilisation (of evaluations):
making use of evaluation
findings and
recommendations.
V
Validity: extent to which an
instrument measures what it
intends to measure.
Values: principles and basic
beliefs about what really
matters, that guide how
things should be done.
Variable: any characteristic
that can vary.
Y
Year plan: a one-year
budgeted plan, outlining
objectives and targets for
the organisation.
135
glossary
charities
evaluation
services
136
other publications from

First Steps in Monitoring and Evaluation
(2002)
First Steps in Quality (2002)
Monitoring Ourselves, 2nd edition (1999)
Managing Evaluation, 2nd edition (1999)
Developing Aims and Objectives (1993)
Does your Money Make a Difference? (2001)
Your Project and its Outcomes (2007)
Evaluation discussion papers
PQASSO (Practical Quality Assurance System
for Small Organisations) 3nd edition
PQASSO CD Rom
PQASSO in Practice (2006)
Using ICT to Improve Your Monitoring
and Evaluation (2008)
For further details, please contact CES at:

4 Coldbath Square
London EC1R 5HL
t +44 (0) 20 7713 5722
f +44 (0) 20 7713 5692
e [email protected]
w www.ces-vol.org.uk
Practical monitoring and evaluation
4 Coldbath Square
London EC1R 5HL
+44 (0) 20 7713 5722
+44 (0) 20 7713 5692
[email protected]
www.ces-vol.org.uk
helping you do better what you do best
Practical
monitoring
and
evaluation
a guide for voluntary
organisations

Practical Monitoring and Evaluation: A Guide For Voluntary Organisations

Uploaded by

Copyright:

Available Formats

Practical Monitoring and Evaluation: A Guide For Voluntary Organisations

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Practical Monitoring and Evaluation: A Guide For Voluntary Organisations

Uploaded by

Copyright:

Available Formats

What is the aim of Charities Evaluation Services (CES)?

What is the aim of Charities Evaluation Services (CES)?

What services does CES provide?

What services does CES provide?

CSE COVER new:CSE cover 2005 07/09/2009 16:51 Page 1

Practical monitoring and evaluation

Charities Evaluation Services

helping you do better what you do best

CSE COVER new:CSE cover 2005 07/09/2009 16:51 Page 2

Charities Evaluation Services

pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 1

Developed and written

CES trustees, staff and associates

pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 2

Now, more than ever, organisations which

Dame Suzi Leather

pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 3

Practical monitoring and evaluation has

feedback for internal learning and planning,

Practical monitoring and evaluation also

Within each section, guidance is offered at a

For a short introduction to monitoring and evaluation,

A resource for PQASSO users

pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 4

There is more about this in the

Further reading relevant to this section.

pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 5

inside this guide

Evaluation and the voluntary sector

The partnership between monitoring and evaluation

Approaches to monitoring and evaluation

Challenges for voluntary sector evaluation

Section 5 Practical toolkit

Collecting and analysing data

Data collection tools 111

pages 1-6 new:practical monitoring* 03/09/2009 13:14 Page 6

organisations are more likely to have to

The partnership between

into monitoring, the fewer you should need

Are we doing the right things?

When and how will the information

The basic guidance in Practical monitoring

and voluntary organisations. In this model,

Evaluation has two main purposes:

The purpose of evaluation will change the

There may be some tension between these

Has the project worked?

Evaluation should not only answer questions.

Who is evaluation for?

Evaluation for learning and development

been personally involved in decision-making.

Quality and evaluation

Many voluntary organisations have developed

The quality and evaluation link

provides data and judgements

can give you data to help you show that

The use of self-evaluation techniques allows

When an organisation uses its own

Self-evaluation processes and findings need