TOOL D14 Monitoring and evaluation: a framework 159

TOOL D14 Monitoring and evaluation: TOOL

a framework D14
For: • Commissioners in primary care trusts (PCTs) and local authorities
• Programme managers
About: This tool provides a framework for evaluating and monitoring local
interventions. It presents a 12-step guide on the key elements of evaluation,
an evaluation and monitoring checklist, and a glossary of terms.
Purpose: To provide local areas with an understanding of the basics of evaluating and
monitoring interventions.
Use: Should be used as a guide to plan and implement an evaluation and
monitoring framework for interventions to tackle obesity.
Resource: Passport to evaluation.203 See: www.homeoffice.gov.uk

When an evaluation of an intervention is undertaken, it is important that it is:

• planned
• organised, and
• has clear objectives and methods for achieving them.

There are three stages to the monitoring and evaluation framework:

1 Pre-implementation (planning)

2 Implementation

3 Post-implementation.

The diagram on the next page outlines the framework, with detailed information provided on

pages 160-170.

160 Healthy Weight, Healthy Lives: A toolkit for developing local strategies

A framework for evaluating and monitoring local interventions

Pre-implementation

Step One Step Three

Confirm objectives/ Step Two Establish performance indicators
expected outcomes and outputs Establish outputs and starting baseline

Step Six Step Five

Step Four
Formulate a timetable for Identify methods of
Identify data to be collected
implementation gathering data

Step Seven Step Eight (Optional)

Estimate the cost of Identify a comparable
planned inputs area

Implementation

Step Nine
Step Ten Implement intervention
Monitor progress and gather data

Post-implementation

Step Twelve
Step Eleven Report and disseminate
Analyse data results

Pre-implementation (planning)
Step One: Confirm objectives/expected outcomes and outputs
Objectives are the key to every successful programme and evaluation. Every evaluation is about
measuring whether the objectives have been achieved. Before starting the evaluation, local areas
must be clear about what the objectives are.

Unless you have a clear idea about what the project is trying to achieve, you cannot
measure whether or not it has been achieved.

A simple way to set objectives is to use SMART objectives:

• Specific – Objectives should specify what you want to achieve.
• Measureable – You should be able to measure whether you are meeting the objectives or not.
• Achievable – Are the objectives you have set achievable and attainable?
• Realistic – Can you realistically achieve the objectives with the resources you have?
• Time – When do you want to achieve the set objectives?
 TOOL D14 Monitoring and evaluation: a framework 161

The National Indicators of success can guide local areas in establishing intervention outcomes.
See Tool D5 for a list of indicators relevant to obesity.

Step Two: Establish outputs for the intervention

Outputs are the things that need to be produced or done in order to achieve the desired
objectives/outcomes. For example, if the intervention is to set up a local football club to increase
the amount of physical activity among children, the outputs might be: organise publicity for the
club in local schools and communities, employ and train volunteers, organise the location for the
club and so on.

Step Three: Establish performance indicators and starting baseline

Once your local area is clear about the objectives and outcomes of the intervention, the next step
is to think about how to measure the extent to which they have been achieved. Performance
indicators (PIs) are a means by which you can do this. They can be quantitative, which means
that they use statistical information to measure the effects of a piece of action. Or they can be
qualitative, which means that they measure things such as feelings and perceptions.

Performance indicators can use any information, from any source, that shows whether objectives
are being met. Obesity prevalence figures are quantitative PIs – they are a direct measure of the
degree of the problem in your area. Other PIs, such as those that measure parents’ perceptions of
their child’s diet, are qualitative. If an intervention’s objective is to educate parents in the target
clusters about healthy eating, qualitative PIs must be used to measure this.

When you are developing performance indicators, it is important to establish a starting baseline
for the intervention against which performance will be measured. Performance indicators are a
key part of any monitoring and evaluation framework, as they enable the measurement of what
actions have been achieved.

Key points
• Be clear about what you are measuring. Having a clear idea of what you are trying to achieve
will help in selecting the right indicators. Always ensure that the data required are available
and easily collected.
• Think about the context. Performance indicators may need to take account of underlying
trends, or the environment in which the intervention is operating.
• Performance indicators can never be conclusive proof that a project is successful; they can
only ever be indicators. This is because external factors, which have not been measured, can
have an impact on an intervention without a local area being aware of them. However, well
chosen indicators that come from a wide range of sources and illustrate different aspects of
an intervention can provide good evidence of its success.

Step Four: Identify data to be collected

The next step in the framework is to decide what data need to be collected to measure the
intervention’s success against the performance indicators. It is important to collect the right
information, at the right time and in the right format. Some questions to be asked at the
beginning are:

• What data are needed to calculate the performance indicators?

It is important to write a list of the data that might be available already, eg local GP lists,
health inequalities data, healthy lifestyle behaviour data, land use statistics, indices of
162 Healthy Weight, Healthy Lives: A toolkit for developing local strategies

deprivation, National Statistics Socio-economic Classification (NS-SeC) data, distance

travelled to work data, and so on.
• How much detail is needed?
The level of data required depends on what the data are going to be used for. Generally
speaking, detailed data help to pinpoint problems and provide an accurate picture of what
has happened, and higher level data are useful for showing general trends. Collecting and
analysing detailed data can be expensive and time-consuming, so plan ahead and only
collect as much as is needed.
• When and how often are data needed?
It is important to have data at the start of the intervention for comparison purposes and at
the end so that the long-term effects can be measured.
• What format are the data required in?
It is important to remember that data come in different forms because of different IT
packages. If the data are not in an accessible format, this may incur extra work to get it in
the right format. Think about the extra work and costs involved.
• Where do the data come from?
Data can come from many different sources, eg partner organisations, GP surgeries,
National Statistics, voluntary organisations, census information, and existing perception
surveys.
• Are the data available, accurate and reliable?
– Availability: If the data are not available, local areas may need to collect it themselves.
Some questions to ask are: Are the data vital to the evaluation? Are the time and cost
worthwhile? Will resources will available? (See Step Five – Identify methods of gathering
data.)
– Accuracy: This is vital. Some important questions to ask are: Is the sample of population
the data were taken from representative of the target population? Are the data record­
ed correctly? Did the analytical package used produce an accurate picture of the raw
data? Have data been collected objectively or has the collector introduced bias?
– Reliability: Some questions to ask are: Are the data available at the times required? Are
the data measuring the same or a similar thing to what you are evaluating? Are the data
current?

Step Five: Identify methods of gathering data

If data are not available or are not of sufficient quality or relevance, local areas may need to
collect data themselves. A selection of methods and techniques for collecting data is shown in the
table on the next page. These are provided to give local areas an idea of what methods are
available to them.
 TOOL D14 Monitoring and evaluation: a framework 163

Methods of gathering data

Method Typical Typical context of use Pros and cons
techniques
Surveys Interviews All-purpose. Easy to carry out.
Mapping Operational: mapping interactions Can produce large numbers of
Questionnaires between actors. responses.
Summative: user satisfaction; user Limited depth in questionnaire
impacts. surveys (more depth in
Learning: surveys of participants’ interviews and focus groups).
experiences. Good in outcome-linked
evaluations.
Field studies Observation All-purpose. In-depth data, giving insights
Task analysis Summative: how users respond to on social construction of
intervention. intervention.
Critical incidents
Operational: how institutional Time-consuming and skill-
Case studies intensive.
structures operate.
Diaries Difficult to utilise in outcome-
Learning: retrospective analysis of
what happened. linked evaluations.

Comparison of different settings.

Modelling Simulations Usually operational and learning Can predict possible outcomes
Soft systems modes. to adjustments in uncertain
Assessing organisational structure, and complex contexts.
dynamics and change. Sometimes highly abstracted.
Cost-benefit analysis. Requires high level of skill.

Optimisation of management
functions.
Interpretative Content analysis All purpose. Deconstruction of ‘hidden’
Used in operational (analysis of meanings and agendas.
meetings etc), summative (analysis Rich interpretation of
of materials or reports) and learning phenomena.
(deconstruction of programme Inherent risk of ideological
reports). bias.
Critical Discourse analysis More theoretical (usually critical As for interpretative methods,
theory) based than content analysis. but emphasises establishment
Typically used to assess structure, of generalisable laws.
coherence and value of large-scale Perceived to be unscientific,
programmes for learning purposes. especially by experimentalist
practitioners.
Participatory Action research Typically in developmental evaluation Encourages real engagement
mode. of subjects of intervention.
Good in highly uncertain
contexts.
Evaluators sometimes get too
involved in intervention itself.
164 Healthy Weight, Healthy Lives: A toolkit for developing local strategies

The table below summarises the broad types of interventions used in tackling obesity, and gives
some examples of evaluation questions and evaluation methods that would be associated with a
particular type of intervention.

Type of initiative Evaluation questions Evaluation methods

Awareness-raising Which cluster group(s) changed their attitudes towards Cross-sectional surveys
campaigns healthy eating and in what ways? Focus groups
How many articles were published in the local media Content analysis of media
and what was the content?
Public participation How can more people become physically active? Focus groups
Should GPs be providing more advice? Questionnaire surveys
Interactive events How many and what type of people attended the Exit polls
(outreach, theatre, event? Quota sample
demonstrations) How engaged was the audience? Analysis of attendance
In what ways did participants’ views of obesity change? records
Observation
Interviews
Education and How many healthcare professionals attended obesity Statistical analysis
training training courses? Questionnaire surveys
How many overweight and obese patients were Interviews
provided with advice by healthcare professionals?
Ongoing profile- To what degree and in what way is obesity covered in Content analysis of sample
raising popular media? of newspapers
What contribution does profile-raising investment make Citation analysis of
to obesity policy and improving the knowledge base? academic journals
Targeted access and Are minority ethnic groups more receptive to advice on Statistical analysis
inclusion actions healthy eating or physical activity than the general Questionnaire surveys
population?
Has this had an effect on the number of obese people
in the target cluster group?
Policy actions Has the implementation of the consultation exercise Focus groups
created new partnerships? Documentation
Analysis
Horizontal and How many schools are taking part in the National Child Statistical surveys
supporting actions Measurement Programme? Documentation
Analysis
Operational reviews Which public engagement approach is most cost- Process evaluation
effective? Cost-effectiveness
Analysis

Key point
Analysis requirements: Bear in mind that the selection of particular methods and techniques
also implies using the appropriate type of data analysis (which has its own resource and skills
implications). In general, large data sets (such as those derived from surveys) normally need
statistical software systems such as SPSS. Interpretative data (derived, for example, from content
analysis) can be analysed with proprietary qualitative software packages such as NVivo. In any
case, a clear coding frame to analyse such data is necessary.
 TOOL D14 Monitoring and evaluation: a framework 165

Step Six: Formulate a timetable for implementation

In order that the programme runs as smoothly as possible and meets deadlines, local areas should
put together a timetable of implementation. As a minimum, the timetable should:
• list all the key stages of work including milestones for key activities, eg football club to be set
up by (date)
• show the dates by which each stage needs to be completed
• show what resources are needed for each stage
• show who needs to be involved at each stage
• include milestones for regular review of the inputs and outputs, and
• be regularly updated to reflect any changes.

An example of a timetable grid for implementation is presented below:

No. Intervention Lead Inputs Outputs Outcome Baseline Performance Timetable

officer measures
1
2
3

Step Seven: Estimate the costs of planned inputs

Estimating the costs of planned inputs at the beginning of and during the intervention will enable
analysis of the cost-effectiveness of the intervention. Some examples of input costs are staff time,
publicity costs, equipment and transport costs, and use of leisure centre. It is important to review
input costs during the intervention to ensure that an accurate analysis of cost-effectiveness is
undertaken.

Step Eight (Optional): Identify a comparable area

Comparing changes in the intervention area with what is happening in another area is useful in
helping to establish whether any changes are a result of the intervention or could have happened
anyway. If local areas undertake this step, they should identify a comparison area (similar in size
and characteristics) not covered by the intervention so that a comparison at the post-
implementation stage can be undertaken. It is important to look at the wider area around the
intervention for comparison.

Implementation
Step Nine: Implement intervention and gather data
The following are some important aspects to consider for the implementation step of the
evaluation framework.
• Contingency planning: As with planning an evaluation in general, anticipating adjustments
and changes to data collection is to be encouraged. It is useful to have a ‘plan B’ with
alternative arrangements for data collection should it become apparent that, for example,
time, skills or operational constraints are likely to conspire against planned activities.
166 Healthy Weight, Healthy Lives: A toolkit for developing local strategies

• Triangulation: The evaluation should already have been designed with regard to the resource
requirements of the choices specified and with the ‘insurance’ of contingency planning in
mind. It is also worth noting that ‘insurance’ also has a methodological component:
triangulation. Triangulation means utilising different methods to cover the evaluation from
different angles (for example, assessing the effectiveness of organisational structures of an
intervention from the points of view of different actors).
• Operational rules: The evaluation should be able to track (and have a record of): what data
are being collected, who collects the data, and in what form and location the data are stored.
Clear rules about operational procedures should be set out and distributed to all those
involved in data collection and analysis. Similarly, it is useful to draw up ‘evaluation contracts’
with other stakeholders, especially those supplying information. These contracts should
specify the objectives of the evaluation and any guarantees that apply (for example, on
confidentiality).

Step Ten: Monitor progress

Make any necessary adjustments to implementation, structures and processes using the pre-
implementation steps.
• Monitor inputs.
• Monitor output and outcome data using the performance indicators identified.
• Monitor key milestones.
• Consider whether there are any core tracking data that do not relate directly to the inputs,
milestones, outputs or outcomes that it may also be useful to collect and monitor.
• Allow the results of the monitoring to dictate any changes to the ongoing implementation of
the intervention.

An example of monitoring the intervention would be: Keep a record of the resources used in
running the intervention, eg number of staff, who the staff are, how many hours staff work, and
costs incurred by the intervention.

Once a framework is established, those running the intervention monitor the data and
feed back the relevant information to the partnership.

Post-implementation
Step Eleven: Analyse data
Before analysing data, local areas need to ask the following questions:
• Are the data in the right format to apply to the performance indicators?
• Are there in-house facilities for analysing the data or do they need to be bought in?
• What methods of analysis are there?

Key point

It is important that data analysis is undertaken by an expert in statistical analysis.

Once the intervention has been implemented and data collected for evaluation, local areas
should:
• compare outcome data with the baseline
• calculate the cost-effectiveness of the intervention
 TOOL D14 Monitoring and evaluation: a framework 167

• calculate the costs of the intervention, including any inputs monitored during the intervention
• examine comparable areas
• examine trends in the wider area and any similar comparison area to assess the impact of the
intervention.

Step Twelve: Report and disseminate results

This step should be a continuation of the evaluation process. In this sense, it is important to give
those involved in the intervention being evaluated, as well as in the evaluation itself, and project
participants a sense of closure of the project and the evaluation, where appropriate, by running
concluding feedback events.

More generally, it is important to the reputation, value and impact of the evaluation to give final
formal feedback to everybody who has contributed in some way to the evaluation (for example,
by sending them a copy of the report or inviting them to a final feedback event).

Dissemination should not be restricted to the circulation of a final report – especially in the case of
developmental process evaluation. Different stakeholders may require different communication
approaches. These might include:
• short summaries of the evaluation, tailored to different audiences
• journal articles for other researchers
• topical articles in the ‘trade’ press
• workshops for specific audiences
• feedback seminars for key decision makers.

The results from the evaluation should always be fed back into the future planning of
interventions.
168 Healthy Weight, Healthy Lives: A toolkit for developing local strategies

Monitoring and evaluation framework checklist

Yes No Action
Pre-implementation
Step One: Confirm objectives/expected outcomes and outputs
Have SMART objectives been developed to show what the intervention is
trying to achieve?
Are outcomes in place to show what the final achievement of the
intervention will be? (This should relate to the overall aim.)
Step Two: Establish outputs for the intervention
Have outputs been established to show what tasks are being carried out to
achieve the outcomes (eg establishing a baseline, producing quarterly
reports)?
Step Three: Establish performance indicators and starting baseline
Have performance indicators been established, taking into account data
availability, surrounding environment and underlying trends of local area?
Has a starting baseline been established?
Step Four: Identify data to be collected
Has the source of data been identified to calculate the performance
indicators?
Do the data need to be collected?
Have the data been checked for accuracy and reliability?
Is extra work required to format the data for analysis?
Step Five: Identify methods of gathering data
Have the methods of data collection been agreed?
Have appropriate analytical methods been agreed?
Have statistical specialists been employed to complete the analysis?
Step Six: Formulate a timetable for implementation
Has an implementation timetable been formulated to ensure the
intervention runs and finishes on time?
Have milestones for key activities of the intervention been established?
Have milestones for regular review of the inputs and outputs been
established?
Step Seven: Estimate the costs of planned inputs
Have the input costs been estimated, to enable the analysis of cost-
effectiveness of the intervention?
Step Eight (Optional): Identify a comparable area
Has a comparable area been identified to ensure any changes are a result
of the intervention?
 TOOL D14 Monitoring and evaluation: a framework 169

Yes No Action
Implementation
Step Nine: Implement intervention and gather data
Has a contingency plan been organised?
Have operational rules been written and sent to all partners?
Step Ten: Monitor progress
Are the inputs being monitored?
Are the output and outcome data being monitored?
Are the key milestones being monitored?
Post-implementation
Step Eleven: Analyse data
Have the outcome data been compared with the baseline?
Has the cost-effectiveness of the intervention been calculated?
Have the costs of the intervention, including any inputs monitored during
the intervention, been calculated?
Has the comparable area been examined?
Have the trends in the wider area and any similar comparison area been
examined, to assess the impact of the intervention?
Step Twelve: Report and disseminate results
Have the results been disseminated to stakeholders in an appropriate
form?
Have the results been fed back into the future planning of interventions?
170 Healthy Weight, Healthy Lives: A toolkit for developing local strategies

Glossary
Aim A simple statement that sets out the purpose of the intervention.
Baseline The situation at the start of an intervention, before any preventive work has
been carried out. The information that helps to define the nature and extent of
the problem.
Evaluation Evaluation is the process of assessing, at a particular point in time, whether or
not particular interventions are achieving or have achieved their objectives.
Evaluation is about measuring the outcomes of a particular intervention. An
outcome is the overall result of an intervention. Evaluation can also be used to
measure whether the processes used in an intervention are working properly.
This is called process evaluation and it measures the inputs and outputs of an
intervention.
Input The inputs to an intervention are the resources used to carry out the work.
Resources can be financial, material or human.
Milestones Key points during the life of an intervention. They are decided at the planning
stage and can be time-based or event-based.
Monitoring The process of continually assessing whether or not particular interventions are
achieving or have achieved their objectives. Monitoring is also used to check
whether the processes being used are working effectively. Monitoring is carried
out throughout the life of an intervention, while evaluation is only carried out at
specific points in time.
Objective A statement that describes something you want to achieve – a desired outcome
of an intervention or an evaluation study.
Outcome The outcome of an intervention is the overall result of applying the inputs and
achieving the outputs.
Output A piece of work produced for an intervention. An output is not necessarily the
final purpose of an intervention. Outputs are usually things that need to be
done in order to produce the desired result. During the life of an intervention,
outputs are monitored to make sure they are being achieved on time and with
the resources available.
Performance The means by which you know whether or not you have achieved your targets
indicator (PI) and objectives. A PI is any information that indicates whether a particular
objective has been met. You can also use PIs that measure whether the inputs
and outputs in an intervention are working. For example, if a project is using
public meetings as one of its inputs, a PI could be used to measure the number
of meetings held and the number of people who attend each meeting. These
kind of PIs are called process PIs.
Process Process evaluation measures the inputs and outputs of a project.
evaluation
Programme A programme is a group or collection of interventions designed to achieve
particular objectives. The interventions in a programme are usually linked to a
particular problem or a particular area and fall under a common aim.
Qualitative PI PIs that measure qualities, which are usually quite intangible things, such as the
perceptions and feelings of individuals and groups.
Quantitative PI PIs that measure tangible things, such as the number of obese children in an
area.