Week 03 - Advanced M&E Concepts

Master of Science in
Project Management (MSPM)
PROJECT CONTROLLING
(MONITORING, EVALUATION & CONTROL OF PROJECTS)
WEEK 03:
ADVANCED M&E CONCEPTS
BASELINES, TARGETS AND PERFORMANCE
MONITORING INDICATORS
ADVANCED PROJECT MONITORING & EVALUATION
(RECOMMENDED READING)
The World Bank has been at

the forefront of promoting
good practices for practition-
ers based on its decades-long
experience in funding
development schemes across
the globe.
Though a slightly older work,
this handbook delivers excel-
lent insights on all aspects of
developing project & pro-
gram monitoring indicators.
(MONITORING INDICATORS: A DEFINITION)
Indicators are a central concept in

project and program (performance)
monitoring and evaluation.
A indicator is a variable whose

purpose is to measure performance
in connection with the attainment of
a pre-specified target – for example,
a project output, outcome or impact
objective.
(MONITORING INDICATORS: TYPOLOGY)
Project and program indicators

categories:
- Input Indicators
- Process Indicators
- Activity Indicators
- Efficiency Indicators
- Effectiveness Indicators
- Output Indicators
- Outcome Indicators
- Impact Indicators
- Sustainability Indicators
- Proxy Indicators
(MONITORING INDICATORS: QUANTITATIVE INDICATORS)
Some indicators can be expressed nume-

rically (i.e. quantitatively).
Examples:
- under-5 child annual mortality rate
- number of accidents at work over time T
- volume of project supplies per month
- traffic intensity per week
- rate of absenteeism per day
- customer complaints received each hour
- volume of freight handled every week
(MONITORING INDICATORS: QUANTITATIVE INDICATORS)
Quantitative indicators tend to be

the preferred category of indicators
on projects and programs.
This is probably because

practitioners feel that they are easy
to measure “objectively” and
observed differences in their
magnitude over time can easily be
determined and tabulated or visually
presented.
(MONITORING INDICATORS: SPECTRUM OF APPLICABILITY)
Quantitative indicators are widely

used in practice – and not only in the
context of project or program
management.
Other fields where quantitative

indicators find routine application
include the economy, agriculture,
industry, finance, investment,
external trade, employment, health,
education, demographics, climate
and the natural environment.
(EXAMPLES OF QUALITATIVE CONCEPTS)
 Level of Cooperation Such concepts are

 Stress, Tension virtually impossible to
 Trust quantify accurately. They
 Satisfaction are based on perception.
 Inter-Personal Relationships However, by using
 Organizational Culture techniques such as
 Adversity, Supportiveness expert judgment and
careful observation it is
 Religiosity
possible to assign them
 Enlightenment numerical values or rank
 Awareness them on a relative scale,
 Power e.g. none, low, moderate,
high, very high.
(MONITORING INDICATORS: GENDER & MINORITY SENSITIVITY)
In recent years indicator developers have

shown increasing sensitivity towards in-
clusion of groups often ignored or under-
represented, such as females or religious
and/or ethnic minorities. Examples:
(1): No. of (girl) students enrolled at school X

(2): No. of (women) workers in industry Y
(3): Percentage of female residents in district Z
(4): Number of Hindus teaching in district A
(5): Percentage of (Christian) voters in elections
(MONITORING INDICATORS: WHERE TO COLLECT DATA?)
Data is the foundation on which monitor-

ing indicators is based.
Data is basically of two types:
(1) primary data, i.e. data collected

directly from the stakeholders through
interviews, surveys and observation, and
(2) secondary data, i.e. already existing
data collected from various published
sources such as reports and statistics.
(MONITORING INDICATORS: ISSUES WITH DATA COLLECTION)
When obtaining data for monitoring

indicators it is important to keep in mind
that data has both quantitative as well as
qualitative considerations attached to it.
 Quantitative means that the data is in
sufficient amount to fulfill its intended
purpose.
Qualitative means that the data
satisfies robust quality standards (see
slides on indicator evaluation systems).
(MONITORING INDICATORS: IDENTIFICATION ISSUES)
Determining monitoring indicators may at

first glance appear as an excercise in trivia.
In practice it is a task fraught with
difficulty and requires experience and
much thought input!
To assist project and program practition-
er’s in the indicator development process,
a number of indicator evaluation systems
have been developed such as DOPA,
SMART and CREAM.
INDICATOR EVALUATION SYSTEM: DOPA
D DIRECT
O OPERATIONAL
P PRACTICAL
A ADEQUATE
INDICATOR EVALUATION SYSTEM: SMART
S SPECIFIC
M MEASURABLE
A ATTAINABLE
R RELEVANT
T TIME-BOUND
INDICATOR EVALUATION SYSTEM: CREAM
C CLEAR
R RELEVANT
E ECONOMICAL
A ADEQUATE
M MONITORABLE
BEWARE OF LOOKING AT NUMBERS ONLY!
No of Recorded Accidents at Project Site

Period 1 17
Period 2 45
Period 3 09
Period 4 22
Period 5 31
Period 2 > Period 5 > Period 4 > Period 1 > Period 3
The data indicates that period 2 has the highest number of recorded
accidents (i.e. is statistically the worst) while period 3 has the lowest
(i.e. is statistically the best). The data, however, does not indicate the
nature of the accidents – these can range from minor cuts, scrapes
and and bruises to loss of limbs and even fatalities. It is conceivable,
for example, that most or all of the accidents in period 2 are minor
whereas period 3 may include more serious cases.
BEWARE OF LOOKING AT NUMBERS ONLY!
No of Complaints Received by Project Team

Week 1 08
Week 2 21
Week 3 39
Week 4 27
Week 5 18
Week 3 > Week 4 > Week 2 > Week 5 > Week 1
The data indicates that week 3 has the highest number of complaints
received (i.e. is statistically the worst) while week 1 has the lowest
(i.e. is statistically the best). The data, however, does not indicate the
nature of the complaints – these can range from relatively insignificant
to very serious. It is conceivable, for example, that most or all of the
complaints received in week 3 are insignificant ones whereas more
serious ones may be recorded in week 1.
MONITORING INDICATORS:
DIMENSIONAL COMPLEXITY
X I I1
Ia
I2
Y Ib
I3
I4
Z I5
MONITORING INDICATORS: COMPLEX EXAMPLE
Possible Performance
Indicators
Assignments
(effectiveness, efficiency)
Work Inter-Personal Skills
Performance (Communication, Coopera-
of a Program tiveness, Negotiation & Conflict
Officer Management Abilities etc.)
Innovativeness & Flexibility

BRIEF EXCERCISE FOR COURSE PARTICIPANTS
Much has been said and written about

the state of higher education in Pakistan
and across the globe.
What indicators can be used to measure

the QUALITY of institutions of higher
education?
Outcomes, Indicators, Baselines & Targets
RESULTS-BASED
MONITORING & EVALUATION
WHY RESULTS-BASED M&E?
Results-Based project manage-

ment has attracted considerable
attention and interest over the
past few decades, especially in
the development sector.
RBM is perceived as being
consistent with good governance
and proper utilization of project
and program resources.
RBM is now a requirement of
most international organizations,
donor agencies and national
governments.
RELEVANCE OF RBM&E IN PAKISTAN
RBM has also emerged as a

priority for the Government of
Pakistan in connection with the
development interventions
included in the Planning
Commission’s vast program and
project portfolio.
The Planning Commission’s
Project Wing has a monitoring
and evaluation cell which is
entrusted with this task.
PROJECT / PROGRAM COMPONENTS
Project Results
Project outcomes and impacts may

PROJECT be positive and negative, and intend-
OUTPUTS ed and unintended. Sometimes
these are unforeseen, esp. when a
project is undertaken the first time.
PROCESSES /
+
ACTIVITIES
OUTCOMES
-
INPUTS IMPACTS
OUTPUTS
Outputs are the immediate and

direct results of a project or pro-
gram.
Outputs can be tangible (e.g. the
construction of a road, airport or
building) or they can be intan-
gible (e.g. developing computer
skills, generating awareness).
Outputs reveal whether a
project or program has achieved
what it intended to achieve.
OUTCOMES
Outcomes are the intermediate

and direct result of a project or
program on the intended target
groups. They are of crucial
importance in development
interventions.
Subsequent outcome analysis
(using performance indicators)
reveals whether or not the
intended project or program
outcome objectives have been
met, not met or exceeded.
IMPACTS
Impacts are the long-term

results of a project or program.
They are a direct as well as
indirect consequence of its
outcomes.
Like outcomes, impacts too are
deemed of great importance in
development interventions.
Impacts are usually positive but
projects or programs can result
in negative ones too (e.g. envi-
ronmental degradation).
RECOMMENDED READING
An insightful information
resource on “Results-Based”
project management in practical
perspective is the handbook pro-
duced by the Canadian Internat-
ional Development Agency CIDA
in December 2000.
The handbook demonstrates the
application of RBM on the basis
of 100 selected project examples.
PROJECTS AND CHANGE
Short, Medium & Long-term Costs & Benefits

Economic
(Fully, Partially & Non-Quantifiable)

Subproject 1
Project Objectives A, B, C ...
Project Social
Goal or End-
Subproject 2 Deliverable(s) Ecological
Subproject 3 Political
Results Technological
Subproject 4
Infrastructural
Subproject 5
Outputs, Legal
Outcomes &
Subproject N Impacts Institutional
Asssessment
Methodology
Change (intended & unintended) Other Areas
PROJECT OUTPUTS, OUTCOMES & IMPACTS:
ROAD EXAMPLE
Project Phase Project Life-Cycle
Initiation, Planning, Project
Output
Implementation and
Closure of the Project
Operations Phase
Economic/Financial Sphere
Short-term Investment, trade, local businesses and the
economy, tourism, employment, inflation,
Possible wealth accumulation and distribution
Outcomes
Medium-term & Impacts Social Sphere
(+ and -) Health and education, travel, migration,
Not Projects: crime, social relations, communities‘ outlook
Routine main- Long-term and values, accidents and personal injury
tenance/repair
Ecological Sphere
Modification Projects: Highway extension, widening, recar- Fauna and flora along the route, pollution
peting, construction of bridges, additional exit and entrance (land, air, water, noise), waste accumulation,
ramps, petrol stations and rest stops etc. processing and disposal
THE LOGICAL FRAMEWORK APPROACH
(THE LOGICAL FRAMEWORK METHOD)
The Logical Framework Method, developed by US-

AID around 1970, owes its existence to the
recognition that the benefits of development project
and program results were not sufficiently reflective
of the resources, time and effort which were
invested in them.
The LogFrame is usually presented as a 4 x 4 matrix

visually. It is a useful tool for documenting the
desired results of a project or program and has since
been adopted by most development agencies.
THE LOGICAL FRAMEWORK
Intervention Logic Indicators Means or Sources of Important Risks &
(objectively Verifications (of Assumptions
verifiable & Indicators)
measurable)
Development
Objective(s) /
Ultimate Goal
Project or Program
Purpose / Outcomes
Project or Program
Outputs
Project or Program
Activities
9 Steps of LFA
(Relevance, Feasibility, Sustainability)
 Analysis of the Context -environment/background
 Stakeholder Analysis – who should be involved – in
planning and implementing the project
 Problem Analysis/Situation analysis
 Objectives Analysis
 Plan of Activities
 Resource Planning
 Indicators/Measurements of Objectives
 Risk Analysis and Risk Management
 Analysis of the Assumptions
Relevance
1. Analysis of the Context - the project’s environ-
ment / background information
2. Stakeholder Analysis - who should be involved –
in planning and implementing the project
3. Problem Analysis/Situation analysis
4. Objectives Analysis
Feasibility
5. Plan of Activities
6. Resource Planning
7. Indicators/Measurements of Objectives
Sustainabilty
8. Risk Analysis and Risk Management
9. Analysis of the Assumptions
Success factors for good goal fulfillment

 Commitment of all parties with ownership and
responsibility
 Definite roles for all parties involved (division of
work & responsibilities)

 Specific links between activities and objectives
 capacity, the project group’s ability to deal with risks
 Flexibility to adjust processes if conditions change
 users /beneficiaries participation in and influence on
the planning
LogFrames in Practice:
Nutrition & Child Development
Project, Uganda
SIDA Drinking Water Project
Youth Economic Empowerment
Project, Pakistan
CREATING A PROJECT MONITORING &
EVALUATION SYSTEM
(RECOMMENDED READING)
Many insightful contributions to

the subject of project and pro-
gram monitoring and evaluation
have been made by the World
Bank over the years.
This acclaimed handbook which

was produced in 2004 provides
detailed guidance on how to es-
tablish a results-based monitor-
ing and evaluation system.
(THE TEN STEP MODEL)
(THE TEN STEP MODEL: STEP 1)
Implementing and sustaining a

results-based M&E system
cannot be done in the absence of
1 national, organizational and

individual commitment, interest
and capacity.
 Politics, Individual Roles &
Readiness Responsibilities, Institutional
Assessment Reforms, Bureaucracy and
Culture, Information Systems,
Training, Cost, Experience.
Outcomes reveal whether or not

a project or program has produ-
ced benefits for its stakeholders.
2 Outcomes are derived from (long

-term) goals and are the direct
consequences of a development
intervention.
Outcomes to  Participatory and Consultative
Monitor & Process for Consensus Attain-
Evaluate ment, Precise Formulations (no
mixing).
Indicators are the means used for

measuring progress towards
attainment of outcomes and con-
3 stitute the basis for all data col-

lection, evaluation and reporting
activities.
 Indicator Selection Quality
Key Indicators to Criteria (e.g. CREAM), Pros &
Monitor Cons of Using of Pre-Designed
Outcomes Indicators.
The baseline is the situation as it

stands prior to the development
intervention and change (i.e. out-
4 comes) which the project or pro-

gram seeks to bring about.
 Data Sources, Data Collection
Methods and Frequency, Data
Baseline Data on Collecting Entities, Cost, Analysis
Indicators & Reporting.
Targets are the desired outcome

which a development interven-
tion seeks to achieve. They are
5 formulated quantitatively (i.e.

with indication of magnitude,
time, location) and can be fixed
or range-based.
 Realistic, Time-Restrictedness.
Target Selection
Monitoring for Results

encompasses (a) implementation
monitoring (i.e. transformation
6 of inputs to outputs in processes

and activities) and (b) results
monitoring (i.e. outcomes).
 Alignment of implementation
Monitoring for and results, Performance Data.
Results
Based on monitoring, evaluation

systematically reviews the chan-
ges brought about by a develop-
7 ment intervention over time and

identifies deficiencies and issues,
what works and what doesn’t,
and provides suggestions for im-
The Role of provement or alternative strate-
Evaluations gies.
 Types of Evaluations, Charac-
teristics.
Reporting serves many functions

(e.g. accountability, education,
convincing, lobbying) depending
8 on the recipient.
 Communication Strategy,
Clarity & Precision of Reports,
Timing, Understanding the
Reporting Requirements of the Audience,
Findings Medium of Communication (e.g.
Written, Oral, Visual).
Using findings means that the in-

sights and knowledge generated
is used towards improving the
9 performance of development
interventions.
 Effectiveness, Efficiency,
Corrective Actions.
Using Findings
Implies long-term existence and

use of the results-based M&E
system. This is only possible
10 when incentive mechanisms are

strengthened and disincentive
mechanisms weakened.
 Ensuring Incentive Systems for
Sustaining the Stakeholders, Eradicating Hurdles
M&E System and Disincentives, External
Validation.
PERFORMANCE MONITORING &
EVALUATION PLANS
(PERFORMANCE MONITORING & EVALUATION PLANS)
All projects and programs necessarily

undergo a planning phase in which all
operational aspects are carefully
analyzed, assessed and documented
by the planning team.
Alterations to all or specific
component plans are possible over
time through version updates as and
when new relevant information
becomes available.
(PERFORMANCE MONITORING & EVALUATION PLANS)
A critical component of a project or program plan is

the performance monitoring & evaluation plan. This
document provides guidance to all key stakeholders:
- the Program/Project Context
- the level of M&E required
- why it is required and how much it will cost
- when to undertake M&E activities (frequency)
- how specifically they are to be undertaken
- by whom they will be undertaken
- how information will be disseminated, and
- what evaluation methodologies will be used.
(PURPOSE OF PERFORMANCE MONITORING & EVALUATION)
 Accountability
 Transparency
 Coordination
 Decision-Making Support / Corrective Actions
 Comparability
 Lessons learned
 Database
 Professional Practice
 Institutional / Donor Requirement
REVIEWING PM&E SAMPLE DOCUMENTS
Critically Review with AF

participants the few
selected sample PM&E
documents.
PARTICIPATORY MONITORING &
EVALUATION
(PARTICIPATORY MONITORING & EVALUATION: DEFINITION)
There are many definitions of participatory

monitoring and evaluation (PM&E).
The FAO defines it as: “the systematic recording and
periodic analysis of information that has been
chosen and recorded by insiders with the help of
outsiders”.
In the literature on development programs and

projects the term Participatory Appraisal
(Assessment), Monitoring and Evaluation (PAME) is
sometimes used.
The World Bank perceives participatory

monitoring and evaluation as
“a process through which stakeholders at
various levels engage in monitoring or
evaluating a particular project, program or
policy, share control over the content, the
process and the results of the M&E activity
and engage in taking or identifying corrective
actions. PM&E focuses on the active
engagement of primary stakeholders”.
Participatory Monitoring & Evaluation now

finds widespread application in program and
projects across the globe.
Some of the areas in question include
community development schemes, health,
education, sanitation, forestry, conservation,
water resource management.
There are many approaches for ensuring
participation, e.g. focus groups and
committees.
(MERITS OF PARTICIPATORY MONITORING & EVALUATION)
 Consistent with Good Governance

 Confers Greater Legitimacy
 Generates Stakeholder Acceptance
 Satisfies Sponsor Requirement
 Provides Opportunity for Feedback (Needs etc.)
 Improvement in Project / Program Implementation
 Helps Keep Track of Progress
 Show Effective Resource Utilization to Stakeholders
 Ensures Transparency
 Gives a Sense of Ownership
 Sign of Respect for Local Communities
 Encourages Creativity
(DEMERITS OF PARTICIPATORY MONITORING & EVALUATION)
 Unflinching Commitment
 Time Factor
 Experience Needed
 Training needed
 Trust Issues
 Ensuring Inclusiveness of Stakeholders
 Possibility of Politicization
 Getting Commitment to Assume Ownership
(Stakeholders)
 Stakeholder Commitment (Distractions of Daily
Routines)
 Comprehension Problems
 Apprehension by Project / Program Implementers
(Failure)
(THE WORLD BANK PARTICIPATION SOURCEBOOK)
In early 1996 the World Bank pub-

lished the World Bank Participation
Sourcebook which provides insight
into the Bank‘s experiences with
stakeholder participatory
approaches used in several projects
funded by it across the globe.
The Sourcebook includes case

Review Sourcebook with studies of projects undertaken in
AF participants
Europe, Asia, Africa, Latin America
& the Middle East.
THANK
YOU!

Week 03 - Advanced M&E Concepts

Uploaded by

Copyright:

Available Formats

Week 03 - Advanced M&E Concepts

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Week 03 - Advanced M&E Concepts

Uploaded by

Copyright:

Available Formats

Master of Science in

Project Management (MSPM)

The World Bank has been at

Indicators are a central concept in

A indicator is a variable whose

Project and program indicators

Some indicators can be expressed nume-

Quantitative indicators tend to be

This is probably because

Quantitative indicators are widely

Other fields where quantitative

 Level of Cooperation Such concepts are

In recent years indicator developers have

(1): No. of (girl) students enrolled at school X

Data is the foundation on which monitor-

(1) primary data, i.e. data collected

When obtaining data for monitoring

Determining monitoring indicators may at

No of Recorded Accidents at Project Site

Period 2 > Period 5 > Period 4 > Period 1 > Period 3

No of Complaints Received by Project Team

Week 3 > Week 4 > Week 2 > Week 5 > Week 1

Innovativeness & Flexibility

Much has been said and written about

What indicators can be used to measure

Results-Based project manage-

RBM has also emerged as a

Project outcomes and impacts may

Outputs are the immediate and

Outcomes are the intermediate

Impacts are the long-term

Short, Medium & Long-term Costs & Benefits

(Fully, Partially & Non-Quantifiable)

The Logical Framework Method, developed by US-

The LogFrame is usually presented as a 4 x 4 matrix

Success factors for good goal fulfillment

work & responsibilities)

 capacity, the project group’s ability to deal with risks

 Flexibility to adjust processes if conditions change

 users /beneficiaries participation in and influence on

Many insightful contributions to

This acclaimed handbook which

Implementing and sustaining a

1 national, organizational and

Outcomes reveal whether or not

2 Outcomes are derived from (long

Indicators are the means used for

3 stitute the basis for all data col-

The baseline is the situation as it

4 comes) which the project or pro-

Targets are the desired outcome

5 formulated quantitatively (i.e.

Monitoring for Results

6 of inputs to outputs in processes

Based on monitoring, evaluation

7 ment intervention over time and

Reporting serves many functions

Using findings means that the in-

Implies long-term existence and

10 when incentive mechanisms are