Results Based Approaches to Program

Logic Models

CES Conference, June 4, 2006

Workshop facilitated by Harry Cummings, PhD.,

University of Guelph and Harry Cummings and
Associates, Guelph, ON

1
 Table of Contents
1. Workshop Objectives ............................................................................... 1
2. What is a Program Logic Model? ............................................................. 2
3. How do you prepare a Program Logic Model?......................................... 2
4. Case Study 1: Resource Jump Teams .................................................... 6
5. Case Study 2: Second Chance Alternative School Program.................... 15
6. Program logic Model Worksheet .............................................................. 19
7. Bonne Bay Primary Health Care Program Logic Model ........................... 20
8. What is an Evaluation Matrix?.................................................................. 27
9. Evaluation Matrix Worksheet ................................................................... 34
10. Bibliography and References ................................................................. 22

List of Tables
Table 1: Program Logic Model with Definitions and Hypothetical Example.. 4
Table 2: Narrative Summary of Program Logic Model – HCA example ....... 5
Table 3: Evaluation Matrix with Definitions and Hypothetical Example ........ 29
Table 4: Evaluation Matrix – HCA example.................................................. 30
1. Workshop Objectives
• To give participants a greater understanding of Program Logic Models and
how to develop them

• To link program logic models to customer satisfaction surveys in the rural

health environment

• What are you hoping to take away from this workshop?

1
2. What is a Program Logic Model (PLM)?
A PLM (refer to Table 1) is a tool to help design and evaluate programs. It is a
“picture” of the logical cause and effect relationships among four program
components: inputs, activities, outputs, and outcomes.

• Inputs are all resources which contribute to program activities

• Activities are events or deliverables supported with program inputs
• Outputs are the observable products of each activity
• Outcomes are the short, medium, or long-term changes taking place as a
result of the program. Long-term outcomes are often referred to as impacts.

The logical relationships can be understood as follows. The inputs must be

made available if the activities are to be done. Activities must be completed for
the outputs to be produced. Outputs must be produced and used if the outcomes
are to be realized.

The description of these program components makes up the first row of the PLM
and is called the narrative summary. The next row is a row of indicators of
performance or success, which is often the focus of monitoring and evaluation
efforts. A row that lists assumptions and risks involved in running a program is
the final row of the PLM.

The PLM in Table 1 contains explanations of what is to be included in each

column for each row and gives an example for clarity. The example used in
Table 1 is for an imaginary poverty reduction program. Although Table 1 does
not include a numbering system, it is often helpful to include one in order to
show, for example, which activity is related to which output. Table 2 is an actual
narrative summary portion of a PLM that was done as part of the Framework for
Evaluating the Quality Assurance Programs of the Colleges of Health
Professions in Ontario written by HCA.

3. How do you prepare a PLM?

The Narrative Summary

A typical place to start when making a PLM is with the narrative summary row
(Row A in Table 1). The narrative summary in each column contains only words
and does not attempt to indicate quantity or quality. Additional pages can be
attached to a PLM when more space for elaboration is required.

Performance Indicators

The indicators row (Row B in Table 1) is more complicated to complete.

Indicators seek to measure results and to provide evidence that progress is being

2
made toward the achievement of the goal. They are often the focus of monitoring
and evaluation efforts. Indicators should consider quantity, quality, and time.
Quantity refers to the number (e.g. the number of households being helped).
Quality refers to characteristics of the beneficiaries (e.g. low-income households).
Time refers to the time period in which the level of the indicator is to be achieved
(e.g. household will benefit in year 1).

A popular way to remember the characteristics of good indicators is the word

SMART.
S = Specific – is the indicator specific and clear?
M = Measurable – will you be able to collect the data?
A = Actionable – can you take action based on the findings of the indicator?
R = Relevant – is the information provided by the indicator necessary for decision
making?
T = Timely – will the indicator tell you what you need to know at the right time?

Indicators must be chosen carefully. There can be a limitless number of

indicators for any program. There are six criteria that can be used to select
which indicators to use.
1. Does the indicator measure the result as directly as possible?
2. Is the indicator a consistent measure over time?
3. Is the indicator sensitive enough to reflect changes in the attributes being
measured?
4. Is the information easy to collect and analyze
5. Will the information be helpful in decision making?
6. Are there sufficient funds to collect and analyze the data?

Assumptions and Risks

The last row of a PLM is for Assumptions and Risks. Assumptions refer to the
external conditions that must exist for the cause and effect relationships
expressed in the PLM to behave as expected. Risk refers to the probability that
the assumptions will not hold true. Risk is rated as low, medium, or high.
Assumptions and risks may apply to more than one column.

3
Table 1: Program Logic Model with Definitions and Hypothetical Example
Inputs Activities Outputs Outcomes
Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
(beyond life of the
program)
A Narrative A listing of the Events supported Completion of an activity Short term changes taking Medium term changes Long term societal
Summary human, financial, with program place as a result of the taking place as a result of changes taking
and physical inputs The directly observable program the program place as a result of
resources used or products of each activity the program
required to Major deliverables Shows the link to program Shows the link to program
implement the No attempt to show objectives objectives Shows the link to
program A description of changes program goal or
how the inputs are E.g. Improved access to E.g. Reduction in level of vision
E.g. Participating used in an E.g. Participation of employment through unemployment for
households organized manner households in job increased skills participating households Eg. Poverty
seminars reduction in City X
E.g. Job searching
seminars
B Performance The units used to The units used to The units used to The units used to measure The units used to measure The units used to
Indicators measure the measure what the measure the quantity, short term changes medium term changes measure long term
quantity, quality, inputs have been quality, and timeliness of impacts
and timeliness of used for each output E.g. # and type of jobs for E.g. % of graduates still
each input participants by month by working by type of job E.g. # of jobs
E.g. # of job E.g. # of households socio-economic status 2,3,and 4 years post created as % of total
E.g. # of low searching seminars participating in job program jobs in City X
income held by month seminars by month
households
registered by
month
C Assumptions Assumptions: The necessary conditions external to the program that must exist if the cause and effect relationships between program components is to
and Risks be as expected
Risks: The probability that the assumptions will not hold true

E.g. households have time to participate, medium risk

E.g. job seminar leaders are available and committed, low risk
E.g. no unexpected changes in employment opportunities, high risk
HCA’s recommended model: adapted from various sources

4
Table 2: Narrative Summary of Program Logic Model – HCA Example
Inputs Activities Outputs Short-term Long-term Outcomes Impacts
Outcomes

Narrative • Public involvement in • Development of • Member approval • Increased • Improved practice • Improved
Summary QAP development management and and support for the awareness of QAP relative to practice population
(though consultation implementation plans for QAP function and standards health status
and participation on the the QAP, including • Members process by • Improved standards of • Improved
QAC) resource allocations and participating in the members practice across the patient
• Human (staff, time lines QAP components • Members’ have professions outcomes
committee members, • Development of QAP • Practice standards increased • Increased bottom-line
consultants) and components (peer guidelines applied awareness of their performance of
financial resources for assessment or review; in the QAPs learning needs and professions as a whole
QAP development, continuing education and • Self assessment styles • Colleges have increased
promotion, professional development; profiles completed • Members aware of knowledge of effective
implementation and practice enhancement and • Peer reviews and compliant with QA activities
maintenance remediation, etc.) conducted practice standards • Improved public access
• Research, consultation • Member education and • Continuing and guidelines to competent
and literature search promotion of the QAP education • Key issue areas in practitioners
undertaken to develop • Development of QAP requirements practice are • Improved quality of
and mange data materials for members fulfilled identified across the health care provision
collection) • Implementation of the QAP • Data from members’ professions and (safe, ethical, client-
• Facilities and • Development of participation in QAP appropriate focused)
equipment to develop procedures and is collected and remediation
and manage the QAP administrative systems for analyzed mechanisms
(computer equipment, QAP implementation • Data collected and applied members’
software, office space) • Development of analyzed to incompetence is
• Knowledge and procedures and data establish base line identified and
capacity to carry out collection systems to performance of the addressed by
QAP monitoring and establish baseline data on profession Colleges
evaluation the profession • Improved
• RHPA Requirements: • Development and knowledge and skill
• Regulations to direct distribution of practice of members
and enforce the QAPs standards
• Functioning QAC • Collaboration / consultation
with other Colleges

5
4. Case Study 1: CONTEXT OF THE HURON HEALTHKICK PROJECT

As described in the HealthKick Huron project proposal (February 2005),

healthcare in rural Ontario is at a crossroads. Rural communities throughout the
province are facing chronic challenges in meeting the healthcare needs of the
resident population. Central to this issue is the persistent and critical problem of
attracting and retaining health care providers. While it is generally accepted that
the medical profession has specialized to the point where personnel such as
transplant surgeons, radiation oncologists or pediatric cardiologists need to work
in larger cities, most of rural Ontario is faced with a lack of healthcare personnel
to deal with basic, common health problems – family physicians, nurses – even
pharmacists.

The acute and persistent

shortages of health care providers ‘Health care is important to rural residents, but it also
in rural Ontario and the never- has an impact on the ability of rural communities to
ending struggle to attract and keep retain and attract investment. We want to make rural
them in rural communities has communities attractive places for relocation by skilled
workers and professionals, such as badly needed health
been well acknowledged for quite
care professionals. We’ll be looking to support
some time. Many small innovative projects that retain and enhance rural health
communities that had doctors 40 services. We’ll support strategies for attracting and
years ago have none today. Other retaining health care professionals.’
communities that had two or three John Gerretsen
now have one, and are concerned Minister of Municipal Affairs & Housing
about their chances of replacing Rural Ontario Municipal Association Conference
those professionals should they February 2004
retire or choose to leave.

The shortage of health care providers in rural Ontario has received considerable
attention from researchers, policy-makers and health care planners. Yet
recruitment and retention tools have to date been narrowly focused on providing
financial incentives such as signing bonuses, and more recently the
establishment of community-owned and operated turn-key health clinics. While
these recruitment initiatives are now viewed as necessary prerequisites, and may
work in some communities, capital is difficult to raise in small communities. At
best, bidding wars are costly and hard to win, even for larger population centres.
Furthermore, despite these efforts, the problems of attraction and retention of
health care professionals have not been resolved.1
1
In 2003, The Ontario Rural Council (TORC) commissioned a series of reports as part of a
Comprehensive Rural Health Human Resource Strategy for Ontario. One of these reports – A
Review and Synthesis of Strategies and Policy Recommendation on the Rural Health Workforce
by Raymond Pong and Noreen Russell makes several useful recommendations for rural
communities facing the challenge of ensuring the existence of adequate health care
professionals. The points as summarized in the original SHARP proposal are presented in
Appendix A.

6
The challenges described above point to the need for exploring alternative and
innovative efforts towards recruitment and retention of healthcare professionals.
A well-designed and comprehensive healthcare attraction and recruitment
strategy is needed if rural communities are to have access to adequate services
in the future. The HealthKick Huron project is responding to this opportunity and
using Huron County as a case study to assess the performance and impact of
the initiative.

The next section of the report introduces the evaluation approach that is being
used for the HealthKick Huron project.

7
The healthcare worker shortage is not exclusive to doctors. Registered nurses
account for approximately 35% of the entire healthcare workforce. In 2000, there
were 11,855 Registered Nurses practicing in rural Ontario. This represents an
8% drop from 1991. 75% of healthcare employers in Bruce, Grey Huron & Perth
state that they have a great deal of difficulty recruiting nurses. Furthermore, the
Canadian Nursing Association is suggesting that recruiting and retaining rural
nurses may become more challenging in the future given the reduced number of
individuals entering and graduating from nursing programs.

To compound the issue of a drop in rural healthcare professionals, the need for
healthcare services in rural Ontario is expected to increase. Rural populations
continue to grow at a rate of half a percentage point a year. But given
demographic trends common to many rural areas (youth out-migration, baby
boomer in-migration), the rural population will age at a significantly faster rate
than the province as a whole. In 15 years, the median age of rural Ontarians will
be 55. This in turn will have major implications for the demands placed on the
healthcare system in those communities.

For the purposes of the pilot, the area under examination is Huron County,
located in southwestern Ontario north of the City of London and west of Toronto
(Figure 3).

8
Figure 3: Huron County in Southwestern Ontario

Source: www.hurontourism.on.ca/PAGES/INFORMATION%20PAGES/findingus.html

As outlined in the February 2005 HealthKick Huron (SHARP) project proposal,

Huron County has many characteristics that make it a prime example of the
state of rural healthcare in Ontario today:

• Huron County’s current population stands at 60,000 people widely

distributed across a patchwork of small towns, villages, hamlets and rural
municipalities.
• Population centres include Clinton (pop 3,000); Exeter (pop. 4,500);
Goderich (pop. 7,500); Seaforth (pop 2,500); and Wingham (pop. 3,000).
Smaller communities include Bayfield (pop. 700); Blyth (pop. 1,100);
Brussels (pop. 1,000); Hensall (pop. 1,200) and Zurich (pop. 600).
• With a population density of just 17.7 residents per square kilometer, and
approximately 55% of County residents living in non-urban areas, the
population of Huron County is one of the most “rural” in Ontario.
• Huron County has a strong agricultural sector with over 18% of the local
labour force employed in agriculture and resource based industries – this
is 4 times greater than the provincial average.

9
 • As with many rural areas Huron County has an older than average
population. 15% of Huron County residents are over the age of 65,
compared to a provincial average of just 13%. Furthermore, the seniors’
population in Huron is expected to grow by 3% over the next decade.
• Huron County’s population is served by 5 community hospitals located in
each of the larger centers. The area has been designated under the
Province’s Underserviced Area Program. When the project proposal was
drafted Huron County had 15 vacancies for family physicians.2
• At the same time, a workforce crisis is looming in the nursing sector.
According to the Grey Bruce Huron Perth District Health Council, 74% of
Huron County’s Registered Practical Nurses are over the age of 50. With
a large numbers of impending retirements in the next decade, the nursing
sector is concerned about being able to replace the current workforce,
especially given that there has been a noticeable decline in the number of
new graduates.
• To make matters worse, the region’s healthcare sector was recently
delivered a severe blow with the announced departure of 6 doctors in the
Goderich area, resulting in 4,800 newly “orphaned” patients. Combined
with an estimated 2,000 orphaned patients in other parts of the County,
this leaves over 10% of the local population stranded without the services
of a family physician.

2
The most recent List of Areas Designated as Underserviced for General/Family Practitioners
shows a total of 11 vacancies in 6 different communities across Huron County: Clinton (1
vacancy), Exeter (1), Goderich (1), Huron East (2), Wingham/Listowel and Area (5), Zurich (1).
Source: Ontario Ministry of Health and Long-Term Care, Underserviced Area Program,
April/May/June 2006.

10
There are five major healthcare centres in Huron County:
• Clinton Public Hospital
• South Huron Hospital Association (Exeter)
• Alexandra Marine and General Hospital (Goderich)
• Seaforth Community Hospital and Health Centre
• Wingham and District Hospital

The nearest secondary and tertiary referral centres are located in London and
Stratford. Additional details on each of the five healthcare centres is provided in
Appendix B.

4.2 Goals and Objectives of the HealthKick Huron Project

The goal of HealthKick Huron is to develop local capacity in Huron County for
creating communities of choice for healthcare professionals in rural Ontario. As
part of this process, HealthKick Huron is testing innovative strategies for rural
healthcare recruitment. The initiative also proposes to develop a new partnership
of health care agencies, community economic development interests and
community volunteers. Specific objectives and related strategies of the
HealthKick Huron project are as follows:

4.2.1 Youth Engagement

Objective: To encourage local youth in the pursuit of health-related careers by

exposing them to a full range of employment opportunities found in the
healthcare sector.

ISSUE STRATEGIES PARTNERS TIMING

Youth Rural Health
Engagement Career Exploration
1. Rural youth have i. Provide relevant Avon Maitland 2005
become disinterested in career information and District School Develop website;
health related careers mentoring to at least Board; Huron attend career fairs
given a perception of 1000 high school Perth District
poor working conditions students in order to Catholic School 2006
and low pay – educate them about the Board; Foundation Attend career fairs
particularly in the rural range of rewarding for Enriching
sector. careers found within Education – Huron 2007
the rural healthcare Perth Attend career fairs
sector.
This will be
accomplished by
developing web-based
materials for use by
guidance counselors,
libraries; and
presentations to

11
 students.

ISSUE STRATEGIES PARTNERS TIMING

Youth Rural Health
Engagement Career Placements
2. Individuals that have ii. Provide Local Healthcare 2005
been raised in a rural opportunities for at Employers Work with partners to
area and have positive least 30 rural youth to (Clinics, establish ten
work experiences in experience working Pharmacies, appropriate summer
their communities are conditions in the local CCAC’s, LTC); placement positions;
much more prone to be health care sector HRC Centres for conduct follow-up
connected and locate through increased Students survey of work
there permanently. summer job placement students and
placements & employers at
internships. completion of
Schulich School of placements
Conduct MapQuest Medicine at the
Camp with University of 2006
approximately 24 rural Western Ontario. Work with partners to
high school students Huron County establish ten
from Huron and Perth Emergency appropriate summer
Counties. Management placement positions;
Committee conduct follow-up
survey of work
placement students and
employers at
completion of
placements; conduct
survey of MedQuest
Camp students at
completion of program
(July 2005)

2007
Work with partners to
establish ten
appropriate summer
placement positions;
conduct follow-up
survey of work
placement students and
employers at
completion of
placements

12
Youth Engagement Activities to Date

Career Information

HealthKick Huron has collected a variety of healthcare career brochures, videos

and posters from a variety of professional healthcare organizations. Interviews
have been conducted with local healthcare service providers and currently there
are 11 Huron County Healthcare Career Profiles on the HealthKick Huron
website with an additional 14 profiles to be added in the next few months.
Project staff attended the Huron Perth Career Expo in Stratford in October 2005
where over 2,000 students attended. Provided a healthcare career presentation
to St. Mary’s grades 7 and 8 with a nurse co-presenter.

Student Work Placements

Student work placements completed in 2005 included 3 placements with

pharmacies in Huron County for part time students and 5 placements with local
Chamber of Commerce in four communities to develop healthcare recruitment
materials.

Proposals are currently being reviewed for the 2006 round of rural healthcare
work placements. Up top 12 placement proposals will be approved to receive
funding of up to $3,000 each to hire a student in a healthcare related summer
job.

MedQuest

The Schulich School of Medicine and Dentistry’s Southwestern Ontario Medical

Education Network (University of Western Ontario) has developed MedQUEST: a
strategic collaborative program that encourages students from Southwestern
Ontario, an area with a substantial rural population, to apply in greater numbers
to medical school.

MedQUEST is a one-week summer program for Grade 10 and 11 students

where local physicians, nurses and medical/nursing students deliver a curriculum
to regional high school students. Students are nominated by their schools for
demonstrating academic skills, leadership qualities, and interest in the field of
medicine.

MedQUEST is designed to provide students with realistic experiences in the field

of medicine using state-of-the-art advanced simulator training, peer workshops,
and the opportunity to watch physicians/nurses in a clinical setting. Over the
course of the week students learn to deliver robotic babies, read x-rays, splint
and cast fractures, create a professional resume and gain an understanding of
the training requirements needed to become a physician. Early exposure to the
practice of medicine can help influence students' career plans and ultimately

13
increase the chance of successfully entering medical school. Similarly, these
types of exposures can influence students to choose a career in nursing.

A MedQuest Camp has been confirmed for Huron and Perth Counties for July
10-14, 2006. The Camp will be conducted in Seaforth. Approximately 24 grade
10 and 11 students from Huron and Perth Counties will participate. Meetings
were conducted with guidance counselor staff at both school boards to outline
student application procedures. Plans for a mock disaster exercise in 2006 are
also being coordinated with the Huron County Emergency Management
Committee.

Schulich has developed a number of evaluation processes that they will conduct
in relation to the activities offered at the MedQuest Camp including daily
questionnaires that are completed by the program participants.

14
5. Case Study 2: MICAH (MICRONUTRIENT AND HEALTH PROGRAM FOR
AFRICA)

The recent World Development Report which focused on ‘attacking poverty’, broadened
the definition of poverty to include the Health, Nutrition and Population (HNP)
dimensions and emphasized reducing exclusion, lessening vulnerability and providing
opportunity (World Bank, 2000). Adequate nutrition is a critical component of human
health. It is particularly critical for children under the age of five and mothers. This
study focuses on infant and child health, and nutrition, which are among the priority
topics in the MDGs and PRSPs. The Hidden Hunger Conference in 1991 and the
International Nutrition Conference in 1992 determined that the micronutrient
supplementation and fortification of foods was recognized as the most cost effective
strategy in combating malnutrition

Micronutrient And Health Program for Africa (MICAH) is a program that intervenes to
reduce micronutrient malnutrition and thus improve overall health for all family
members. MICAH targets women and children, who are the most vulnerable to
micronutrient deficiencies because of their increased nutrient needs. MICAH,
implemented under the coordination of World Vision Canada and funded by CIDA, has
produced positive results in terms of its goals and has reduced the burden of malnutrition.
However, a question that remains unanswered is did MICAH’s interventions actually
benefit the most disadvantaged groups? If so, how and if not, why not? How can
MICAH programming be more effective in this regard? The MICAH results have not
been analyzed with respect to poverty and this study provides an ideal opportunity to
carry out an incidence analysis. This study will assess the reach of micronutrient and
health programs to the most disadvantaged groups in Ethiopia and Malawi, two of the
five MICAH countries. An important reason for this study is to improve on MICAH’s
programming activities for the second phase of its program, which begins later this year.
This study will determine how well MICAH health and nutrition programs have reached
the poorest and most disadvantaged groups in Ethiopia and Malawi and the reasons for
their success or failure.

PROGRAM/POLICY DESCRIPTION
The Micronutrient And Health Program for Africa (MICAH) was established in 1995 to
reduce micronutrient malnutrition through integrated health and nutrition interventions.
MICAH’s goal was to improve the nutrition and health status of women and children
through the most cost effective and sustainable interventions.

The objectives and activities of MICAH include:

Increasing intake and bio-availability of micronutrients (Iodine, Iron and Vitamin A)
through supplementation, fortification, promotion of exclusive breast feeding, dietary
modification

Reduce the prevalence of diseases that affect micronutrient status (diarrhea, parasitic and
vaccine preventable) through water and sanitation education and infrastructure
improvement, immunization, malaria control, and treatment of worms and parasites.

15
Build local capacity for delivery systems to improve micronutrient statistics by –
equipping laboratories, developing the MICAH guide, training staff and influencing
national policy or compliance to policy.

MICAH was implemented in Malawi, Ethiopia, Ghana, Senegal and Tanzania and
completed its first Phase in 2000. The projects have had a profound impact on the target
communities in these countries. Some of these results, for the two countries selected, are
discussed below.

From the baseline survey of 6004 households, the micronutrient needs of 20 districts
representative of Ethiopia were identified. Through MICAH’s interventions including the
provision of Vitamin A capsules, community education and community gardens, VAD
was significantly reduced. A follow-up survey of 6573 households showed Bitot’s spots
decreased from 6.4% to 1.4% in pre-school children (MICAH, 2002). MICAH Ethiopia
supported diversification of food sources through promotion of backyard gardens,
provided training to health workers in Micronutrient and other health issues, promoted
construction of latrines, de-worming campaigns and provided access to safe water.
MICAH Ethiopia covered 20 project sites, worked with 6 partner agencies with a budget
of CAD$10 million. MICAH Ethiopia provided a national representation with 1.5
million beneficiaries. Harry Cummings coordinated the mid-evaluation plan, which
recommended that efforts be improved with respect to poverty targeting. The baseline
and follow up survey contain a variety of indicators that could be used for incidence
analysis. This analysis remains to be done.

Malawi was MICAH’s second largest program with a budget of CAD$8.2 million and
benefited 1.8 million people (almost 20% of the total population). Baseline and follow-
up surveys, available to the researchers (approximately 2300 MICAH households
surveyed at baseline and follow-up as well as 2300 non-MICAH households), suggested
the MICAH interventions reduced stunting (low height-for-age) from 56% to 40%. The
baseline and follow up survey contain a variety of indicators that will be used by this
proposed research to analyze poverty incidence. MICAH Malawi reduced anemia caused
by iron deficiency through supplementation, fortification and treatment of parasitic
infections. Furthermore, new water sites were provided, pit latrines constructed and
children were treated for hookworms and schistosomiasis. MICAH Malawi advocated
for monitoring and enforcing of iodized salt legislation through provision of equipment
and capacity building, and installed fortification facilities in 6 community flourmills.
MICAH’s program in Malawi (and Ethiopia) combined with the data from surveys and
field checks could be used to analyze the impact on the poor compared to the better off.
3. Literature Review/Hypothesis Statement

Malnutrition hinders the achievement of other human development goals and is closely
linked with poverty. The correlation between malnutrition and extreme poverty has been
demonstrated utilizing a region specific analysis. Just as low income is a contributing
factor to poor health and malnutrition; poor health, malnutrition and large family size are
key reasons for the persistence of poverty. Malnutrition is the manifestation, cause and

16
consequence of poverty, and has intergenerational effects. In Tanzania and Pakistan, the
poorest income quintiles had the highest rate of underweight and stunting. Factors
associated with income poverty also contribute to malnutrition – e.g. female illiteracy,
poor sanitation, lack of access to health care, and food insecurity. A study in Lagos,
Nigeria, demonstrated a high prevalence of protein-energy malnutrition in children whose
families had low incomes and lived in single room dwellings with limited access to water.
Furthermore, low literacy rates of mothers was also another factor that attributed to high
malnutrition levels. Anthropometric indicators such as stunting and wasting can be
correlated with poverty. These measurements are standardized, objective, inexpensive to
measure and easily comparable. The anthropometric indicators and poverty indicators
have been collected for the MICAH project and will be correlated as part of the research
proposed here. Nutritional status can further be used to assess development programming
needs and can be used as a targeting mechanism as demonstrated in Ecuador for poverty
mapping and in Honduras for a food coupon program. This study will help assess the
reach of HNP programs to the poor, their impacts and the reasons for their successes or
failure.

Evidence from Mexico suggests that programs that are more carefully targeted toward the
extreme poor are more successful, even though the targeting may cost more. Simulated
transfer schemes using household data for Venezuela, Mexico and Jamaica indicated that
targeting of interventions to narrowly defined geographic regions helped reduced poverty.
This was compared with transfer schemes involving no targeting, such as general food
subsidies. Resources and capacities are limited and hence it may be necessary to target.
Poverty occurs at household and community levels hence it may be necessary to assess
the reach of interventions at these levels. Household surveys and discussions with the
community may assist in determining why interventions do not reach the extreme poor.
Allocative efficiency and equity are important considerations for channeling resources to
the poor. People with more assets such as access to land and livestock (both variables
collected in MICAH surveys) may have greater access to nutrition programming
activities. Health and nutrition programs are usually targeted toward the poor or most
disadvantaged groups but for many reasons, the most disadvantaged do not take full
advantage. This study will correlate poverty status (land ownership, household assets,
livestock, housing and other socio economic characteristics, anthropometric indicators)
with MICAH nutrition programming initiatives (community gardens, health and nutrition
education, micro nutrient supplementation initiatives) and seek to determine the reasons
for the correlations observed.

There are several reasons why HNP interventions do not reach the poor. In a study
conducted in rural Ethiopia, heavy workloads, lack of access to health services,
traditional practices, poverty, social status and decision-making power were the factors
that affected the health of women in Bujatira. In Malawi, the burden of AIDS is
impinging upon household resources thus making the poor more vulnerable. One of the
reasons for poor reach or inadequate impact of HNP program on the poor, is that many
programs target children who go to school. The school attendance group is biased toward
the “relatively better off” families.

17
The landless or those who are marginalized tend to get neglected from development
interventions. Furthermore, within the beneficiary groups there is a wide range in the
income status of those reaping the benefits. In India benefit-incidence analysis
demonstrates that government health service expenditures benefit the rich considerably
more than the poor.

This study will seek to determine how well health and nutrition interventions are reaching
the disadvantaged groups and if they are not the reasons for this. The research hypothesis
to be tested is that MICAH programming participation and impact is correlated with
household poverty status. The sub research hypothesis is that poor households are less
likely to benefit from MICAH initiatives than better off households.

The analysis is to be completed using the baseline and follow-up survey data from Phase
1 MICAH for MICAH and non-MICAH areas as well as focus groups with selected
participants from field sites in Ethiopia and Malawi.

Baseline and Final Survey Data of MICAH and

Non-MICAH Areas
P
h
a
s Correlate Data as per Hypotheses Focusing on Poverty Indicators
e and MICAH Participation and Impact Indicators
1

Data from National Statistics and other available data.

Focus Group Discussion with MICAH participants – Determine

P characteristics of poverty. Identify poor and better-off groups.
H Key Informant Interviews
A
S
E FGD with MICAH
participants -
2 Better-off

FGD with
Non-Micah
group
FGD with MICAH
Key
participants – Poor
Analysis

18
6. Program Logic Model Worksheet
Use either case study to fill out this worksheet
Inputs Activities Outputs Outcomes
Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
(beyond life of the
program)
A Narrative
Summary

B Performance
Indicators

C Assumptions
and Risks

HCA’s recommended model: Adapted from various sources.

19
 Page 1 of 7
7. Bonne Bay Primary Health Care Project: Project Logic Model (PLM) – September 2004
Bonne Bay Primary Health Care Project: PLM Narrative Summary (1)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
• Dept. of Health and Community • OPHC consultations with Project • PHC Team and PHC Network enhanced • Increased understanding of • Enhanced satisfaction of • Improved health status for
Services - Office of Primary Health Coordinator and local PHC working • Participation of PHC Team members in PHC Team goals and PHC professionals the residents of the Bonne
Care (OPHC) group monthly team meetings objectives Bay region
• Primary Health Care Advisory • Develop PHC Team • Participation of PHC Team members in • Increased understanding and • Increased efficiency of
Council • Conduct monthly PHC Team meetings team building conferences knowledge of the role and health care system • Increased self-reliance
• Evaluation Advisory Group • Develop inventory of job descriptions • Inventory of job descriptions for PHC ability of each PHC Team among community
• Working Groups (Wellness, Scope for PHC Team members Team members member • Increased participation by members in regards to
of Practice, etc.) • Develop practice protocols and referral • Formal practice protocols and referral • Increased provider client/patient in decisions health care
• Western Health Care Corporation processes processes participation in PHC related to self, family, and
• Health and Community Services • Develop conflict resolution processes • Formal conflict resolution processes planning, implementation community programs
Western Region • Develop and conduct PHC orientation • Participation in PHC orientation sessions and evaluation processes
• Western Regional Wellness sessions for new employees for new employees • Increased support provided • Increased community
Coalition • Conduct PHC Team building activities • Participation of PHC Team members in to individual Team members satisfaction with health care
• Local Medical Advisory • Assess interdisciplinary training needs interdisciplinary training sessions • Improved communication access and quality of health
Committee (e.g. ISSP training for PHC Team • Participation of PHC Team members and between PHC Team care
• Building Better Tomorrows members working with children and PHC Network members in consultations members
• Community profile information youth under the age of 21; education and • Improved coordination of
(Community Health Needs and training for ambulance services) intervention services
Resources Assessment, Profile of • Consultations between PHC Team and • Enhanced scope of practice
Youth in Western Region, PHC Network for PHC Team members
Community Accounts, etc.) • Develop Internet based inventory of • Increased involvement of
• PHC Project Coordinator various programs, services and referral community and intersectoral
• Health Care service providers processes of the PHC Team partners in planning and
(physicians, nurse practitioners, • Develop working groups to address • Internet based inventory of various delivery of programs.
community health nurses, coordination of specific services and programs, services and referral processes
dieticians, physiotherapists, mental initiatives (e.g. youth services, diabetes of the PHC Team • Increased community
health counsellors, addictions management, cervical screening) • Working groups to address coordination of participation in PHC
counsellors, child youth and family • Develop Terms of Reference for specific services and initiatives initiatives/programs
services social workers, etc.) working groups • Terms of Reference for working groups • Increased participation by
• Primary Health Care Network • Establish the Community Advisory established client/patient in decisions
• Other key stakeholders (Personal Committee (CAC) • Reports/recommendations submitted by related to self, family, and
Care Homes, School District 3, • Develop CAC Terms of Reference the working groups community programs
Tree House Family Resource • Develop PHC orientation and team • CAC established with Terms of Reference
Centre, RCMP, Private Ambulance building activities for CAC members • PHC orientation sessions and team
Operator, etc.) • Conduct monthly CAC meetings building activities provided to the CAC
• Community Members/Partners • Establish communication structure • Participation of community members in
• Clients/Patients between CAC and PHC Team monthly CAC meetings
• Health Care Facilities, equipment, • Input, feedback, and recommendations
supplies, etc. from the CAC to PHC Team
• Traveling clinics
• Financial Resources

20
 Page 2 of 7
Bonne Bay Primary Health Care Project: PLM Narrative Summary continued (2)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
• Develop PHC promotional materials and • Articles written and published in local • Increased community
public awareness campaign for the newspapers highlighting PHC programs awareness and knowledge of
community and health issues PHC services/programs
• Portable PHC information board provided in the Bonne Bay
developed and displayed at community region
events
• Public information sessions on the PHC
Team conducted by PHC Team
members/Project Coordinator
• All correspondence formally identifies
project areas as part of PHC
• Increased community
• Develop Terms of Reference for Local • Local Wellness coordinator hired participation in PHC
Wellness Coordinator • Participation of PHC Team members and initiatives/programs
• Develop workshop to establish local community partners in a one-day
wellness coalition workshop to develop a local wellness
• Identify wellness initiatives (e.g. coalition
physical, mental, youth, seniors health • Local Wellness Coalition established
programs) • Wellness initiatives identified, planned and
• Identify funding sources to support implemented
wellness initiatives • Funding secured to support wellness
• Recruit volunteers to assist with initiatives
delivering wellness initiatives • Volunteer base for Wellness initiatives
established
• Increased access to and
• Introduction of Broadband • Broadband communication available in all effective use of video-
communication capacity PHC sites conferencing by PHC Team
• Installation of video conferencing • Video conferencing equipment available in members
equipment all PHC sites
• Increased effectiveness of
• Register the PHC Project population • Bonne Bay region population registered technology in PHC Team
with the PHC Team with the PHC Team communications
• Meditech and Client Referral • All relevant service providers have access
Management System (CRMS) is made to Meditech and Client Referral
available to all relevant service Management System (CRMS)
providers • Electronic client/patient record established • Increased use of common
• Development of electronic client/patient (agreed record structure for file recording client/patient records
record and sharing of information)

21
 Page 3 of 7
Bonne Bay Primary Health Care Project: PLM Performance Indicators (1)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
• Number of paid OPHC person • Number of meetings between OPHC • Number of health care service providers by • Increased understanding of • Enhanced satisfaction of • Improved health status for
days invested in Bonne Bay Project and Bonne Bay Project within the 1st professional discipline participating on the PHC Team goals and PHC professionals (Score the residents of the Bonne
on an annual basis and associated and 2nd year of the Project PHC Team objectives (Score on TET on TET questions Bay region (Selected health
wage and benefit costs • Development of the PHC Team within • Number of PHC Team members questions A1,A4-A8,A10: A10,B13,C11,D8,E1-E3: status indicators: Increase
• Number of participants in the the 1st year of the Project participating in monthly meetings 2004 baseline results 2004 baseline results in the percentage of women
Primary Health Care Advisory • Number of PHC Team meetings • Number of PHC Team job descriptions compared to June and Dec. compared to June and Dec. having annual pap smears –
Council and number of meetings conducted on an annual basis listed in the Internet based inventory 2005 results) 2005 results ; Score on SPT baseline results compared
on an annual basis • Development of an inventory of job • Practice protocols and referral processes questions C1-C4: 2004 to 2005 results)
• Number of participants in the descriptions for PHC Team members formally defined by the end of year 1 • Increased understanding and baseline results compared to
Evaluation Advisory Group and within the 1st year of the Project and • Conflict resolution processes formally knowledge of the role and June and Dec. 2005 results) • Increased self-reliance
number of meetings on an annual updated semi-annually defined by the end of year 1 ability of each PHC Team among community
basis • Development of practice protocols and • Number and percentage of new employees member (Score on SPT • Increased efficiency of members in regards to
• Number of paid PHC Project referral processes within the 1st year of attending PHC orientation sessions on an questions A1-A4, B12: 2004 health care system (Score health care (Score on
Coordinator person days and wage the Project annual basis baseline results compared to on SPT questions CPST: 2004 baseline
and benefit costs • Development of conflict resolution • Number and percentage of PHC Team June and Dec. 2005 results) A6,A7,B9,B12: 2004 results compared to March
• Number of Health Care service processes within the 1st year of the members participating in team building baseline results compared to and Sept. 2005 results)
providers by professional Project activities on an annual basis • Increased provider June and Dec. 2005 results
discipline participating in the • Number of PHC orientation sessions • Number of PHC Team members by participation in PHC
Project (e.g. Primary Health Care offered to new employees on an annual professional discipline participating in planning, implementation • Increased participation by
Team, Primary Health Care basis interdisciplinary training sessions on an and evaluation processes client/patient in decisions
Network) • Number of PHC team building activities annual basis (Score on TET questions related to self, family, and
• Number of other key conducted on an annual basis • Number of health care service providers by A2,A3,A9: 2004 baseline community programs (Score
stakeholders/interest groups • Number and type of interdisciplinary professional discipline participating in the results compared to June and on TET questions D2,D6:
participating in the Project training sessions offered to service PHC Network Dec. 2005 results) 2004 baseline results
• Quarterly or semi-annual financial providers on an annual basis (e.g. ISSP • Number of partnerships established with compared to June and Dec.
statements (comparison of planned training, education and training for other stakeholders/interest groups and • Increased support provided 2005 results)
budget vs. actual) ambulance services) types of activities to individual Team members
• Number and type of consultations • Number of PHC programs and services (Score on TET questions C1- • Increased community
between PHC Team and PHC Network listed on the Internet site C11: 2004 baseline results satisfaction with health care
• Development of partnerships with other • Number of PHC Team members by compared to June and Dec. access and quality of health
key stakeholders/interest groups professional discipline accessing the 2005 results) care (Score on CPST: 2004
• Development of Internet based Internet inventory on a quarterly basis baseline results compared to
inventory of PHC programs, services • Improved communication March and Sept. 2005
and referral protocols within the 1st year between PHC Team results)
of the Project and updated on a quarterly members (Score on TET
basis questions B1-B13: 2004
baseline results compared to
June and Dec. 2005 results)

22
 Page 4 of 7
Bonne Bay Primary Health Care Project: PLM Performance Indicators continued (2)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
• Number and type of working groups • Number of PHC Team members by • Improved coordination of • Improved coordination of
established to address coordination of professional discipline and number of intervention services intervention services
specific services and initiatives (e.g. community partners by sector/interest involving PHC Team involving PHC Team
youth services, diabetes management, group participating in each working group members and network members and network
cervical screening) on an annual basis • Number of reports/recommendations providers (Score on providers (Number of
• Terms of Reference established for each prepared by working groups and submitted TET/SPT questions working group
working group to governing agency and/or senior D3,D4/B12,B16: 2004 recommendations submitted
• Number of meetings conducted for each management sub-committee on an annual baseline results compared to compared to the number
working group on an annual basis basis June and Dec. 2005 results) acted on as reported in the
• Enhanced scope of practice APR: year 1-2 baseline
for PHC Team members compared to year 3, 4, 5)
(Score on SPT questions A1-
A7, B1-B15, C1-C4: 2004
baseline results compared to
June and Dec. 2005 results;
Increase in the number of
ISSP meetings and ISSP
profiles submitted 2004-
• CAC established with Terms of 2005)
Reference within the 1st year of the PHC • Number of participants in the CAC by • Increased involvement of
Project gender, sector/interest group and place of community and intersectoral • Increased involvement of
• Number of CAC meetings conducted on residence partners in planning and community and intersectoral
an annual basis • Number of CAC members participating in delivery of programs (Score partners in planning and
• Number of CAC team building sessions monthly CAC meetings on TET questions D1,D4,D5: delivery of programs
conducted on an annual basis • Number of CAC members participating in 2004 baseline results (Number of CAC
CAC team building sessions compared to June and Dec. recommendations submitted
• Number of reports/recommendations 2005 results) compared to the number
prepared by the CAC and submitted to the • Increased community acted on as reported in the
• PHC promotional articles and PHC Team on an annual basis awareness and knowledge of APR: year 1-2 baseline
advertising developed within year 1 PHC services/programs results compared to year 3,
• Portable PHC information and display • Number of articles or promotional features provided in the Bonne Bay 4, 5)
board developed within year 1 appearing in local newspapers on an region (Score on TET
• PHC public information sessions annual basis questions D7: 2004 baseline
developed within year 1 (note: the • Number, type and location of events where results compared to June and
display board could be used in the portable display is used on an annual Dec. 2005 results; Increased
conjunction with these events) basis percentage of the population
• PHC signature logo developed for all • Number, type and location of PHC public registered with the Bonne
communications within year 1 information sessions conducted on an Bay PHC Team 2004-2005)
annual basis
• PHC signature logo applied to all
communications related to PHC project
areas within the 1st year of the project

23
 Page 5 of 7
Bonne Bay Primary Health Care Project: PLM Performance Indicators continued (3)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
• Development of Terms of Reference for • Wellness coordinator hired within the first • Increased community
Local Wellness Coordinator within year year of the PHC Project participation in PHC
1 • Number of PHC Team members by initiatives/programs (e.g.
• Development of one-day workshop to professional discipline and number of health and wellness
establish the Local Wellness Coalition community partners by age, gender, programs) as reported in the
within year 1 sector/interest group and place of residence APR: year 1 baseline results
• Local Wellness Coalition established attending the one-day workshop to compared to year 2 results;
within year 1 establish the Local Wellness Coalition Increase in program leaders
• Number of Wellness Coalition meetings • Number of Wellness Coalition members and volunteers as reported in
conducted on an annual basis participating in meetings the APR: year 1 baseline
• Number of wellness strategies identified • Number of wellness strategies results compared to year 2
on an annual basis implemented on an annual basis in results)
• Number and type of funding sources comparison to the number of strategies
identified to support wellness strategies identified
within year 1 and 2 • Number of citizens by age, gender,
• Development of volunteer base within education, income and place of residence
year 1 and 2 participating in Wellness initiatives
• Amount and type of funding (cash and in-
kind) raised to support initiatives on an
annual basis
• Number of volunteers recruited in year 1
by age and gender
• Number of new volunteers recruited in
year 2 by age and gender and number of
year 1 volunteers retained in year 2

• Number and location of PHC sites with or

• Introduction of Broadband without Broadband access by end of year 1, • Increased use of video
communication capacity in PHC sites in 2, 3 etc. conferencing equipment
Bonne Bay region by March 2006 • Number and location of PHC sites with or (Increase in hours of use and
(pending approval of a Broadband without videoconferencing equipment by reduction of travel costs as
initiative) end of year 1 and 2 reported in APR: year 1
• Installation of video conferencing baseline results compared to
equipment in PHC sites by March 2006 year 2 results)
• Increased effectiveness of
technology in PHC Team
communications (Score on
TET question B6: 2004
baseline results compared to
June and Dec. 2005 results)

24
 Page 6 of 7
Bonne Bay Primary Health Care Project: PLM Performance Indicators continued (4)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact
• Development of the PHC client/patient • Number and percentage of population • Increased use of common
roster within year 1 and 2 of the project registered with the Bonne Bay PHC Team client/patient records
• All relevant service providers have by age, gender and place of residence by (Score on SPT question B8:
access to Meditech and CRMS the end of year 1 and 2 2005 baseline results
• Development of electronic client/patient • Number and percentage of relevant service compared to 2006 results –
record providers using Meditech and CRMS by this timeline is contingent
the end of year 1 and 2 on the development and
• Number and percentage of PHC Team activation of a common
members using the common electronic patient record system)
patient record by the end of 2006 (this
timeline is contingent on the development
and activation of a common client/patient
record system)

25
 Page 7 of 7
Bonne Bay Primary Health Care Project: PLM Assumptions and Risks (1)
Outcomes
Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact

Assumptions:

• The majority of health care service providers in the Bonne Bay region are committed to pursuing PHC approach to service delivery.
• Improvements/adjustments will be made to current funding methods to provide fair remuneration to Fee for Service Physicians and other private service providers (i.e. for time and involvement in activities such as consultations,
case conferences, PHC Team meetings, interdisciplinary clinics, and investment involved in the transition to an electronic network).
• All health service providers will work collaboratively as custodians of client/patent records to ensure the confidentiality of files.
• Improvements/adjustments will be made to current funding methods for salaried employees to address PHC Team collaboration and communication time needs including flexible time arrangements and overtime where
applicable.
• Salaried employees work responsibilities will be redefined to include scheduled interactions between various providers (activities such as consultations, case conferences, PHC Team meetings, interdisciplinary clinics, etc.)
• Community members will take an interest in providing input throughout the process.

Risks:

• Loss of interest by local leaders could result in lost momentum.

• Integration of boards could impact the delivery of coordinated services.
• The past merger of institutional boards and of community boards with programs from Department of Human Resources and Employment are still evolving processes.
• Participation by service providers may change if an adequate funding method is not developed - could result in loss of providers.
• PHC staff turnover may disrupt and slow pace of adoption.
• The system introduced may not be sustainable beyond the funding horizon.
• Funding is not secured to introduce Broadband communications capacity.
• Sustainability.
• Geography – the Bonne Bay region has a small population for a large geographic area.

26
8. What is an Evaluation Matrix?
Evaluation matrices are a tool for systematically identifying evaluation questions,
indicators of success, and appropriate data sources and data collection methods.
Whereas the PLM gives a picture of the whole program, the matrix indicates the
aspects of the program that will be the focus of the evaluation. Completing an
evaluation matrix will ensure that all necessary issues will be covered. Table 3 is
an example evaluation matrix that provides explanations for each column and
continues with the example of a poverty reduction program. Table 4 is an actual
evaluation matrix from a Framework for Evaluating the Quality Assurance
Programs of the Colleges of Health Professions in Ontario written by HCA.

Evaluation Issues and Questions

Some of the more common issues addressed in evaluations are rationale,

efficiency, effectiveness, access/reach, and impact. Each issue forms one
separate row in the matrix. Rationale refers to the extent to which a program
contributes to the overall goal or strategic direction of the organization. The
exploration of rationale considers the question “Is this the most appropriate way
to achieve the goal?” Efficiency refers to the extent to which program inputs were
supplied and managed and activities organized in the most appropriate manner
at the least cost to produce the outputs. It is the link between outputs and
outcomes in the PLM. Efficiency questions usually refer to the timeliness, quality
and quantity of the delivery of inputs in relation to the program plans and needs.
Effectiveness refers to the extent to which the program outputs produced
outcomes (or results) and thereby contributed to the program goal. It records
changes in the beneficiary group(s) that have happened as a result of the project.
It is the link between outputs and outcomes in the PLM. Effectiveness questions
examine whether the program outputs are being used in a manner consistent
with the program goals. Access and reach are concerned with determining
whether the project and its benefits are accessible and of benefit to all members
of the population and whether the community was given the opportunity to
participate in the development of the project. It applies to all columns of the PLM.
Impact refers to the long term and sustainable changes experienced as a result
of the project.

Indicators

Indicators seek to measure results and to provide evidence that progress is being
made toward the achievement of the goal. Please refer to the section on
indicators in the PLM section for more details.

Data Required and Sources of Data

The Data Required column contains the information that is required to determine
the indicator. The Sources of Data column contains the individuals,

27
organizations, documents, or reports from which the data will be obtained. A
data source should be specified for each indicator. Data sources should remain
consistent as switching sources may jeopardize the reliability of the data.

Method of Collection and Analysis

The Method of Collection column lists the methods and techniques that will be
used to gather the data that will measure results. Some examples include,
interviews, testimonials, participant observation, or document analysis. When
deciding on methods it is good to consider whether sampling techniques need to
be random or purposive, how the data collection instruments will be designed,
and what level of disaggregation of the data is required (e.g. gender, income
level, age). The Analysis column explains how the data will be analyzed. It is
important to consider whether it is necessary to compare disaggregated data,
what baseline data already exists, how several sources can be used to increase
validity, and whether it will be cost effective.

Responsibility and Timing

These columns are meant to ensure that those involved in the evaluation agree
on whom will do what and the time frame in which they are expected to do it.

28
Table 3: Evaluation Matrix with Definitions and Hypothetical Example
Issue Evaluation Indicator Data required Source of data Method of Analysis Responsibility Timing
questions collection
List the key List the key The units used What data will Individuals or How will the How will the Who will collect When will the
evaluation questions that to measure be needed to organizations or data be data be and analyze the collection and
issues that need to be quantity, determine the documents from collected? analyzed? data? analysis of data
need to be asked in order quality, and indicators which data is be carried out
addressed to determine timeliness of obtained E.g. survey of E.g. data will be E.g. John will or completed?
how the inputs, E.g. household disaggregated collect and
E.g. program is activities, employment E.g. participants into two parent Jane will E.g. Collection
effectiveness doing with outputs, and statistics and organization’s households and analyze done by 1
regard to the outcomes. household household one parent September 03 –
issue being registration registry, households and Analysis done
researched E.g. % change households compared to by 1 December
in employment pre-program 03
E.g. Have the for participating baseline data
program households
interventions
improved the
ability of low-
income
households to
have access to
employment?
HCA’s recommended model: adapted from various sources.

29
Table 4: Evaluation Matrix-HCA Example
Evaluation Key Questions Indicators Data Sources and Collection
Issue Methods
Rationale 1) Does the QAP follow the principles of the 1) QAPs contain measures to: a) identify 1) content analysis of College
RHPA and the Ministry Guidelines? deficient members, b) remediate members, c) QAPs
2) Is the selected mix of QAP components increase competence of members, and d) 2a) review of QAP background
likely to achieve the intended goals? address behaviour and remarks of a sexual documents and literature search; 2b)
3) Is the program acceptable to College nature peer review panel of independent
members? 2a) Literature/consultation research supports experts
4) Is the program flexible enough to meet the Colleges’ selection of QA methods; 2b) 3a) college tracking system connected
different learning style of members? independent experts support the methods with annual registration
5) Is the time investment for college members selected 3b) college tracking of member
manageable? 3a) % members complying with the QAP complaints
6) Does participation in the QAP interfere with 3b # of member concerns and complaints 3c) members’ survey or focus groups
members’ practice responsibilities? lodged with the College 4) content analysis of College QAPs;
7) Does the QA approach reflect current 3c) member satisfaction with QAP members survey or focus groups
principles of CQI? Is the QA approach still 4) existence of options for members’ 5) members’ survey; college tracking
relevant? participation system connected with annual
5) amount of time members spend annually in registration
QAP 6) members perceptions
6) members’ perceptions
Efficiency 1) Have the QAP regulations been passed? 1) Regulations passed 1) Regulations
2) Is the program being implemented as 2a) Implementation is on schedule 2a) And 3a) comparison of progress
scheduled in the implementation plan? # 2b) of QAP components implement3ed and to implementation plan
3) Have adequate resource been allocated for degree of implementation 2b) And 3c) and management
timely and efficient implementation of the 3a) implementation is on schedule consultation / program records review
QAP? 3b) actual expenditures correspond with 3b) Financial records/statements
4) Is there sufficient capacity in QAP planned expenditures 4a) And 4b) management and staff
management in the college? 3c) materials/ products produced as planned consultation
5) Have adequate data collection systems 4a) staff secured with experience and skills in 5) Administrative records;
been established? QAP management management consultation
6) Is the program affordable and sustainable 4b) staff and management’s perceptions of 6a) , 6b) and 6c) administrative
in the long-run? Colleges’ management capacity records
5) software and information tracking
systems/data base have been established
6a) cost of QAP / member
6b) % of QA P budget to overall college
budget
6c) increases in members fees

30
Evaluation Matrix -HCA Example Cont’d
Evaluation Key Questions Indicators Data Sources and Collection
Issue Methods
Effectiveness 1) Have college members been adequately 1a) # and % of members calling for further 1a) 1b) and 1c) college tracking and
informed and educated about the QAP information records system
2) Are members complying with and 1b) % of member not properly participating 2) college tracking system
participation gin the program? What are the 1c)variety and # of promotion methods used 2b) member survey; focus group
barriers to participation/compliance? 2a) % members complying with QP 3a) college tracking system
3) What members participated in the QAP and 2b) barriers identified 4a) test administration
in which components 3a) # of members participating in different 4b) and 4c) member survey
4) Have the clinical knowledge, skills, and QAP components 5) assessment records
judgement of practitioners increased? To what 4a) test scores from courses 6) re-assessment records
extent ware members applying this knowledge 4b) % of practitioners indicating increased 7) administrative / program records
to their practice? knowledge due to QAP 8a) and 8b) administrative / program
5) Is the QAP identifying deficient members? 4c) % of members applying new knowledge to records
6) Is remediation improving the skills of their practice 9) administrative / program records
deficient members? 5) # of deficient members identified 10) member survey
7) Are quality and/or deficiencies issues 6a) reassessment undertaken and 6b) # of 11) review of guidelines/standards
across the profession being identified? members found deficient, and non-deficient 12) member survey
8) Have monitoring systems been put in following remediation
place? Has QAP data been collected and 7) systems established to monitor aggregate
analyzed? issues and the nature of the profession as a
9) Are members being informed about the whole
findings, results and learning of the QAP? 8a) systems established 8b) data collected
10) Have adequate tools been developed to and analyzed
facilitate member participation in the QAP 9) information dissemination mechanisms
11) Have practice standards/guidelines been have been put in place
established? 10) members use of tools; members
12) Has members’ knowledge of standards satisfaction with the tools
and guidelines increased? 11) standards and guidelines developed
12) % members indicating increase in
knowledge and application of standards

31
Evaluation Matrix-HCA Example Cont’d
Evaluation Key Questions Indicators Data Sources and Collection
Issue Methods
Access and 1) Does the program involve clients and the 1a) Mechanisms are in place, and 1b) have 1a) And 1b) review of program
Equity public? been used for public feedback and records
2) Has consultation to develop and mange the involvement 2a) QAC and management
QAP involved members from diverse 2a) Diverse repres4entation on QAC; 2b) consultation
geographic areas, practice settings and diverse participation in consultation processes 3a) Review of program records
scopes of practice? Will improved patient care 3a) Existence of provisions for different forms 3b) Member survey or focus group
be available to all people in the province? of QAP participation, 3b) members perception 4) Program records
3) Do practitioners in different geographic of their access to the QAP 5a) , 5b), 5c) and 5d) review of
areas, different practice settings, and different 4) Leave provisions are in place and utilized assessment records and QAP
scopes of practice have equal access to all objective assessment criteria have been regulations
program components and to the benefits of established and are utilized
the program? 5a) Members have the right to ask for
4) Have provisions been put in place to alternate assessors
protect members on leave from practice 5b) Members are provided with results of their
5) Are members’ rights to fair and judicious assessments in a timely manner
assessment processes and assessors 5d) Member’s rights are clearly outlined in the
protected? QAP regulations
Impact 1) Has the QAP resulted in improved delivery 1a) Level of client satisfaction 1a) Client focus group; questionnaires
of care? 1b) Decrease in client complaints 1b) And 1c) college tracking records
2) Have there been any unplanned effects 1c) Decrease in complaints from other 2a) Members survey, focus groups
practitioner 2b) College tracking system
2) Perceptions of members and other
professionals

32
9. Evaluation Matrix Worksheet
Use either case study to fill out this worksheet

Issue Evaluation Indicator Data required Source of data Method of Analysis Responsibility Timing
questions collection

HCA’s recommended model: Adapted from various sources.

33
10. BIBLIOGRAPHY AND REFERENCES
Canadian Evaluation Society. (1989). The Value in Evaluation. A Statement for
Managers. Canadian Evaluation Society.

Canadian International Development Agency. (1997). The Logical Framework:

Making it Results Oriented – Guidelines. Performance Review Branch.
Canadian International Development Agency. Available online at:
http://www.acdi-
cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/c36ebd57
1b6fa02985256c620066cd6f?OpenDocument

Canadian International Development Agency. (1999). Results-Based

Management in CIDA: An Introductory Guide to the Concepts and
Principles. Performance Review Branch, Canadian International
Development Agency. Available online at: http://www.acdi-
cida.gc.ca/cida_ind.nsf/0/b83025bcf2da296785256c6b001a1b36?OpenDo
cument

Canadian International Development Agency. (2000). CIDA Evaluation Guide.

Performance Review Branch, Canadian International Development
Agency. Available inline at: http://www.acdi-
cida.gc.ca/cida_ind.nsf/0/061a4e025fc6fe2785256c6b00155e2d?OpenDo
cument

Canadian International Development Agency. (2000). RBM Handbook on

Developing Results Chains. Results-Based Management Division.
Canadian International Development Agency. Available online at:
http://www.acdi-
cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/d321e7d0
6cb2a65885256c62001b1b6c?OpenDocument

Canadian International Development Agency. (2001). How to Perform

Evaluations – A Ten Part Series Performance Review Branch, Canadian
International Development Agency. Available online at http://www.acdi-
cida.gc.ca/cida_ind.nsf/0/d4207e5842f66aa385256c5000589701?OpenDo
cument

Canadian International Development Agency. (2001). A Results Approach to the

Implementation Plan. Performance and Knowledge Management Division,
Canadian International Development Agency Available inline at:
http://www.acdi-
cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/05dde51fd
3e6e77f85256c620019b585?OpenDocument

34
Canadian International Development Agency. (2000). Training workshop -
Introduction to Results-Based Management. Performance and Knowledge
Management Division, Canadian International Development Agency.
Available online at: http://www.acdi-
cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/daa38a8fb
771568985256c5600191ddc?OpenDocument

Cummings, F. H. (1992). Monitoring and Evaluation in Community Economic

Development. Community Futures Training Project. Employment and
Immigration Canada.

Cummings, F. H. (1997). Logic Models, Logical Frameworks and Results-Based

Management: Contrasts and Comparisons. Canadian Journal of
Development Studies, 28, 587-596.

Cummings, H. (1998). Evaluation Models. Guelph: Harry Cummings and

Associates.

Cummings, H. (1998). The Logical Framework. Guelph: Harry Cummings and

Associates.

Cummings, H. and Bhanjee, T. (2002). Evaluation Plan of Practice Review for

the Quality Assurance Program – Ontario College of Pharmacists.
Guelph: Harry Cummings and Associates.

Cummings, H. and Bhanjee, T. (2003). RBM and Evaluation Workshop. Guelph:

Harry Cummings and Associates.

Cummings, H. and Kora, G. (1999). An Evaluation Manual for Rural Community

Based Organizations. Rural Planning and Development. University of
Guelph.

Cummings, H., Lipski, U. and Wasteneys, C. (1997). A Framework for

Evaluating the Quality Assurance Programs of the Colleges of Health
Professions in Ontario. Guelph: Harry Cummings and Associates.
Available online at:
http://www.hprac.org/english/pageDisplay.asp?webDocID=2195

Cummings, H. and Morris, K. (1999). Monitoring and Evaluating the Success of

Your Community Project. New Tecumseth Economic Development
Corporation. Guelph: Harry Cummings and Associates.

Den Heyer, Molly. (2001). Literature Review from: The Development of a

Temporal Logic Model. Available online at: http://web.idrc.ca/fr/ev-27872-
201-1-DO_TOPIC.html

35
Framst, G. (1995). Application of Program Logic Model to Agricultural
Technology Transfer Programs. Canadian Journal of Program Evaluation
10 (2),123-132

International Development Research Centre. (1994). Worksheets for Evaluation

Planning. Evaluation Unit. Ottawa: International Development Research
Centre.

International Development Resource Centre. (2002). Available online at:

http://www.worldbank.org/poverty/impact/index.htm

Kellogg Foundation. (1997). Evaluation Handbook. Michigan: Kellogg

Foundation. [Online]. Available:
http://www.wkkf.org/Knowledgebase/Pubs/

Kellogg Foundation. (2000). Logic Model Development Guide. Michigan:

Kellogg Foundation. [Online]. Available:
http://www.wkkf.org/Knowledgebase/Pubs/

Sawadogo, J. and Dunlop, K. (1997). Managing for Results with a Dynamic

Logical Framework Approach: From Project Design to Impact
Measurement. Canadian Journal of Development Studies, 28, 597-612.

Taylor, A. R. and Botschner, J. V. (1998). Evaluation Handbook. Ontario

Community Support Association. Centre for Research and Education in
Human Services.

Thomas, L. and C. Wasteneys. (1998). OP2000 Evaluation Handbook for

Community Based Organizations. School of Rural Planning and
Development, University of Guelph

Treasury Board of Canada Secretariat. (1981). Guide on the Program

Evaluation Function. Program Evaluation Branch. Treasury Board of
Canada Secretariat.

Treasury Board of Canada Secretariat. (1991). Program Evaluation Methods -

Measurement and Attribution of Program Results. Program Evaluation
Branch. Treasury Board of Canada Secretariat.

Treasury Board of Canada Secretariat. (2002). Results for Canadians: A

Management Framework for the Government of Canada. Treasury Board
of Canada Secretariat. Available online at: http://www.tbs-
sct.gc.ca/res_can/rc_e.asp

36
Treasury Board of Canada Secretariat. (2001). Guide for the Development of
Results-based Management and Accountability Frameworks. Available
online at: http://www.tbs-sct.gc.ca/eval/home_accueil_e.asp

UNDP, (1997). Results-Oriented Monitoring and Evaluation. A Handbook for

Programme Managers. New York: ESP Handbook Series.

Weir, Claire. (2002). Summative Evaluation: Huron Manufacturing Strategic

Initiatives Evaluation. Program Evaluation Course, 2002. University of
Guelph.

World Bank - PovertyNet. (2002). Impact Evaluation website at:

http://www.worldbank.org/poverty/impact/index.htm

37