AT3 – EVALUATION PLAN: IMPACT

SHEPPARTON CULTURE KITCHEN

Table of Contents
1. Health Promotion Program..............................................................................................................3

1.1. Introduction to Program...........................................................................................................3

1.2. Issue Addressed by Program...................................................................................................3
1.3. Program Goal..................................................................................................................................3

2. Evaluation Strategy...............................................................................................................................4

2.1. Evaluation Question.....................................................................................................................4

2.2. Type of Evaluation: Impact.......................................................................................................4
2.3. Evaluation Design.....................................................................................................................4-5
2.4. Evaluation Method and Data Collection..........................................................................5-6
2.5. Timeline.............................................................................................................................................6
3. References..............................................................................................................................................7-8
 Health Promotion Program
Introduction to Program
The Shepparton Culture Kitchen is a community health program developed by the Municipal
Association of Victoria (MAV) and the Greater Shepparton City Council. It is concerned with
fostering healthy and culturally diverse eating, food knowledge, training, and local events in
Shepparton.

Issue Addressed by Program

Members of this community are culturally and culinarily diverse, but the dietary and nutritional
framing common to their region is primarily Western. As of 2016, the population of Shepparton is
129,971; 26,522 of these residents were born overseas; and 20,347 have at least one parent who
was born overseas (ABS 2016). Besides the predominant Australian and English population, there
are individuals and families from Ireland, New Zealand, Italy, Greece, India, and Afghanistan (ABS
2016). There are Greek Orthodox, Hindu, Sikh, and Islamic religious communities in Shepparton
(Idcommunity 2016). These are the cultures, each with their own specific foods and food practices,
that the Shepparton Culture Kitchen wishes to target and utilise in order to address food insecurity.
Food insecurity is characterised by the inability to access foods conducive to a nutritional and
healthy life (Rosier 2011). Access to, and availability of, these foods depend on: transportation,
storage, preparation facilities, knowledge and skills, location of outlet, price, quality, and variation
(Rosier 2011). Food insecurity can be separated into two categories: food insecurity with hunger
and food insecurity without hunger. Food insecurity with hunger applies to people who go without
meals, and food insecurity without hunger applies to those who do not go without meals but who
experience anxiety about obtaining them (Rosier 2011). Food insecurity affects the body’s physical
health (Health Direct 2019) as well its mental health (VicHealth 2016). The health consequences of
food insecurity include: malnutrition, overweight and obesity, cardiovascular disease, poor oral
health, longer hospitals stays, and depression and anxiety disorders (Health Direct 2019; VicHealth
2016).
Globally, Australia ranks 15th in countries afflicted with food insecurity (Innes-Hughes et al. 2010).
The Australian government has recognised the disproportionate effect food insecurity has on the
culturally and linguistically diverse (Rosier 2011). The Victorian government measures food
insecurity in Greater Shepparton at 4.6% (VAHI, 2014). The proportion of Greater Shepparton,
within that 4.6%, suffering from food insecurity with hunger is 15.3% (Innes-Hughes et al. 2010).
The Shepparton Culture Kitchen estimates 65% of its population lives in areas with poor
availability, with access to outlets over 3km away from their homes. Food insecurity can improve in
this population with community development and empowerment (Innes-Hughes et al. 2010). The
Shepparton Culture Kitchen uses the culturally diverse members of the community to deliver food
knowledge, training, and events for this purpose.

Program Goal
The Shepparton Culture Kitchen’s goal is to reduce food insecurity in their culturally diverse
population through community empowerment over a one-year time period.
 Evaluation Strategy
Evaluation Question
Is the Shepparton Culture Kitchen Program successfully reducing the proportion of food insecurity in
its culturally diverse population during the first three months of its intended one-year time period?

Type of Evaluation: Impact

Impact evaluation measures whether or not the goals and objectives of a health program produce
successful results, but also measures any unintentional impacts, not put forward by the program,
that occurs as a result of intervention (Rogers 2014). An example of a similar impact evaluation
undertaken in the past which produced useful findings is a study (Meiklejohn et al. 2017) which
evaluated the FoodMate program. This health intervention’s goal was to reduce food insecurity in
targeted communities in Western Melbourne, and it found many positive impacts related to the
educational, social, and communal features of the program (Meiklejohn et al. 2017). The World
Fund Programme (WFP), the world’s largest humanitarian organisation has also highlighted the
vital role impact evaluation can play particularly in the area of food insecurity (WFP 2021).
Impact evaluations are undertaken to determine whether a program needs improvement to achieve
its desired goals and objectives (formative impact evaluation), or to determine whether it is
suitable to discontinue a program due to its lack of positive impact (summative impact evaluation)
(Rogers 2014). We will use formative impact evaluation during implementation to determine, with
causal attribution, whether the Shepparton Culture Kitchen is making progress toward reducing
food insecurity in its population. This type of impact evaluation is appropriate to support analysis
intended to inform improvements in the health program (Rogers 2015). The result of this type of
impact evaluation can generate information related to funding allocation, resources, other specific
—perhaps missing, unsuitable, or under-utilised—features of design and implementation (Rogers
2015). Formative impact evaluation during implementation is also appropriate for the Shepparton
Culture Kitchen because the scope of time is large enough in order to use our results to inform the
direction and improvement of the program as it proceeds (Rogers 2014).

Evaluation Design
A pre-test/post-test quasi-experimental design is a non-randomised design used to analyse the
benefits of an intervention before and after its implementation (Harris 2006). A time series design,
which we will use for this evaluation, is a non-experimental variant of the pre-test/post-test quasi-
experimental design and employs a series of measurements taken over a period of time (BCcampus
2021). A time series design measures variables in the population dependent on the health
intervention before and after implementation, like the pre-test/post-test quasi-experimental
design, but uses multiple pre- and post-test measurements (BCcampus 2021) which complement
the formative impact during implementation type of evaluation we are conducting. Longitudinal
cohort study designs conducted prospectively—which the time series design falls under—allow us
to monitor changes attributable to the Shepparton Culter Kitchen program that may or may not
take place in the target population over a period of time (Budiu & Moran 2021). We will use this
design to gather data at increments during the first three months of implementation to inform
adjustments as the program proceeds in real time. Some relevant strengths of prospective cohort
study designs are connecting exposure with events and establishing their sequence, eliminating the
potential for recall bias, and the ability to analyse the impact of time-related factors on an
individual basis within the cohort (or correcting for the ‘cohort effect’) (Budiu & Moran 2021).
Researchers have developed forecasting models for food insecurity over several at-risk countries
by using a time series design, and their results justify the use of continuous data collection of this
kind in this context (Fioni et al. 2021).

Evaluation Method and Data Collection

Our evaluation question is best answered using an observational quantitative approach.
Quantitative methods are well suited to analyse the consequences of intervention choices (Gillies
2016). They are also well suited to collect impact data before and after implementation and at
incremental points during implementation (ATSDR 2015) consistent with our evaluation design
and type. Quantitative methods are traditionally more precise, consistent, and generalisible
approaches compared to others (ATSDR 2015), which affords them a healthy external validity; they
are also better equipped to detect some biases (e.g., selection) which affords them a healthy
internal validity (Gillies 2016). Employing the quantitative approach here is appropriate to
measure impact and casual attribution and therefore inform any adjustments the Shepparton
Culture Kitchen should consider as it proceeds.
The quantitative data needed to analyse the metrics of impact and attribution in relation to the goal
of the Shepparton Culture Kitchen—primarily, to reduce food insecurity in its target population—
will be collected using self-administered questionnaires. In accordance with ethical considerations
(Lavrakas 2008), the questionnaire does not pose personal questions and therefore does not
require measures designed to protect the participant’s identity. The questionnaire also includes an
introductory sentence briefly explaining that participation is completely voluntary. The questions
are structured to flow logically to minimise measurement error (Fanning 2005). Exclusively closed
questions were included in accordance with the quantitative method of evaluation we are
employing, affording the questionnaire easier codifiability and analysability (Kelly et al. 2003).
Closed questions also make completing the questionnaire less confusing for participants (Thomas
2004). The questions included focus on measuring the reduction of food insecurity, which is the
goal of the Shepparton Culture Kitchen and our evaluation question, but also measuring factors
which contribute to food insecurity in the target population. Some examples of questions include:
“Have you experienced MORE, LESS, or, NO, change in your ability to access nutritious food?”, “Have
you experienced MORE, LESS, or, NO, change in your anxiety in relation to the access of nutritious
food?”, “Have you experienced MORE, LESS, or, NO, change in the cultural appropriateness of the
food to which you have had access?”, and “Have you acquired MORE, LESS, or NO knowledge and
skills in relation to preparing nutritious food?”. Researchers have found questionnaire-based data
collection to be an effective tool for measuring food insecurity (Frongillo et al. 1997).
The data will be collected at three increments: first month of implementation, second month of
implementation, and third month of implementation. This data collection structure is important
(BCcampus 2021) to first gather baseline data, and then measure the impact of the Shepparton
Culture Kitchen on key variables in at the two following data points. We will be distributing self-
administered questionnaires to participants in the Shepparton Culture Kitchen program during
events, workshops, and meet-ups. Using non-random purposive sampling, we aim to target fifty
participants in the program. This number is consistent with recommendations for sample sizes in
quantitative studies and generalisability to the greater population (Budiu & Moran 2021). The
questionnaire is then pilot tested using respondent debriefing to ensure all necessary
improvements are made before they are distributed to our population group (Thomas 2004).
Results will be anaylsed by technical specialists to determine whether or not the Shepparton
Culture Kitchen is reaching its goal of reducing food insecurity in its culturally diverse population
through community empowerment, and if not, what direction is needed, what improvements can be
made, and where recourses should be reallocated (Rogers 2015).

Timeline
Graph 1. Evaluation Timeline

The first step of the evaluation process is to go over the ethical considerations which factor in to
our implementation, namely: assuring that participation is purely voluntary and the personal
identities and information of participants is kept private if collected. Next the evaluation is set up,
the questionnaire is constructed based on our evaluation question and requires the longest amount
of time to research, develop, and test before it is implemented. The evaluation is then implemented
in three data collection points over a three-month period of time. This provides a sufficient amount
of time for outcomes to manifest and causation to be established (Smart 2020). Experts in
quantitative data analysis then analyse the data over a one-month period. The data analysts will
determine meaningful patterns in the evidence through a systematic process of classification,
summation, and comparison in relation to our evaluation question (WHO 2013). Once the data has
been analysed and findings have been concluded, the findings are then written up for the
Shepparton Culture Kitchen. Key information provided in the evaluation report include: the
evaluation question being addressed, the methodology, sampling, data analysis, ethics, findings,
limitations, strengths, and recommendations (Smart 2020). This step would usually require more
time than we have allotted if multiple different reports were required to be distributed to multiple
stakeholders (Smart 2020). The final step in our timeline is sharing the evaluation report with the
Shepparton Culture Kitchen.
 References
ABS (Australian Bureau of Statistics) (2016) 2016 Census QuickStats, Australian Government,
accessed 30 September 2021.
ATSDR (Agency for Toxic Substances and Disease Registry) (2015) Evaluation Methods, U.S.
Department of Health & Human Services, accessed 30 September 2021.
BCcampus (2021) Quasi-Experimental Research, Research Methods in Psychology – Second Edition,
accessed 30 September 2021.
Budiu, R. & Moran, K (2021) How Many Participants for Quantitative Usability Studies: A Summary of
Sample-Size Recommendations, Nielson Norman Group, accessed 1 October 2021.
Fanning, E (2005) Formatting a Paper-based Survey Questionnaire: Best Practices. Practical
Assessment, Research and Evaluation, Vol. 10, No. 12.
Foini, P., Tizzoni, P., Paolotti, D., Omodei, P (2021) On the forecastability of food insecurity, medRxiv,
accessed 30 September 2021.
Frongillo, E., Rauschenbach, B., Olson, C., Kendall, A & Colmenares, A (1997) Questionnaire-Based
Measures Are Valid for the Identification of Rural Households with Hunger and Food Insecurity,
The Journal of Nutrition, Vol 127, No. 5, pp.699–705.
Gillies C, Freemantle N, Grieve R, et al. (2016) Advancing quantitative methods for the evaluation of
complex interventions, Health Services and Delivery Research, No. 4, Vol. 16, Southampton, UK.
Harris, A.D., McGregor, J.C., Perencevich, E.N., Furuno, J.P., Zhu, J., Peterson, D. E., & Finkelstein, J
(2006) The use and interpretation of quasi-experimental studies in medical informatics. Journal of the
American Medical Informatics Association, JAMIA, No. 13, Vol. 1, pp.16–23.
Health Direct (2019) Malnutrition, Health Direct, accessed 23 September 2021.
Idcommunity (2016) Religion, Greater Shepparton, accessed 30 September 2021.
Innes-Hughes C, Bowers K, King, L, Chapman K, Eden B (2010) Food security: The what, how, why
and where too food security in NSW, NSW.
Kelly, K., Clark, B., Brown, V. & Sitzia, J (2003) Good practice in the conduct and reporting of survey
research, International Journal for Quality in Health Care, Vol. 15, No. 3, pp.261–266.
Lavrakas, P. J (2008) Encyclopedia of survey research methods, Thousand Oaks, CA: SAGE
Publications, Inc.
Meiklejohn S.J., Barbour L, Palermo CE (2017) An impact evaluation of the FoodMate programme:
Perspectives of homeless young people and staff. Health Education Journal, No. 76, Vol. 7, pp.829-841.
Rosier, K (2011) Food insecurity in Australia: What is it, who experiences it and how can child and
family services support families experiencing it?, Australian Institute of Family Studies, accessed 30
September 2021.
Rogers, P (2014) Overview of Impact Evaluation, Methodological briefs, UNICEF.
Rogers, P., Hawkins, A., Mcdonald, B., Macfarlan, A., Milne, C (2015), Choosing appropriate designs
and methods for impact evaluation, Department of Industry, Innovation and Science.
Smart, J (2020) Planning an evaluation, Australian Institute of Family Studies, accessed 1 October
2021.
Thomas, S. J (2004) Using web and paper questionnaires for data-based decision making, SAGE
Publications, Inc.
Thomas, S.J (2004) Pilot Testing the Questionnaire, In Using Web and Paper Questionnaires for Data-
Based Decision Making, pp.107-118.
VAHI (Victoria Agency for Health Information) (2014) Challenges to Healthy Eating – Food
Insecurity in Victoria, Victorian Government.
VicHealth (2016), Obesity and Healthy Eating in Australia: Evidence Summary, Victorian Health
Promotion Foundation, accessed 23 September 2021.
WFP (World Food Programme) (2021) Impact Evaluation, WFP, accessed 30 September 2021.

WHO (World Health Organization) (2013) WHO evaluation practice handbook, World Health
Organization, pp.54-56.