Review

J Educ Eval Health Prof 2019;16:40 • https://doi.org/10.3352/jeehp.2019.16.40

How to execute Context, Input, Process, and Product Evaluation Model in medical health
education

So young Lee, Jwa-Seop Shin, Seung-Hee Lee*

Department of Medical Education, Seoul National University College of Medicine, Seoul, Korea

* Corresponding email: [email protected]

Editor: A Ra Cho The Catholic University, Korea

Received: November 29, 2019; Accepted: December 28, 2019; Published online: December 28, 2019
This article is available from: https://jeehp.org

© 2019, Korea Health Personnel Licensing Examination Institute

(cc) This is an open-access article distributed under the terms of the Creative Commons Attribution
License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,
and reproduction in any medium, provided the original work is properly cited.

1
Abstract

Improvements to education are necessary in order to keep up with the education requirements of today.

The CIPP evaluation model was created for the decision-making towards education improvement, so this

model is appropriate in this regard. However, application of this model in the actual context of medical

health education is considered difficult in the education environment. Thus, in this study, literature survey

of previous studies were investigated to examine the execution procedure of how the CIPP model can be

actually applied.

For the execution procedure utilizing the CIPP model, the criteria and indicators were determined

from analysis results and material was collected after setting the material collection method. Afterwards,

the collected material was analyzed for each CIPP element, and finally, the relationship of each CIPP

element was analyzed for the final improvement decision-making. In this study, these steps were followed

and the methods employed in previous studies were organized. Particularly, the process of determining

the criteria and indicators was important and requires significant effort. Literature survey was carried out

to analyze the most widely used criteria through content analysis and obtain a total of 12 criteria.

Additional emphasis is necessary in the importance of the criteria selection for the actual application of

the CIPP model. Also, a diverse range of information can be obtained through qualitative as well as

quantitative methods. Above all, since the CIPP evaluation model execution result becomes the basis for

the execution of further improved evaluations, the first attempt being performed without hesitation is

essential.

Keywords: CIPP model; CIPP evaluation model; Educational evaluation

2
Introduction

As times change and the education environment changes along with the students, education always

possesses an unending possibility of change. So, experts are constantly contemplating on how medical

health education can be improved. Effective improvements can be achieved when which aspects and how

are determined. Thus, the suitable education evaluation method can facilitate the improvement of

education.

Various education evaluation models exist depending on the meaning and perspective of the model

and Worthen, Sanders, and Fitzpatrick(1997) categorized these models largely into objective-oriented,

management-oriented, consumer-oriented, expertise-oriented, participant-oriented, and adversary-

oriented evaluation models[1]. For these approaches Kim HS, Baek SG, Sung TJ summarized[2-4]:

The objective-oriented evaluation approach focuses on establishing goals in advance also then

determining how far the goals have been achieved. However, the emphasis is only on evaluations of

outcomes such as the effectiveness of education and student achievement, which can overlook the

evaluation of the teaching and learning process itself.

The management-oriented evaluation approach considers an evaluation to assist decision making by

providing decision makers with the necessary information. The limitation is that it is rather complex to be

fully implemented, but this approach allows evaluator to evaluate all aspects of program implementation.

In addition, it helps to clarify the focus of the evaluation by helping the evaluator create important

questions to be addressed at each stage.

The consumer-oriented evaluation approach regards everything used for education as an education

product, and furthermore, conducts education as a service. Accordingly, attention is drawn to what

consumers and consumers of education programs want and need. However, this approach can reduce the

motivation of teachers and curriculum developers by considering only the consumer's position.

The expertise-oriented evaluation approach is the oldest and most widely used model, and the

method of evaluating education by expert judgment. The limitations of this approach can lead to

irrational deliberations, since the weights of the criteria for the trivial and the important are not presented.

In addition, manager prejudice can influence the formation of a review team.

3
 The adversary- oriented evaluation approach is able to collect the opinions of the subjects broadly

by dealing with all the opposing views in one evaluation, and help shed light on the advantages and

disadvantages of the education program.

The participant-oriented evaluation approach attempts to take a holistic approach to humanistic

issues in complex contexts and is characterized by value pluralism, and therefore can take a very different

approach than other assessment approaches. But subjective or prejudiced interventions in the assessment

cannot be ruled out. In addition, by excluding the evaluator’s role in the assessment, the evaluation

method itself cannot perform the role of the assessment.

Among these, the management-oriented provides the necessary information to the decision-maker

to help in the decision-making[4] thus, this model type is appropriate in presenting the important

information for education improvement.

Representative evaluation models are Alkin's CSE model and Stufflebeam's CIPP model in the

management-oriented approach. The CIPP model is a cyclic evaluation model[5] that can be modified at

any time by detecting errors or deficiencies at each stage by providing information on decision-making

about program planning, structuring, execution, and improvement as well as evaluating activities.

Therefore, the CIPP model is suitable for quality management of curriculum[6].

The CIPP evaluation model is the most commonly used in the education field[6]. The main

characteristic of this model is that the major objective of the evaluation is on improving rather than

proving[7]. CIPP is an acronym for Context, Input, Process, and Product. Since evaluation is conducted

using detailed criteria regarding these components, they are useful in carrying out systematic and structure

evaluations[8].

In the world medical health professions education field, CIPP evaluation model is introduced and

used for educational evaluation[9-16]. In addition, many educational sectors in Korea also use the CIPP

evaluation model[5, 6, 9, 17-27]. However, only a few studies can be found in educational evaluation in

the field of medical health professions education in Korea[8, 28]. As such, there are many advantages in

evaluating education based on the CIPP model, but there is not much research and utilization in the field

of medical education in Korea using the CIPP model. Although the complex characteristics of medical

4
education[29] go through a rather complicated process to implement the CIPP evaluation model[4], this

study analyzes several previous studies and shows how they can be applied.

In this study, the experience of using the CIPP model to evaluate the Medical Humanities

Course(MHC) at the Seoul National University College of Medicine, where the researchers of this study

are affiliated under, and various previous studies were comprehensively investigated to determine how

this model can be executed with what kind of procedure in the context of an actual medical education.

Education Evaluation Execution Procedure using the CIPP Evaluation Model

The following procedure is carried out when performing an evaluation using the CIPP model. First,

the criteria and indicators are determined. Next, which materials are necessary and the method with

which such materials will be collected for the evaluation are planned. Third, the collected materials are

analyzed according to the criteria and indicators of each section of the CIPP model. Lastly, the

relationships between the CIPP sections are analyzed[8]. Recognizing how the CIPP model can be applied

by following these evaluation steps can enhance the level of understanding.

Development of CIPP Evaluation Criteria & Indicators

When the ultimate goal of the evaluation is set, the very first step in the evaluation of education is

determining the evaluation criteria and indicators. This step is also an important step in determining the

direction of the evaluation. The evaluation criteria refer to the standard, principle, rule, or sign for the

evaluation[30]. The criteria, which are the standard of the evaluation, facilitates communication between

the evaluator and evaluation requestor regarding the evaluation subject or content based on the evaluation

activity[17] as well as systematic judgment[18]. For these reasons, this step of determining the criteria and

indicators has sufficient research merit in itself. The process of this step and its results have actually led to

significant research[17-21]. Like this, the first step of evaluation requires substantial preparation and

caliber as this step is significant enough to be treated as an independent study subject.

The activity of setting the criteria and indicators is as follows. First, these steps are for when literature

is used as the basis[8, 9, 20-22]: interview such as FGI[20, 21], using the Delphi technique[19], and

agreement between experts[8, 10-12, 19, 22].

5
 Understanding what each of the Context, Input, Process, and Product comprising the CIPP model

means is important for a systematic evaluation. Context evaluation is the evaluation of the need, problem,

asset, and opportunity within a situation[17]. Input evaluation assists in the decision-making of how

facilities, human resources, and budget will be determined and constituted to achieve the goal of the

education[28]. Process evaluation is the continuous examination of the program execution plan and

process records[30]. The assessment of this step provides information regarding the schedule, method of

progress, input activity type, and education method related to the education program to the education

director so that this information contributes to the smooth progress fitting for the education goal[31].

Product evaluation measures and analyzes the results during and after the education[23] and examines the

overall efficacy of the program[17]. This step has to inspect the intended effects, unintended effects,

positive effects, as well as negative effects[24].

The CIPP model is employed as the evaluation method in various fields including science education,

mathematics education, local education, education research and development, achievements through

testing, education reliability of the government, school improvements, teacher training, human resource

development, social welfare improvement, services of non-profit organizations, and technical

development[25]. As the model can be utilized in various fields, the criteria can be diversely set according

to the characteristics of the institution and program for evaluation, evaluation objective, evaluation

context, and evaluation characteristics. Table 1 shows the criteria of literature and it can be observed that

various criteria were used depending on the evaluation objective and field, which included institution

evaluation, education evaluation, nursing education, and medical education.

However, the following common criteria can be obtained when content analysis was conducted for

these criteria based on their frequency(Table 2.). Goals (6) and Necessity or Needs (5) had the highest

frequency for context evaluation, Material resources, facilities (6), Human resource (6), Contents (5), and

Curriculum (3) for input evaluation, Educational and service process (7), Program evaluation (4), and

Educational courses and programs (3) for process evaluation, and finally, Global satisfaction (5), Students’

and service achievement (4), and Program performance (4) for product evaluation. This result can

become resourceful for future study and evaluations based on the CIPP model.

6
 Especially, when evaluating a college of medicine curriculum or education program, the education

objective, achievement, focus, and operation guideline have to be considered[8]. Drawing up a blueprint

by integrating these elements with each element of the CIPP model contributes to the systematic

decision-making through the evaluation.

Material Collection Method Determination and Material Collection

The material collection method determination and material collection can be largely divided into the

method for quantitative evaluation and the method for qualitative evaluation as shown in Table. 1.

Stufflebeam presented material collection methods possible for each CIPP element. For context, systems

analysis, survey study, literature survey, public hearings, interviews, diagnostic assessment, and the Delphi

technique were presented; for input, available human and material resources, resolution strategy, design

procedure, possibility and economic analysis, literature survey, pilot program survey, advocacy groups,

and pilot attempt were presented; for process, procedural disorder identification and accidental disorder

awareness, detailed information acquisition for scheduled decision-making, describe the actual process,

continuous interaction with the program operation staff, and observation or their activities were

presented; for product, operational definition and measurement of the performance standards and

collection of the judgments by the interested parties were presented. These various techniques can be

used as methods of material collection for actual evaluation, and among them, the methods presented in

Table. 1 were mainly used in published academic papers. Specifically, quantitative material collection is

possible through questionnaire, literature survey, and grades. For qualitative material collection, the

methods of short answer surveys, interviews, meeting minutes, curriculum, syllabus, and literature were

used in previous studies.

As shown in Table. 1, in many cases, students and professors were included as subjects of material

collection. However, there were also many cases where various related people were included as subjects.

In order to persuade various people, it is necessary to understand the relationships between the various

related people and their evaluation demands through multi-faceted evaluation methods[32].

7
 Collected Material Analysis and Relationship Identification

The CIPP evaluation model was developed with the purpose of providing systematic information for

the decision-making as a proactive evaluation from the very beginning. Thus, an evaluation is defined as a

process for planning, obtaining, and providing useful information necessary for determining decision-

making solutions[33]. In the CIPP evaluation model, 4 types of decisions are made to improve the

evaluation subject and these decisions are the planning decision, structuring decision, implementation,

and recycling decision. The planning decision sets the objectives, the structuring decision composes the

procedural method necessary to achieve those objectives, implementation is a practical decision regarding

the selected procedure, and the recycling decision determines the continuation, termination, and

modification of the program[24]. For these 4 decision types, the CIPP evaluation model proposed by

Stufflebeam involved the goals, plans, actions, and outcomes of the core value of the program being

examined and modified through the context evaluation, input evaluation, process evaluation, and product

evaluation, respectively, as shown in Figure. 1[17]. In detail, decision-making regarding objective

determination, order of priority, and distribution guideline can be done through context evaluation. With

regard to the selection of strategy for the program, the collected information can become a guideline and

can be inputted into the design of detailed procedure through the input evaluation. Process evaluation

contributes to execution guidelines and product evaluation contributes to guidelines for termination,

continuation, modification, and initiation[7].

Meanwhile, the results of each element can not only be utilized for the improvement of the

corresponding elements but also the relationship between them can be identified for improvement. In the

2019 study by Lee[8], a need to actively reflect the demands and capacity of the students was observed in

the context evaluation, and this impacted the input, process, and product aspects as well. In addition, the

strategy in the input element was positive but there was a need to faithfully carry out the execution of the

original plan in the process element. Another study in 2012 by Al-Khathami AD showed that problems

found in the process also affected the product[11]. Like this, since education is a continuing, single system,

the relationships between the elements can be identified to make improvements, and when the analysis

for each CIPP model element is completed, important messages can be obtained when the relationships

8
between the elements are determined.

Conclusion

Up to now, various previous studies were investigated with a focus on the CIPP evaluation model and

explored from a practical perspective on which procedures and methods were employed. These results

showed that evaluations using the CIPP model, which can be considered rather difficult, can provide the

basis for education improvement and no longer be considered a tough task.

With regard to the execution of this model, the setting of the criteria has to be emphasized once more.

The model may not be able to address unplanned evaluation questions[13]. Thus, this setting of the

criteria can act as definitive evidence that determines whether an evaluation is successful.

Especially, the omission of evaluation of unset parts becomes more vulnerable for quantitative

evaluations. In this regard, a method of material collection that can cover this criteria determination is

collecting as much qualitative material as possible. These materials can contribute in obtaining a diverse

range of opinions that quantitative materials cannot explain.

Rather than utilizing a single group such as the student group as the evaluation material collection

source, having a balanced perspective of the various interested parties regarding education can improve

the reliability and validity of an evaluation, which can then be utilized as convincing base data.

In this study, even if there is any hesitation to using the CIPP evaluation model due to realistic

resource limitations despite completely understanding the CIPP evaluation model and recognizing its

importance, it is recommended that the attempt be carried through because the result of the evaluation

conducted now can be used as the basis for determining the criteria and material collection method of a

future CIPP evaluation to be carried out.

ORCID

So Young Lee https://orcid.org/ 0000-0002-7268-2261

Seung-Hee Lee https://orcid.org/0000-0001-8672-5253

9
Jwa-Seop Shin https://orcid.org/0000-0002-6251-3616

Authors’ contributions

Conceptualization: SYL SHL JSS.

Data curation: SYL

Formal analysis: SYL SHL.

Methodology: SYL, SHL.

Project administration: SHL SYL

Visualization: SYL.

Writing – original draft: SYL.

Writing – review & editing: SYL SHL JSS.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

None

Acknowledgments

None

Data availability
None

Supplement materials
None

References

10
1. Worthen BR, Sanders JR, Fitzpartrick, JL. Program Evaluation: Alternative Approached and Practical

Guidelines. NY: Longman.

2. Kim HS. Theory and practice of educational programme evaluation. Kyoyookbook.; 2015. p. 45-164.

3. Baek SG. Theory and practice of educational evaluation. Kyoyookbook; 2019. p. 33-76.

4. Sung TJ. Evaluation model and school evaluation. In: Modern educational evaluation: Hakjisa.; 2011. p.

477-505.

5. Kang CY. Development of evaluation model and indicators for school curriculum: Ministry of

education.; 2013. 146p.

6. Kim OJ. A Study on the Measures for Managing the Quality of Curriculum of Early Childhood

Education Department in College with the Application of CIPP Model Based on PDCA. Journal of

the Korea Convergence Society. 2019; 10: 215-226. https://doi.org/10.15207/JKCS.2019.10.1.215

7. Stufflebeam DL, Shinkfield AJ. Systematic evaluation: Springer.; 1985. 368p.

8. Lee SY, Lee SH, Shin JS. Evaluation of Medical Humanities Course in College of Medicine Using the

Context, Input, Process, and Product Evaluation Model. J Korean Med Sci. 2019 Jun 4.

https://doi.org/10.3346/jkms.2019.34.e163

9. Rooholamini A, Amini M, Bazrafkan L, Dehghani MR, Esmaeilzadeh Z, Nabeiei P, Rezaee R, Kojuri J.

Program evaluation of an integrated basic science medical curriculum in Shiraz medical school, using

CIPP evaluation model. J Adv Med Educ Prof. 2017 Jul; 5: 148-154.

10. Neyazi N, Arab PM, Farzianpour F, Mahmoudi MM. Evaluation of selected faculties at Tehran

University of Medical Sciences using CIPP model in students and graduates point of view. Eval

Program Plann. 2016 Dec;59:88-93. https://doi:10.1016/j.evalprogplan.2016.06.013.Epub2016Aug11.

11. Al-Khathami AD. Evaluation of Saudi family medicine training program: the application of CIPP

evaluation format. Med Teach. 2012;34:81-89. https://doi:10.3109/0142159X.2012.656752.

12. Narges N, Mohammad A, Freshteh F, Mahmood M. Identifying weaknesses in undergraduate

programs within the context input process product model framework in view of faculty and library

staff in 2014. Korean J Med Educ. 2016; 28(2): 185-194. https://doi.org/10.3946/kjme.2016.29

11
13. Mirzazadeh A, Gandomkar R, Hejri SM, Hassanzadeh G, Koochak HE, Golestani A, Jafarian A, Jalili

M, Nayeri F, Saleh N, Shahi F, Razavi SH. Undergraduate medical education programme renewal: a

longitudinal context, input, process and product evaluation study. Perspect Med Educ. 2016 Feb;5(1):p.

15-23. https://doi: 10.1007/s40037-015-0243-3.

14. Ashghali-Farahani M, Ghaffari F, Hoseini-Esfidarjani SS, Hadian Z, Qomi R, Dargahi H. Neonatal

Intensive Care Nursing Curriculum Challenges based on Context, Input, Process, and Product

Evaluation Model: A Qualitative Study. Iran J Nurs Midwifery Res. 2018 Mar-Apr;23(2): p. 111-118.

https://doi: 10.4103/ijnmr.IJNMR_3_17.

15. Yarmohammadian MH, Mohebbi N. Review evaluation indicators of health information technology

course of master's degree in medical sciences universities' based on CIPP Model. J Educ Health

Promot. 2015 Mar 27;4:28. https://doi: 10.4103/2277-9531.154122.eCollection 2015.

16. Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE Guide No.67 .Med

Teach. 2012; 34: e288-99. https://doi: 10.3109/0142159X.2012.668637.

17. Yi BJ, Park JY, Park EH. A study of the assessment model for art museum educational programs .

2011; 11: 85-103.

18. Cho YS, Lee YS. Development of evaluation criteria for school consulting based on the CIPP

evaluation model. The Journal of Educational Administration. 2015 Oct 30 277-301.

19. Shin YJ, Kim ST, Song HD. Development of evaluation indicators for Job capability strengthening

program for vocational high school with application of CIPP evaluation model. The Journal of

Vocational Education Research. 2018; 37: 1-23.

20. Jang ES, Hwang HS. Development of assessment criteria for education and training program of

government-financed scientific and technical research institute based on CIPP. 2015; 15: 323-341.

21. Jung KY, Choi YH. Development of assessment criteria for scientific technology education program

by science museum based on CIPP. KJTE. 2011; 11: 85-103.

22. Lee YM. Examining the perceived impacts of training programs for small and midium-sized

corporation employees’ competency development on their performance, Using the CIPP evaluation

model. The Journal of Training and Development. 2012 March: 1-20.

12
23. Kim DR, Yum SC. Developing an evaluation scale for the college tutoring program based on the

CIPP Model. Educational Research. 2017;39: p. 63-84.

24. Jung JG. Clarification and Implemenation of CIPP Evaluation for the Improvement of Education.

Journal of educational research. 2000;21: p. 103-121.

25. Baik Y. Study on Effects of Arts Education in ‘2011 Orchestra of Dream' Applied CIPP Model.

Journal of Arts and Cultural Management. 2012 Aug 31;5: p. 29-50.

26. Jung SH, Moon YK. A Study on Developing an Index for Evaluating Social Service Using the CIPP

Model . The Journal of Korean Policy Studies. 2013;13(2): p.233-254.

27. Shin YJ, Kim ST, Song HD. Development of Evaluation Indicators for Job Capability Strengthening

Program for Vocational High School with application of CIPP Evaluation Model. The Journal of

Vocational Education Research. 2018;37(3): p. 1-23.

28. Kim YJ, Son EK. A Study on Satisfaction of New Nurse Orientation Program Applying CIPP

Evaluation Model Focusing on Affiliated Hospitals of an Educational Foundation in South Korea.

JKAIS. 2017; 18(9): 226-235. https://doi.org/10.5762/KAIS.2017.18.9.226

29. Mennin, Stewart. Self‐organisation, integration and curriculum in the complex world of medical

education. Medical education, 2010, 44.1: 20-30. . https://doi: 10.1111/j.1365-2923.2009.03548.x.

30. Stufflebeam DL, Shinkfield AJ. Evaluation theory, models and application. Jossey-Bass.; 2007. 764p.

31. Stufflebeam DL, Coryn CLS. Evaluation theory, models and application (Research Methods for the

Social Sciences) 2nd Edition. Jossey-Bass.; 2007. 800p.

32. Baek SG, Yu YL. An Educational Policy Evaluation on the 2009 Secondary School Teacher

Appointment Examination. Journal of Educational Evaluation. 2008; 21(3): p. 69-91.

33. Stufflebeam DL. The Relevance of the CIPP Evaluation Model for Educational Accountability.

Journal of Research and Development in education. 1971 Feb 24. p. 1-31.

13
Table.

<Table 1> Literature regarding the criteria for each CIPP model element

Item Context Input Process Product

Define institution Flaw check or prediction
Collection of
situation, of procedures in progress
System capability, technology/judgment
Learner identification and or the execution process,
Solution program regarding achievements,
demand inspection, Information provision for
strategy, Linkage with information
Criteria Search for demand preplanned sequential
Design procedure of on the objective, situation,
satisfaction opportunity, decisions,
strategy execution, input, process,
Problem diagnosis and report and judgment of
Budget, schedule check Value and advantage
determination of events and activities
analysis
objective appropriacy regarding the execution
procedural disorder
Available human and identification and
material resources, accidental disorder Operational definition and
Utilization of systems
resolution strategy, design awareness, detailed measurement of the
analysis, survey study,
Material procedure, possibility and information acquisition for performance standards and
literature survey, public
Stufflebeam DL. et al. (1985) collection economic analysis, scheduled decision-making, collection of the judgments
hearings, interviews,
method literature survey, pilot describe the actual process, by the interested parties,
diagnostic assessment,
program survey, continuous interaction with qualitative/quantitative
Delphi technique
advocacy groups, pilot the program operation analysis
attempt staff, and observation or
their activities
Necessary for decision- Necessary for support Necessary for program Operational definition and
making regarding resource, resolution planning, procedure, and measurement of the
education objective and strategy, and design improvement, performance standard,
purpose when education procedure necessary for actual collection of judgments by
Purpose begins(provide standard selection(provide the situation basis provision in interested parties regarding
of change plans, basis for change activity performance analysis the performance,
performance judgment) composition, execution >use in the decision- Qualitative/quantitative
>use in the decision- process judgment basis) making for education analysis
making for education >use in the decision- execution >use in the decision-
14
 planning making for education making for recycling
structuralization

Budget, human resource

Service demand and management, facility and Service activity, service
Jung SH. et al. Service application, service
Institution Criteria situation, service resource environment, satisfaction, service
(2013) performance
objective domain service operation and evaluation
content
Execution plan(human
Program activity, program Program
Shin YJ. et al. Demand analysis, resource, procedure,
School Criteria management and performance(achievement,
(2018) objective setting support system, etc.),
evaluation satisfaction, effectiveness)
performance detail
contents of hospital Usefulness of the program,
introduction, Senior Composition and Feeling involvement
Criteria Intention and necessity
nurses' working facilitation through activities, Global
Kim YJ. et al.
Nursing experience satisfaction
(2017)
Material
collection Questionnaire Questionnaire Questionnaire Questionnaire
method
Biomedical approach
Incomprehensive More emphasize on
Preferring routine work
curriculum theoretical education
instead of professional job
Inappropriate Lack of professional The overlap of credits with
Tendency to leave the job
Criteria infrastructure NICU nursing mentors each other and the
Clitical incompetency of
Unknown duties Inappropriate admission inconsistency among the
Medical graduates
Ashghali-Farahani process of NICU mentor
health Dissatisfaction of graduates
M. et al. (2018) students Ineffective assessment
professions
Lack of NICU skill labs
Material
collection semi-structured interview , open question
method
Target of
NICU student, NICU graduate nurse, Neonatologist, Faculty member, Nurse
evaluation

15
 Student research activity
Interest and
educational courses and Efficiency of research and
understanding of
programs, teaching and educational programs,
students towards field
Goals, Organization and learning process teaching and learning
Criteria and labor market
management area Student progress process to increase
Faculty members
evaluation knowledge and job
Neyazi N. et al. Research and educational
Evaluated factors for performance of graduates
(2016) spaces and equipment
graduates
Material
collection researcher-made questionnaires inspired from the CIPP model and internal Evaluation literatures
method
Target of
students, graduates
evaluation
Overall impression about
Alternative procedural
Process involved in to the program
Achievement of program design for:
learning activities Barriers to achieve goals,
goals Contents
Criteria Trainers objectives, and needs
Barriers to achieve goals, Academic sessions
Theoretical sessions Assessment tools
objectives, and needs Hospital sessions
Al-Khathami AD. Clinical sessions Enjoyment
HDR sessions
(2012) Satisfaction
Material
collection Questionnaire(quantitative, qualitative)
method
Target of
trainee
evaluation
activities of group
human specialists and head of department, manager, students,
scientific services for faculty, students, administrators of library;
Criteria
needs of the local curriculum, funding, scientific research and
community training facilities teaching–learning activities
Yarmohammadian
of faculty
MH. et al. (2015)
Material
collection Questionnaire
method
Target of
directorates, faculty members, students, and library staff
evaluation
16
 Educational courses and
programs, learning and
Goals, management, and teaching process
Criteria Facility and spaces Graduates
organization area Administration and
financial
Narges N, et al.
Program evaluation
(2016)
Material
collection Questionnaire
method
Target of
department head, faculty members, and library staff
evaluation
Perceptions of learning
Perceptions of teachers
Academic self- Students’ Performance
The process of learning
Criteria perceptions Content of curriculum The process of teaching
Process of teaching
Perceptions of the and learning
environment
Social self-perceptions
1-Researcher–made
Review of current 1-Learner centered
questionnaires for
Rooholamini A, et evidence on integration integrated basic science
evaluating the quality of
al. (2017) Material and consultation with portfolios
A researcher made each integrated course
collection experts 2-Brainstorming (students)
questionnaire 2-Researcher–made
method Modified Dundee Ready 3- Semi- structured
questionnaires for
Education Environment interview (professors of
evaluating the quality of
Measure (DREEM) basic sciences)
ECE
First and second year
Faculties
Target of Students, faculty and First and second year medical students
and curriculum
evaluation administrators medical students Professors of basic
committee
sciences

17
 Available input
Implementation according Goal achievement
resources(human and
Goals, Necessity or to plan
Criteria material resources)
Needs Evaluation of the program Satisfaction of the
by students curriculum
Lee SY. et al. Educational strategy
(2019)
Material Questionnaire, FGI, Questionnaire, FGI,
Questionnaire, FGI, Questionnaire, FGI,
collection Meeting minutes, Meeting minutes, Time
Meeting minutes, Syllabus Meeting minutes, Grades
method Syllabus, Curriculum table
Target of
Students, faculty
evaluation

<Table 2> Content analysis of previous studies in Table 1 according to the reference frequency for each CIPP model section

Item Context Input Process Product

Material resources, facilities (6) Global satisfaction (5)
Human resource (6) Educational and service process (7) Students’ and service
Goals (6)
Contents (5) Program evaluation (4) achievement (4)
Necessity or Needs (5)
Keywords Curriculum (3) Educational courses and programs Program performance (4)
infrastructure (2)
extracted by Funding (2) (3) Efficiency of research and
Organization (2)
content Academic approach (1) Student progress evaluation (2) educational programs to
management (2)
analysis Admission process of students Composition (1) increase knowledge and
Intention (1)
(frequency) (1) Facilitation (1) performance (1)
duties (1)
Interest and understanding of Administration and financial (1) Barriers to achieve goals (1)
Barriers to achieve goals (1)
students (1) Service satisfaction (1) The process of teaching and
Educational and strategy (1) learning (1)
18
Implement plan (1)

19
<Figure 1> Relationship between key elements of the CIPP evaluation model and the program (Stufflebeam & Shinkfield(2007)

20