Research Report

A Core Components Framework for Evaluating

Implementation of Competency-Based Medical
Education Programs
Elaine Van Melle, PhD, Jason R. Frank, MD, MA(Ed), Eric S. Holmboe, MD,
Damon Dagnone, MD, MSc, MMEd, Denise Stockley, PhD, and Jonathan Sherbino, MD, MEd,
on behalf of the International Competency-based Medical Education Collaborators

Abstract
Purpose organized into a draft framework. Using programmatic assessment. With some
The rapid adoption of competency- a modified Delphi approach, the second modification in wording, consensus
based medical education (CBME) step examined consensus amongst an emerged amongst the panel of
provides an unprecedented opportunity international group of experts in CBME. international experts.
to study implementation. Examining
“fidelity of implementation”—that is, Results Conclusions
whether CBME is being implemented Two different viewpoints describing Typically, implementation evaluation
as intended—is hampered, however, by how a CBME program can bring about relies on the creation of a specific
the lack of a common framework. This change were found: production and checklist of practices. Given the ongoing
article details the development of such a reform. Because the reform model was evolution and complexity of CBME, this
framework. most consistent with the characterization work, however, focused on identifying
of CBME as a transformative innovation, core components. Consistent with recent
Method this perspective was used to create a developments in program evaluation,
A two-step method was used. First, a draft framework. Following the Delphi where implementation is described
perspective indicating how CBME is process, five core components of CBME as a developmental trajectory toward
intended to bring about change was curricula were identified: outcome fidelity, identifying core components is
described. Accordingly, core components competencies, sequenced progression, presented as a fundamental first step
were identified. Drawing from the tailored learning experiences, toward gaining a more sophisticated
literature, the core components were competency-focused instruction, and understanding of implementation.

C ompetency-based medical education describe a common framework that will of CBME that was studied, however, was
(CBME) is rapidly being adopted across permit reaching a deeper understanding described as devoting 15% of curriculum
the globe.1 Consequently, educators and of CBME programs, the influence of time to competency development
program leaders are in an unprecedented context, and the conditions under which without any significant changes to
position to study implementation they can work most effectively.5 teaching or learning. In contrast, the
and evaluate outcomes of innovative landmark description of CBME states,
curricula. However, these studies are “Implementation of such a system
hampered by the lack of a common The Importance of Evaluating demands substantial redefinition of
description of what constitutes a CBME CBME Program Implementation faculty and student roles and responsibili
program. A similar situation faced Implementation evaluation is a ties.”10(p55) Consequently, it is questionable
problem-based learning (PBL) three specific form of program evaluation as to whether this study actually
decades ago when the absence of a that examines the question “Is the examined a CBME curriculum.
standard description contributed to a program operating as intended?”6 It
series of inconsistent and inconclusive allows researchers to open the “black Avoiding a Type III error is the most
arguments regarding the impact of box” of program functioning.7 Without common reason cited for undertaking
PBL.2–4 The purpose of this article is to this information, the risk is present implementation evaluation. Other
of producing a Type III error—that reasons include documenting deviations
Please see the end of this article for information is, attributing negative findings to a from, and differences in, implementation;
about the authors.
failure in program theory when negative allowing for more meaningful
Correspondence should be addressed to Elaine Van findings may actually reflect an error in comparisons of interventions; and
Melle, 33 Hill St., Kingston, Ontario, Canada, K7L
2M4; email: [email protected].
program implementation.8 promoting external validity by providing
adequate guidelines for implementation.11
Acad Med. 2019;94:1002–1009. For example, a 2013 study concluded Implementation evaluation allows
First published online April 9, 2019
doi: 10.1097/ACM.0000000000002743
that competency-based curricula researchers and educators to provide
Copyright © 2019 by the Association of American do not produce graduates who are evidence if what occurred in the
Medical Colleges better prepared for medical practice,9 program can be reasonably connected
Supplemental digital content for this article is thereby challenging a key assumption to outcomes.6 Asking questions about
available at http://links.lww.com/ACADMED/A670. underlying CBME. The implementation the connection between CBME program

1002 Academic Medicine, Vol. 94, No. 7 / July 2019

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

activities and outcomes is particularly One issue with a checklist approach than on a checklist of practices. Defined
important so that education can be “involves the dynamic nature of as “an essential and indispensable
connected to health care practice or programs”17(p330) because judgments element of an intervention,”31(p3) these
patient outcomes.12,13 about which program activities are core components provide an overarching
essential evolve over time. For example, organizer. The components are specific
in CBME program implementation, the to the innovation yet robust enough
The Need for a Common establishment of a clinical competency to embrace different practices; in
Framework committee (CCC) seems to only recently other words, they can be applied to a
Evaluating the fidelity of design elements be emerging as an essential practice.23,24 range of program contexts. Such core
in CBME, however, has been hampered Furthermore, even assuming that there components are identified by making
by the lack of a shared understanding could be consensus regarding the essential explicit the underlying perspective
of what constitutes a CBME program.14 practices, the intent of such lists is to create framing the change and identifying
For example, in describing competency- a dichotomous yes-or-no judgment.19,20 the key components that align with
based education, Spady15 lists outcomes, The quality of the implementation is the viewpoint. Cousins et al describe
time, instruction, and measurement as not taken into consideration. Quality the development as taking place in
the four absolute minimum defining of implementation, however, makes a a collaborative fashion in order to
characteristics of competency-based significant difference. For example, recent strengthen validity and use.19 Accordingly,
education. In translating competency- research reveals that all CCCs do not this was the approach taken in developing
based education to the medical field, operate in a common fashion.25 Some the CBME Core Components Framework
Frank and colleagues16 describe a focus committees adopt a problem approach (CCF). More specifically, we used a two-
on curricular outcomes, an emphasis focusing primarily on identifying step method to develop a framework
on abilities, a de-emphasis on time- residents in difficulty, whereas others use and then to achieve consensus among an
based training, and the promotion of a developmental approach focusing on international group of medical education
learner centeredness as key elements. the progress of all residents. A checklist experts on a CBME CCF.
More recently, Carraccio and Englander1 approach does not allow for this critical
expanded the qualities of CBME to differentiation. Step 1: Developing a draft CCF
include the standardization of desired To develop the CBME CCF, we used
outcomes; a clear model of the trajectory Another challenge is that educational an iterative process that unfolded
for becoming an expert physician; innovations such as CBME are often over a 10-month period between
evidence-based learning strategies; complex26; they have many different February and December 2015. As
assessment tools based on care delivery; operating parts that can contribute to a recommended, we began by making
an emphasis on formative assessment; variety of outcomes.27 Such innovations, explicit our understanding of the
direct observation of learners; and the therefore, are highly sensitive to context,28 underlying perspective characterizing
existence of quality relationships between and so the expectation of FOI—that a the nature of the intended curricular
learners with patients, mentors, and program will be implemented exactly change.18 Situating the change
health care team members as essential the same way, using exactly the same within the educational literature on
practices in a CBME program. practices in every circumstance—is curriculum development led to an initial
simply unrealistic. Pérez et al argue29 that identification of core components. To
under such conditions adaptation—that create the framework, we drew from the
Challenges in Creating a Common
is, allowing for changes to the original literature on educational innovations
CBME Framework
design as long as the integrity of the and program evaluation. Accordingly,
Traditionally, the term “fidelity of innovation is not compromised—should the CBME CCF was designed using three
implementation” (FOI) is applied be the primary concern. Meeting this successive layers.32 The foundational layer
to studies that evaluate program challenge requires avoiding a checklist links each core component to theories,
implementation. FOI is defined as the or recipe-like approach and, rather, models, or best practices informing
“proportion of program components identifying components that “provide CBME.33 A principles layer offers “a form
that were implemented” and so guidance that must be interpreted and of rich high-level counsel”34(p195) that
represents “the adherence of actual, applied contextually.”18(p254) can be used to guide implementation,
treatment delivery to the protocol and a practice layer capture the details
originally developed.”17(p316) Simply The purpose of this study was therefore of implementation. With support from
stated, ensuring fidelity means to develop a common framework health science and education librarians,
implementing a program as designed that allows the central question in two authors (E.V.M. and D.S.) examined
exactly the same way every time.18 implementation evaluation, “Has CBME and drew from the literature on medical
Typically, fidelity studies rely on creating been implemented as intended?” to be education and education theory to
consensus regarding the essential consistently applied across differing identify seminal or key influential
practices one would expect to see in a contexts.21,30 literature informing the core components
particular program.19,20 Although generic and accompanying layers. The CBME
overarching categories can be used to CCF was developed in consultation with
organize the criteria (e.g., structure or Method a cross-section of stakeholders located
process elements), the end result tends To meet these challenges, Cousins at Queen’s University in Kingston,
to be a checklist of specific activities et al19 describe focusing on important Ontario, Canada, a Canadian university
used to measure and rate FOI.17,21,22 components of the innovation rather integrally involved in implementing

Academic Medicine, Vol. 94, No. 7 / July 2019 1003

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

CBME. Consisting of program directors At the beginning of each round, we Results

at the planning, early, and later stages emailed participants a survey. The five Developing a draft CCF
of implementation; institutional core components of the CBME CCF
CBME leads; education and assessment were used to create the survey for Round As we worked to make explicit our
specialists; and education researchers, 1. Initiated in January 2016, this round understanding of the intended curricular
this group of 12 individuals met twice to also included a sixth question asking change through CBME, we found two
review the draft framework. if a core component was missing from different perspectives used to frame the
the list. The principle statements from change to an outcomes-based approach
Step 2: Determining consensus the CBME CCF formed the survey for in medical education: production and
We adopted a Delphi approach to validate Round 2, which took place in March reform (Table 1).43,44 The production
two layers of the CBME CCF.35,36 The 2016. In both rounds the expert panel viewpoint draws from the manufacturing
Delphi approach was selected because, as was asked to indicate their opinion industry, where “medical schools, like
a consensus technique, it allows for views among these options: “agree as worded,” factories, can produce highly desirable
to be expressed anonymously, thereby “agree with rewording,” “disagree,” or products adapted to user needs
eliminating the influence that can occur “not sure.” Both rounds also included and desires.”43(pS41) Alternatively, the
through face-to-face meetings.37 We the opportunity for additional reform position focuses on “flexible,
focused on two layers of the framework, comments for each of the survey items. individually tailored programs that can
the core components and principle The surveys were developed by the adapt to variable rates of competence
statements, because the Delphi approach principal author (E.V.M.) and reviewed attainment.”43(pS4) These different
works best with higher-level concepts by the research team before distribution perspectives of CBME can lead to
as opposed to an in-depth exploration to the expert panel. emphasizing the importance of different
of the topic.38 We created the expert curricular elements. For example, in
panel by inviting an international group Because there is no standard method for the production perspective, emphasis is
of CBME scholars to participate in the defining consensus in a Delphi study, placed primarily on assessment for the
study. Referred to as the International it is important to be explicit about the purposes of identifying problem learners.
Competency-based Medical Education choice of decision point.42 In our case, the In the reform scenario, however, equal
Collaborators, this unique partnership research team agreed that ≥ 70% of the importance is placed on all curricular
was convened approximately 10 years ago respondents was a reasonable predefined elements, aligned in such a way so that all
to examine conceptual issues and current standard for indicating consensus. learners have the opportunity to develop
debates in CBME.39,40 With approximately The responses “agree” and “agree with the required competencies.44,45
60 members, this group participates rewording” were added together to
in monthly teleconferences to discuss determine whether we had reached the In deciding which perspective should
developments in CBME, has produced ≥ 70% level. At the end of each round the guide the creation of the CBME CCF,
a series of publications on CBME, and principal author (E.V.M.) summarized we noted that the adoption of CBME
has hosted a set of webinars as well as the results for discussion by the team is often described as a transformative
two world summits.41 For the purposes of authors. The summary included change,1,46,47 requiring a significant shift
of this study, those indicating interest a synthesis of comments regarding in behaviors.48 Consequently, we adopted
among this international group of experts implications for required revisions to the a reform characterization in developing
formed the Delphi expert panel. CBME CCF. the framework.

Table 1
Characterizations of How CBME Brings About Change: Production and Reform
Perspectives, From a Delphi Study to Produce a Framework for Evaluating
Implementation of CBME Programs, 2015–2016

Role of Production Reform

Competencies Used to standardize the “product” Form the basis for significantly redesigning assessment practices,
instructional methods, and learning experiences
Assessment practices Concerned primarily with identifying problem Focused on promoting learner growth and development through
learners frequent formative assessment
Instructional methods No overt linkage between competencies and Rich in feedback individualized to the learner and grounded in the
instruction desired competencies
Learning experiences To provide just enough training to achieve To provide rich and diverse learning experiences, steeped in clinical
proficiency (e.g., fine tuning of competencies practice where learners can stay along as required
becomes the focus)
Overall goals To increase efficiency; to reduce education costs; To ensure that physicians develop internally guided concepts of good
to reduce training time practice that are used to guide the development of expertise well
into practice
To ensure that residency education meets population needs for care
Theoretical roots Behaviorism Constructivism
Abbreviation: CBME indicates competency-based medical education.

1004 Academic Medicine, Vol. 94, No. 7 / July 2019

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

other and existing in a balanced

ecosystem.44 More specifically, in an
outcomes-based curriculum, constructive
alignment requires teaching, learning,
and assessment practices to be oriented
toward learning outcomes.44,45

We then explored these curricular elements

in light of our search for seminal literature
informing CBME. Creating an ecosystem
was supported by the description of CBME
as an outcomes-based curriculum where
“competence represents the goal of an
educational programme, and a curriculum
provides the mechanism through which
competence is to be acquired.”10(p51)
Accordingly, the articulation of explicit
outcome competencies was identified
as a central core component guiding the
development of teaching, learning, and
assessment. Furthermore, the literature
on teaching, learning, and assessment in
Figure 1 Constructive alignment of curricular elements into a balanced ecosystem, as required a CBME curriculum described distinct
for competency-based medical education. From a Delphi study to produce a framework for theoretical foundations reinforcing their
evaluating implementation of CBME programs, 2015–2016. Abbreviation: CBME indicates inclusion as separate core components.
competency-based medical education. We noted, however, that mastery learning,
a concept fundamental to CBME, was
To identify specific core components, and of “constructive alignment,” which not yet represented in the evolving
in keeping with the reform perspective, emphasizes the importance of all framework. Described as the need to arrange
we drew from the educational concept curricular elements supporting each competencies as a “sequential path through

Figure 2 The five core components of competency-based medical education: an organizing framework, from a Delphi study to produce a framework
for evaluating implementation of competency-based medical education programs, 2015–2016. Practice: Competencies required for practice and/or for
graduates form the basis for the planning and delivery of all curricular elements. Principle: Given adequate preparation, unambiguous goals, sufficient
learning resources, and time flexibility, students can, with rare exceptions, achieve defined competencies. Curricular components working together
create meaningful learning experiences. Conceptual frameworks: mastery learning and constructive alignment.

Academic Medicine, Vol. 94, No. 7 / July 2019 1005

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

the programme” while still allowing for During Round 1, over 70% agreement The principle statements were the specific
considerable flexibility in individual was reached for all five of the core focus of Round 2. Once again, agreement
learner progression, mastery learning is components (Figure 3). From the reached the predefined consensus level
a fundamental conceptual framework analysis of the comments, two specific of 70% for all of the principle statements
illuminating how CBME is supposed wording changes were identified. First, (Figure 3). Analysis of the comments did
to work.33,49 Accordingly, the sequenced that 56% of the respondents chose not demonstrate any specific themes in
progression of competencies was identified “agree with rewording” for Component relation to suggested rewording. Rather,
as a core component. Conceptualizing 2 was notable (Figure 3). Analysis of the comments indicated agreement with
CBME from a reform perspective therefore the comments resulted in a revision of the principle but a desire to add in more
resulted in identification of the following the original wording of “competencies detail and to clarify terminology.
five core components: are arranged progressively” to
“competencies and their developmental
• Outcome competencies markers are sequenced progressively” Discussion
• Sequenced progression (Figure 2). The second change was We began development of a common
in response to respondents noting framework to guide evaluation of CBME
• Tailored learning experiences consistently that learners do not program implementation by making
• Competency-focused instruction follow a predefined trajectory as they explicit our perspective that the adoption
acquire competencies. Accordingly, of CBME requires comprehensive
• Programmatic assessment for the second wording change, the curricular reform (Table 1). Drawing
phrase “the progressive development from the literature on medical education
of competencies” appearing in three and education theory, we then identified
Responding to the characterization of
of the five components was revised five core components central to CBME
constructive alignment as occurring
to “the developmental acquisition of program implementation. Initially,
within an ecosystem, we envisioned the
competencies.” All changes described based on the concept of “constructive
outcome competencies and sequenced
above were fully discussed and endorsed alignment,” the five core components
progression as central core components
by the research group. were portrayed as an ecosystem, thereby
guiding the development of learning,
teaching, and assessment practices
(Figure 1). In turn, this ecosystem is
moderated by features unique to the local
context such as the size of the program,
availability of resources including
learning experiences, and qualities of the
learning environment.

These five core components were then

used to organize and populate the full
framework, with each layer informing
the next, allowing for a core components
framework specifically customized for
a CBME program (Figure 2). We also
created a list of seminal articles emerging
from our work critical to informing each
of the core components (Supplemental
Digital Appendix 1, available at http://
links.lww.com/ACADMED/A670).

Reaching consensus on the five core

components
The five core components and associated
principle statements formed the basis
for the consensus process. Of the 59
members of the international CBME
collaborators group, 25 agreed to
participate. Because it is suggested that
a panel size should range from 10 to 30
participants,50 with an ideal size being
no less than 10,37 this panel size was Figure 3 Results of expert panel responses to Round 1, conducted in January 2016, from a
considered to be adequate. The two Delphi study to produce a framework for evaluating implementation of competency-based
rounds had response rates of 100% and medical education programs, 2015–2016. Do you agree or disagree with the inclusion of the
96% consecutively. The specific results are following core component? Panel drawn from the International Competency-based Medical
described as follows. Education Collaborators (N = 25).

1006 Academic Medicine, Vol. 94, No. 7 / July 2019

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

The possibility for local innovation

suggests that various configurations
of CBME may emerge over time. For
example, programs may use different
combinations of academic advisors,
CCCs, coaches, and mentors. Given the
potential for evolution of variability in
specific practices, implementation itself
is described as a developmental trajectory
toward fidelity.20 Over the course of
this development, however, there may
come a point where changes in surface
features compromise the attainment and/
or maintenance of fidelity.32 Because a
synthesis of studies will be required to
provide the rich explanatory analysis to
understand program impact,5 identifying
when an innovation such as CBME
actually represents the proposed model
becomes a critical question.52 The CCF
we present here can help in answering
this question, thereby avoiding a Type III
error.

As well, CBME is a complex service

intervention consisting of multiple
activities and outcomes. Therefore,
creating an in-depth understanding of
the relationship between activities and
outcomes as implementation unfolds is
critical.27 The five core components can
support a more systematic and organized
approach toward implementation
evaluation—an approach that also allows
for the identification of unanticipated
outcomes, which is an important aspect of
any evaluation effort. Ultimately, this level
Figure 4 Results of expert panel responses to Round 2, conducted in March 2016, from a Delphi of clarity is called for if we wish to establish
study to produce a framework for evaluating implementation of competency-based medical
how to enhance future educational practice
education programs, 2015–2016. Does the following statement reflect the underlying principle
of the corresponding core component? Panel drawn from the International Competency-based in a way that leads to the improvement of
Medical Education Collaborators (N = 23). patient outcomes.53

Finally, beyond assisting program

reinforcing their applicability across all referred to as more surface-level features evaluation efforts, this framework can
settings, balanced with the possibility of CBME,32 might differ from program to also provide guidance to the adoption
for variation in specific practices as program, the existence of a programmatic of CBME in any situation new to these
influenced by local context (Figure 1). We approach to assessment should be concepts. For example, framed by the
then used the five core components to common across all programs. Embracing principle of local implementation being
form the basis of a multilevel framework the influence of context is consistent guided by global considerations, or
(Figure 2). Using a Delphi approach, an with recent developments in program “glocalization,” the CCF is being used
expert panel validated the identification evaluation where it is recognized that to translate CBME into training in
of five core components (Figures 3 under dynamic conditions, “one size Taiwanese specialty medical education.54
and 4). This work addresses a number of does not fit all,” and so the reliance on
challenges identified in evaluating CBME rigid rules is being replaced by guiding Limitations of the study
programs. principles.51 FOI accordingly focuses The identification of core components
on the extent to which the program specific to CBME was dependent on
The focus on core components, rather exemplifies integrity to key concepts articulating a particular perspective seen to
than a checklist of practices, ensures that rather than specific practices.18 In this underlie the intended change. As described,
the framework can be applied equally fashion, the use of core components we chose a reform perspective in which all
across all contexts. For example, although allows for the promotion of local curricular elements, not just assessment
specific assessment practices, or what are innovation while fidelity is maintained. practices, are described as being integral

Academic Medicine, Vol. 94, No. 7 / July 2019 1007

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

to the change process. We fully recognize, deliberate linking of theory and practice, 3 Berkson L. Problem-based learning: Have
however, that as a medical education as exemplified by the full CBME CCF, the expectations been met? Acad Med.
1993;68(10 suppl):S79–S88.
innovation, CBME is still in the early stages provides a significant step forward toward 4 Vernon DT, Blake RL. Does problem-based
of implementation. It is very possible meeting this challenge, ultimately bringing learning work? A meta-analysis of evaluative
that other perspectives will continue to us one step closer to creating a refined research. Acad Med. 1993;68:550–563.
emerge as we gain more experience with understanding of the conditions under 5 Pawson R, Greenhalgh T, Harvey G, Walshe
K. Realist review—A new method of
implementation, suggesting the need for which CBME works most effectively to
systematic review designed for complex
further work in identifying and describing enhance patient care outcomes. policy interventions. J Health Serv Res Policy.
core components. 2005;10(suppl 1):21–34.
Acknowledgments: The authors would like to 6 Patton MQ. Essentials of Utilization-Focused
acknowledge and thank the group of medical Evaluation. Thousand Oaks, CA: Sage
As well, the CCF is primarily focused education leaders and stakeholders at Queen’s Publications; 2012.
on the design elements of CBME. It has University in Kingston, Ontario, Canada for 7 Funnell SC, Rogers PJ. Purposeful Program
been recently proposed, however, that providing feedback to the initial draft framework. Theory: Effective Use of Theories of Change
the learning conditions underpinning The authors would also like to acknowledge and Logic Models. San Francisco, CA: Jossey-
the educational model are also important support for conducting the Delphi process Bass; 2011.
provided through the Royal College of Physicians 8 Dobson D. Avoiding a Type III error
in influencing fidelity.47 That different in program evaluation: Results from a
and Surgeons of Canada. In particular, the authors
learning conditions may exist is would like to thank the International Competency- field experiment. Eval Program Plann.
supported by our identification of two based Medical Education Collaborators for their 1980;3:269–276.
different orientations when implementing ongoing interest in, and support of, this project. 9 Kerdijk W, Snoek JW, van Hell EA, Cohen-
CBME: a problem approach and a Schotanus J. The effect of implementing
Funding/Support: None reported. undergraduate competency-based
developmental approach.25 Indeed, there medical education on students’ knowledge
is evidence to suggest that the experience Other disclosures: Eric Holmboe receives royalties acquisition, clinical performance and
of learning in an environment that from Elsevier Publishers. perceived preparedness for practice: A
focuses on deficiencies is very different comparative study. BMC Med Educ.
Ethical approval: This study received ethical 2013;13:76.
from that driven by a growth mindset.55 10 McGaghie WC, Miller GE, Sajid AW,
clearance through the Queen’s University Health
Although the reform perspective Sciences and Affiliated Teaching Hospitals Tedler TV. Competency-Based Curriculum
underlying the core components Research Ethics Board (HSREB) #6015151. Development in Medical Education: An
assumes the importance of developing Introduction. Geneva, Switzerland: World
Previous presentations: An initial version of the Health Organization; 1978.
competence in all learners (Table 1), an
framework was presented at the World Summit 11 Bryson JM, Patton MQ, Bowman RA.
explicit connection between the core Working with evaluation stakeholders: A
on Competency-Based Education, August 27 and
components and learning conditions has 28, 2016, Barcelona, Spain. rationale, step-wise approach and toolkit.
not yet been considered. In particular, the Eval Program Plann. 2011;34:1–12.
application of complexity science could 12 Chahine S, Kulasegaram KM, Wright S, et al.
E. Van Melle is senior education scientist, Royal A call to investigate the relationship between
be particularly useful in understanding College of Physicians and Surgeons of Canada, education and health outcomes using big
this relationship.28,56–58 Ottawa, Ontario, Canada.
data. Acad Med. 2018;93:829–832.
J.R. Frank is director of specialty education, 13 Weinstein DF, Thibault GE. Illuminating
In conclusion strategy and standards, Office of Specialty Education, graduate medical education outcomes
Royal College of Physicians and Surgeons of Canada, in order to improve them. Acad Med.
Similar to PBL, it has been suggested Ottawa, Ontario, Canada. 2018;93:975–978.
that there are many challenges to 14 Glasgow NJ, Wells R, Butler J, Gear A. The
E.S. Holmboe is senior vice president for milestones effectiveness of competency-based education
the widespread uptake of CBME.59 development and evaluation, Accreditation Council
in equipping primary health care workers to
Nonetheless, CBME is an innovation for Graduate Medical Education, Chicago, Illinois.
manage chronic disease in Australian general
that is rapidly being adopted into D. Dagnone is competency-based medical practice settings. Med J Aust. 2008;188(8
practice. Initial evaluation of CBME education faculty lead and associate professor, suppl):S92–S96.
programs points to the need for a Department of Emergency Medicine, Faculty of 15 Spady WJ. Competency based education: A
Health Sciences, Queen’s University, Kingston, bandwagon in search of a definition. Educ
more sophisticated understanding of
Ontario, Canada. Res. 1977;6:9–14.
implementation. Learning from our 16 Frank JR, Mungroo R, Ahmad Y, Wang M,
experience with PBL, this article provides D. Stockley is professor and scholar in higher
De Rossi S, Horsley T. Toward a definition of
education, Office of the Vice-Provost (Teaching and
a standardized approach to moving this Learning), Queen’s University, Kingston, Ontario, Canada.
competency-based education in medicine: A
recommendation forward. With a focus systematic review of published definitions.
J. Sherbino is assistant dean, Program for Med Teach. 2010;32:631–637.
on CBME as a transformative innovation, Education Research and Development, Faculty of 17 Mowbray CT, Holter MC, Teague GB,
designed to promote the growth and Health Sciences, McMaster University, Hamilton, Bybee D. Fidelity criteria: Development,
development of all learners, the CBME Ontario, Canada. measurement, and validation. Am J Eval.
CCF promotes clarity and consistency in 2003;24:315–340.
evaluating program implementation. 18 Patton MQ. What is essential in
References developmental evaluation? On integrity,
1 Carraccio CL, Englander R. From Flexner to fidelity, adultery, abstinence, impotence,
The need for these studies to be theory competencies: Reflections on a decade and long-term commitment, integrity, and
based is also a recommendation the journey ahead. Acad Med. 2013;88: sensitivity in implementing evaluation
stemming from our experience with 1067–1073. models. Am J Eval. 2016;37:
2 Albanese MA, Mitchell S. Problem-based 250–265.
PBL.60,61 This lesson learned parallels the learning: A review of literature on its 19 Cousins BJ, Aubry TD, Fowler HS, Smith
call for more theory-driven efforts in outcomes and implementation issues. Acad M. Using key component profiles for the
the field of program evaluation.62 The Med. 1993;68:52–81. evaluation of program implementation in

1008 Academic Medicine, Vol. 94, No. 7 / July 2019

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
 Research Report

intensive mental health case management. findings into practice: A consolidated steps for outcomes-based medical education.
Eval Program Plann. 2004;27:1–23. framework for advancing implementation Acad Med. 2015;90:1215–1223.
20 Hall GE, Hord SM. Implementing Change: science. Implement Sci. 2009;4:50. 48 Eckel PD, Kezar A. Key strategies for making
Patterns, Principles, and Potholes. 4th ed. 32 Varpio L, Bell R, Hollingworth G, et al. Is new institutional sense: Ingredients to higher
New York, NY: Pearson; 2015. transferring an educational innovation education transformation. Higher Educ
21 Century J, Cassata A, Rudnick M, Freeman actually a process of transformation? Adv Policy. 2003;16:39–53.
C. Measuring enactment of innovations Health Sci Educ Theory Pract. 2012;17: 49 McGaghie WC. Mastery learning: It is
and the factors that affect implementation 357–367. time for medical education to join the 21st
and sustainability: Moving toward common 33 Bordage G. Conceptual frameworks century. Acad Med. 2015;90:1438–1441.
language and shared conceptual understanding. to illuminate and magnify. Med Educ. 50 Nair R, Aggarwal R, Khanna D. Methods of
J Behav Health Serv Res. 2012;39:343–361. 2009;43:312–319. formal consensus in classification/diagnostic
22 Holter MC, Mowbray CT, Bellamy CD, 34 Shulha LM, Whitmore E, Cousins BJ, Gilbert criteria and guideline development. Semin
MacFarlane P, Dukarski J. Critical ingredients N, al Hudib H. Introducing evidence-based Arthritis Rheum. 2011;41:95–105.
of consumer run services: Results of a principles to guide collaborative approaches 51 Patton MQ. Principles-Focused Evaluation:
national survey. Community Ment Health J. to evaluation: Results of an empirical process. The Guide. New York, NY: The Guilford
2004;40:47–63. Am J Eval. 2016;37:193–215. Press; 2018.
23 Andolsek K, Padmore J, Hauer KE, Edgar 35 Humphrey-Murto S, Varpio L, Wood TJ, 52 Horsley T, Regehr G. When are two
L, Holmboe E. Clinical Competence et al. The use of the Delphi and other interventions the same? Implications for
Committees: A Guidebook for Programs. 2nd consensus group methods in medical reporting guidelines in education. Med Educ.
ed. Chicago, IL: Accreditation Council for education research: A review. Acad Med. 2018;52:141–143.
Graduate Medical Education; 2017. https:// 2017;92:1491–1498. 53 Cook DA, West CP. Perspective:
www.acgme.org/Portals/0/ACGME 36 Valentijn PP, Vrijhoef HJ, Ruwaard D, Reconsidering the focus on “outcomes
ClinicalCompetencyCommitteeGuidebook. Boesveld I, Arends RY, Bruijnzeels MA. research” in medical education: A cautionary
pdf. Accessed March 28, 2019. Towards an international taxonomy of note. Acad Med. 2013;88:162–167.
24 Royal College of Physicians and Surgeons of integrated primary care: A Delphi consensus 54 Chou F, Hsiao C, Chen CC. Does CBME
Canada. Competence by Design. Competence approach. BMC Fam Pract. 2015;16:64. translate across systems? Learning from the
Committees. Ottawa, Ontario, Canada; 37 Waggoner J, Carline JD, Durning SJ. Is there Taiwanese experience. Workshop presented
Royal College of Physicians and Surgeons a consensus on consensus methodology? at: International Conference on Residency
of Canada; 2017. http://www.royalcollege. Descriptions and recommendations for Education; October 18–20, 2018; Halifax,
ca/rcsite/cbd/assessment/competence- future consensus research. Acad Med. Nova Scotia, Canada.
committees-e. Accessed March 28, 2019. 2016;91:663–668. 55 Dweck CS. Mindset: The New Psychology
25 Hauer KE, Chesluk B, Iobst W, et al. 38 Hsu CC, Sanford BA. The Delphi technique: of Success. New York, NY: Random House;
Reviewing residents’ competence: A Making sense of consensus. Pract Assess Res 2016.
qualitative study of the role of clinical Eval. 2007;12(10). http://pareonline.net/ 56 Hawe P, Shiell A, Riley T. Complex
competency committees in performance getvn.asp?v=12&n=10. Accessed April 10, interventions: How “out of control” can
assessment. Acad Med. 2015;90:1084–1092. 2019. a randomised controlled trial be? BMJ.
26 Craig P, Dieppe P, Macintyre S, Michie S, 39 Frank JR, Snell L, Englander R, Holmboe 2004;328:1561–1563.
Nazareth I, Petticrew M. Developing and ES; ICBME Collaborators. Implementing 57 Cilliers P. Understanding complex systems.
Evaluating Complex Interventions: New competency-based medical education: In: Sturmberg JP, Martin CM, eds. Handbook
Guidance. London, UK: Medical Research Moving forward. Med Teach. 2017;39: of Systems and Complexity in Health. New
Council; 2006. https://mrc.ukri.org/ 568–573. York, NY: Springer; 2013.
documents/pdf/complex-interventions- 40 Snell LS, Frank JR. Competencies, the tea 58 Snyder S. The Simple, the Complicated
guidance. Accessed March 28, 2019. bag model, and the end of time. Med Teach. and the Complex: Educational Reform
27 Van Melle E, Gruppen L, Holmboe ES, 2010;32:629–630. Through the Lens of Complexity Theory.
Flynn L, Oandasan I, Frank JR; International 41 International Competency-based Medical Organisation for Economic Co-operation
Competency-based Medical Education Education Collaborators. http://gocbme. and Development (OECD) Education
Collaborators. Using contribution analysis org/icbme-site/index.html#about. Accessed Working Papers. Vol 96. Paris, France: OECD
to evaluate competency-based medical March 28, 2019. Publishing; 2013.
education programs: It’s all about rigor in 42 von der Gracht HA. Consensus measurement 59 Boucher A, Frank JR, Van Melle E,
thinking. Acad Med. 2017;92:752–758. in Delphi studies: Review and implications Oandasan I, Touchie C. Competency
28 Holmboe E. The journey to competency- for future quality assurance. Tech Forecast Based Medical Education: A White Paper
based medical education: Implementing Soc Change. 2012;79:1525–1536. Commissioned by the AFMC Board of
milestones. Marshall J Med. 2017;3. 43 Hodges BD. A tea-steeping or i-Doc model Directors. Ottawa, Ontario, Canada:
http://dx.doi.org/10.18590/mjm.2017. for medical education? Acad Med. 2010;85(9 Association of Faculties of Medicine of
vol3.iss1.2 suppl):S34–S44. Canada; 2017. https://mededconference.
29 Pérez D, Van der Stuyft P, Zabala Mdel C, 44 Biggs J. Teaching for Quality Learning at ca/sites/default/files/AFMC-
Castro M, Lefèvre P. Erratum to: “A modified University. 2nd ed. New York, NY: Open CompetencyBasedMedicalEducation_en.pdf.
theoretical framework to assess implementation University Press; 2003. Accessed March 28, 2019.
fidelity of adaptive public health interventions.” 45 Morcke AM, Dornan T, Eika B. Outcome 60 Norman GR, Schmidt HG. Effectiveness
Implement Sci. 2016;11:106. (competency) based education: An of problem-based learning curricula:
30 Carraccio C, Englander R, Van Melle E, exploration of its origins, theoretical basis, Theory, practice and paper darts. Med Educ.
et al; International Competency-based and empirical evidence. Adv Health Sci Educ 2000;34:721–728.
Medical Education Collaborators. Advancing Theory Pract. 2013;18:851–863. 61 Norman GR, Schmidt HG. Revisiting
competency-based medical education: A 46 Carraccio C, Wolfsthal SD, Englander R, “Effectiveness of problem-based learning
charter for clinician–educators. Acad Med. Ferentz K, Martin C. Shifting paradigms: curricula: Theory, practice and paper darts.”
2016;91:645–649. From Flexner to competencies. Acad Med. Med Educ. 2016;50:793–797.
31 Damschroder LJ, Aron DC, Keith RE, Kirsh 2002;77:361–367. 62 Brousselle A, Champagne F. Program theory
SR, Alexander JA, Lowery JC. Fostering 47 Holmboe ES, Batalden P. Achieving the evaluation: Logic analysis. Eval Program
implementation of health services research desired transformation: Thoughts on next Plann. 2011;34:69–78.

Academic Medicine, Vol. 94, No. 7 / July 2019 1009

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.