Chambers et al.

Implementation Science 2013, 8:117

http://www.implementationscience.com/content/8/1/117
Implementation
Science

DEBATE Open Access

The dynamic sustainability framework: addressing

the paradox of sustainment amid ongoing change
David A Chambers1*, Russell E Glasgow2 and Kurt C Stange3

Abstract
Background: Despite growth in implementation research, limited scientific attention has focused on understanding
and improving sustainability of health interventions. Models of sustainability have been evolving to reflect
challenges in the fit between intervention and context.
Discussion: We examine the development of concepts of sustainability, and respond to two frequent assumptions —
‘voltage drop,’ whereby interventions are expected to yield lower benefits as they move from efficacy to effectiveness to
implementation and sustainability, and ‘program drift,’ whereby deviation from manualized protocols is assumed to
decrease benefit. We posit that these assumptions limit opportunities to improve care, and instead argue for
understanding the changing context of healthcare to continuously refine and improve interventions as they are sustained.
Sustainability has evolved from being considered as the endgame of a translational research process to a suggested
‘adaptation phase’ that integrates and institutionalizes interventions within local organizational and cultural contexts.
These recent approaches locate sustainability in the implementation phase of knowledge transfer, but still do not address
intervention improvement as a central theme. We propose a Dynamic Sustainability Framework that involves: continued
learning and problem solving, ongoing adaptation of interventions with a primary focus on fit between interventions and
multi-level contexts, and expectations for ongoing improvement as opposed to diminishing outcomes over time.
Summary: A Dynamic Sustainability Framework provides a foundation for research, policy and practice that supports
development and testing of falsifiable hypotheses and continued learning to advance the implementation, transportability
and impact of health services research.
Keywords: Sustainability, Maintenance, Adaptation, Dissemination, Implementation, Framework, Model

Background While this is progress, frequently used conceptualizations

As implementation science has grown [1,2], researchers of sustainability implicitly replicate assumptions and
have advanced from study of facilitators and barriers limitations inherent in the traditional research-to-practice
that influence uptake of effective programs and policies pathway [4,5], or in its more recent conceptualization
to investigations of strategies to improve uptake. How- as translational research [6]. These conceptualizations of
ever, often studies evaluate only initial intervention adop- knowledge translation often assume that interventions are
tion and implementation. Sustained practice change and optimized prior to implementation, and that they are largely
broader scale-up of interventions [3] rarely are investi- independent of the context in which they are delivered [7].
gated, often due to the constrained timeframes for research The presumed linear model of intervention develop-
that are set by grant mechanisms, and the budgetary and ment, efficacy testing and implementation has resulted
political necessity of many decision-makers to take on a in the development of an armamentarium of efficacious
short-term lens. healthcare treatments, preventive strategies, and public
Recently, there has been interest in understanding and health interventions. While these discoveries have made
influencing the sustainability of implemented interventions. advances in a number of health domains, they are often
difficult to implement in a myriad of practice settings
* Correspondence: [email protected] and even harder to sustain over time and in many real
1
Division of Services and Intervention Research, National Institute of Mental world and low-resource settings [8]. In addition, inter-
Health, 6001 Executive Blvd, Rockville, MD, USA
Full list of author information is available at the end of the article ventions are traditionally expected to perform worse in

© 2013 Chambers et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative
Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and
reproduction in any medium, provided the original work is properly cited.
Chambers et al. Implementation Science 2013, 8:117 Page 2 of 11
http://www.implementationscience.com/content/8/1/117

real-world practice than in the laboratory or the rarified manuals to ensure its consistent delivery with fidelity [16],
clinical trial setting. We argue for a new approach to and then minimizing deviations from the intervention. For
sustainability that instead integrates the themes of adap- example, the COMMIT stop smoking national project had
tive, contextually sensitive continuous quality improve- expert researchers design the complex, lengthy interven-
ment (CQI) and a learning healthcare system with the tion protocol based on research evidence, and then gave a
challenge of intervention sustainment. several hundred page manual of operations to local staff
The purposes of this article are to explicate: to implement in their communities [17].
The importance of internal validity to the scientific pro-
1. Evolving understandings of sustainability and of cess should not be ignored, but its overemphasis relative to
related concepts of CQI and the learning generalizability and adaptation runs the risk of creating
healthcare system; interventions that will not fit within different, complex or
2. An iterative, dynamic approach to sustainability, changing settings and of failing to benefit settings, clini-
termed the ‘Dynamic Sustainability Framework’ (DSF) cians, and patient populations who are underrepresented in
that integrates and extends these concepts; and the intervention testing process [7,16]. Two key implicit
3. Implications of this framework for research, policy, assumptions within the traditional intervention develop-
and practice. ment approach may limit ultimate progress toward inter-
vention sustainability and population impact:
Given the variation with which terms central to dis- First, interventions are often developed with the idea
semination and implementation research can be used, that they can be optimally constructed, manualized, and
we include a table that lays out working definitions for then tested in a single form applicable across settings
the central terms of this debate Table 1. and over time. Efficacy trials are designed to screen out
noise in the system (patient comorbidities, competing de-
mands and skill variance of clinicians, resource limitations,
Moving beyond ‘voltage drop’ and ‘program drift varying motivations of patients) [7], and thus maximize out-
While the traditional linear process of intervention deve- comes. As interventions move to effectiveness and into
lopment, derived from pharmaceutical medication deve- implementation, one expects that the individual benefit of
lopment models [13], has often resulted in the creation of the intervention will likely drop, due to the added complex-
initially successful interventions, it may be less helpful in ity of heterogeneous patients, providers and settings. This is
enabling these innovations to maximally benefit health. A referred to as ‘voltage drop’ (Figure 1). The assumption of
linear approach may be particularly challenging to apply ‘voltage drop’ results in missed opportunities to refine and
to complex, multi-component interventions, psychosocial improve the intervention, instead concluding that the de-
treatments, treatment of the growing number of people clining benefit is expected and acceptable, and the best pos-
with multimorbid conditions [14,15], and systemic ap- sible outcome is that which is achieved at the efficacy stage.
proaches to care [15]. Linear approaches place a premium Second, the assumption that interventions can be opti-
on creating and ‘freezing’ an intervention, developing mally constructed in the early stages of the development
and testing process, independent of context, suggests
Table 1 Definitions of key terms used in this paper that, even at the stages of implementation and sustain-
Term Definition ability, change to the intervention is expected to have
Implementation The process of putting to use or negative consequences, and that the further a practi-
integrating evidence-based interventions tioner deviates from the manual, the lower the benefit.
within a setting [9].
This is the concept of ‘program drift’ (Figure 1). Delive-
Sustainability To what extent an evidence-based ring the intervention within an efficacy trial may require
intervention can deliver its intended
benefits over an extended period of time adherence to protocols that are challenging to deliver
after external support from the donor within real-world practice. Fidelity ratings then assume
agency is terminated [9]. that 100% fidelity to original protocols will yield optimal
Sustainment The continued use of an intervention outcomes, and effort is expended to ensure that practi-
within practice [10]. tioners do not deviate from the manual. Where clini-
Voltage drop The phenomenon in which interventions are cians do deviate from the protocol, the field expects that
expected to yield lower benefits as they move
from efficacy to effectiveness and into
the resulting ‘program drift’ will compromise outcomes
real world use (adapted from [11]). [12]. We see that this over-reliance on quality assurance
Program drift The phenomenon whereby deviation from to prevent ‘program drift’ leads to extensive pressure on
manualized protocols in real-world delivery of real-world practices to adhere to the intervention proto-
interventions is expected to yield decreasing cols without evidence that this adherence will lead to
benefit for patients (adapted from [12]).
optimal outcomes. Quality assurance may inadvertently
Chambers et al. Implementation Science 2013, 8:117 Page 3 of 11
http://www.implementationscience.com/content/8/1/117

Figure 1 Program drift and voltage drop. Illustrating the concepts of ‘program drift,’ in which the expected effect of an intervention is presumed
to decrease over time as practitioners adapt the delivery of the intervention (A), and ‘voltage drop,’ in which the effect of an intervention is presumed
to decrease as testing moves from Efficacy to Effectiveness to Dissemination and Implementation (D&I) research stages (B).

hamper sustainability and ongoing improvement, cus- initial process of embedding interventions within set-
tomization and optimization of interventions to the det- tings and ‘sustainability,’ which relates to the extent that
riment of population health. these interventions can continue to be delivered over time,
In contrast, we reject the notion that an intervention can institutionalized within settings, and have necessary ca-
be optimized prior to implementation, and explicitly reject pacity built to support their delivery.
the validity of ‘program drift’ and ‘voltage drop.’ Rather, we Recent articles [22-24] have advanced the idea of an
suggest that the most compelling evidence on the maximal adaptation phase that bridges from the initial implemen-
benefit of any intervention can only be realized through tation effort to a longer-term sustainability phase. They
ongoing development, evaluation and refinement in diverse argue the need to examine the fit between the practice
populations and systems [18]. Instead of viewing contextual setting and the intervention and make changes necessary
factors as interfering with the delivery of an effective inter- to improve the integration of the intervention into on-
vention and needing to be controlled, we see the opportu- going care processes. This is consistent with the institu-
nity to learn about the optimal fit of an intervention to tional theory of organizations, which argues that the final
different care settings [2]. For example, strategies have been stage of innovation requires the ‘institutionalizing’ of the
developed to adjust organizational characteristics (e.g., cul- new practice so that it becomes a working part of the
ture, climate, structure) to enable improved fit between the organization [25].
intervention and the setting [19]; harnessing the under- As a consequence, assessment of organizational cha-
standing of context can enable beneficial adaptation of the racteristics (e.g., structure, climate, culture, resources) is
intervention and improve sustainability. Without rejecting seen as an essential component of sustainability, and in-
these assumptions, we reify early phase interventions tested deed, the fit between context and the intervention is at
in the most artificial settings, set quality assurance of inter- the center of a sustainability phase [24]. There has also
ventions as an optimal outcome, and miss opportunities been an emphasis on planning for sustainability much
for continued learning and development. earlier in the intervention process [26]. Recent approaches
to sustainability locate key efforts squarely in the imple-
Understanding and advancing sustainability research mentation phase, arguing that once a practice has been
As the field of implementation science has matured [20,21], implemented within a care system, those who manage the
more emphasis has been placed on understanding sustain- delivery of that practice should turn their attention to
ability. Researchers have recognized that implementation ensuring that the practice can be maintained over time
of interventions, which can often require substantial re- [6,24]. Authors typically suggest that this entails attention
sources, is meaningless without successful long-term use. to issues of long-term financing, training of the workforce,
Following Rabin et al.’s glossary for Dissemination and supervision, and organizational support for the practice
Implementation Research in Health [9], we draw dis- [26,27]. A characteristic of this approach is to postpone
tinctions between ‘implementation,’ which relates to the emphasis on sustainability until after implementation is
Chambers et al. Implementation Science 2013, 8:117 Page 4 of 11
http://www.implementationscience.com/content/8/1/117

well underway, assuming that implementation and sus- where we have seen inattention to constant change limit
tainability are sufficiently independent. the ability to which implemented interventions are sus-
Authors have also highlighted the utility of assessing tained over time in complex clinical and community set-
outcomes of those who have received the practice, some- tings. The DSF (Figure 2) emphasizes that change exists in
thing infrequently collected in routine practice [24,28]. the use of interventions over time, the characteristics of
Seldom is it demonstrated that continued delivery of an practice settings, and the broader system that establishes
intervention confers benefit on the patient population the context for how care is delivered. As classical thinking
that receives the intervention or the system that delivers eloquently captures, change impacts the ability of health
it (e.g., cost containment, efficiency of care, quality met- interventions to be optimally used and sustained over
rics). Measurement of outcomes over time to determine time. This dynamism exists in the evidence base for inter-
continued benefit has been shown to support sustain- ventions that links causal factors to health outcomes, as
ability of the practice [29,30]. judged by the continual stream of new publications in aca-
Recent implementation projects have created new demic journals that add to available evidence on the ef-
tools and scales to study sustainability, including needs fectiveness of interventions, as well as ongoing practice
assessments, long-term action plans, tracking of pro- surveillance systems that capture intervention impact.
gram adaptation, financial planning, mapping of com- Dynamism exists in the interventions that support the evi-
munity networks, and measurement of the degree to dence, which acknowledge ad hoc adaptation and experi-
which practices are integrated and institutionalized into mentation of evidence-based interventions. Furthermore,
service systems [31-33]. While this emerging focus on it exists in a constantly changing multi-level context [34],
sustainability is an advance, many studies still assume a internal to a clinical or community setting and the broader
largely static service delivery system that needs to be care system, be it an organization, community, county,
assessed only at key time points, but not in an ongoing state or country.
manner. To better reflect complexity and change within The DSF, like many implementation models, centers
the system and in context, a more dynamic approach to on a few major elements: the intervention, the context
sustainability is needed. in which the intervention is delivered, and the broader
ecological system within which the practice settings exist
Framework and operate. Distinct from those models, however, is the
The dynamic sustainability framework (DSF) consideration of these elements over time. The interven-
As Heraclitus observed, ‘The only constant is change.’ tion, as shown in the figure, often includes a set of in-
The Dynamic Sustainability Framework has developed dividual components chosen for their ability to effect
out of our evolving thinking and our collective experience behavior or biochemical change, an assumed set of cha-
in conducting and advancing implementation science, racteristics defining who should deliver the intervention,

FIT
FIT
PRACTICE ECOLOGICAL
SETTING SYSTEM
INTERVENTION (Context) • Other Practice
• Components Settings
• Staffing T0 T1 Tn
• Practitioners T0 T1 Tn • Info Systems
T0 T1 Tn • Policy
• Outcomes • Regulations
• Org. Culture/
• Delivery Platform • Market Forces
Climate Structure
• Business Model • Population
• Training Characteristics
• Supervision

Figure 2 The dynamic sustainability framework. Illustrating the goal of maximizing the fit between interventions, practice settings, and the
broader ecological system over time (represented by T0, T1,…,Tn), each of which has constituent components that may vary.
Chambers et al. Implementation Science 2013, 8:117 Page 5 of 11
http://www.implementationscience.com/content/8/1/117

targeted, patient-centered outcomes that the interven- appropriate quality assurance within healthcare systems
tion should generate as a result of its use, and a delivery where clear assessments of an appropriate standard of care
platform (e.g., face-to-face, telephonic, web-based, mo- are made through knowledge of core intervention com-
bile health app, etc.). Other constructs may also define ponents. However, the DSF recognizes the limitations of
the intervention. intervention evidence solely from clinical trials and argues
The DSF anchors the ultimate benefit of the interven- that quality improvement processes focused on interven-
tion in terms of its ability to fit within a practice setting, tion optimization are ultimately more relevant to achieve
typically a clinical or community setting. This context car- sustainment.
ries its own set of characteristics, including human and The DSF, which has benefitted from the authors’ on-
capital resources, information systems, organizational cul- going dialogue with the Implementation Science com-
ture, climate and structure, and processes for training munity about the challenge of sustainability, follows the
and supervision of staff. The DSF, consistent with other spirit of a number of existing models that emphasize
models, argues that these practice characteristics will three things—importance of context, the need for on-
directly influence the ability of the intervention to reach going evaluation and decision-making, and the goal of
the patient population that could benefit, and thus mea- continuous improvement. These include Wandersman’s
surement of these contextual constructs is paramount to Getting to Outcomes model [31], Continuous Quality
resolving fit. Improvement (CQI) [34], system dynamics [35], com-
At a third level, the DSF identifies the ecological system plexity theory [36], adaptive management [37], and the
as an additional driver of the successful implementation Evidence Integration Triangle [30]. In addition, the DSF
and sustainability of an intervention. The ecological sys- is consistent with alternative views of organizational
tem consists of many practice settings that influence those development [38] and the principles of system science
working to incorporate a particular intervention, as well [39]. Distinct in the DSF from many of these other
as the legislative and regulatory environment, characteris- models is the emphasis on omnipresent change, and the
tics of local, regional, state and national markets, and central goal of continuously optimizing the fit between
characteristics of the broad population. The ecological sys- the intervention and a dynamic delivery context to achieve
tem is influenced by changes to available interventions maximal benefit. The DSF is anchored around the follo-
and practice settings, and in turn, influences them. wing seven tenets, for which we think there is evidence,
Specific to the DSF, as emphasized by the dotted lines but recommend explicit testing in this context:
in Figure 2, is the expectation that change is constant at
each of these levels (and ripples across multiple levels), An intervention should not be optimized prior to
and thus the success of an intervention to be sustained implementation, or even prior to ‘sustainability
over time lies in the measured, negotiated, and recipro- phase’ onset
cal fit of an intervention within a practice setting and Interventions benefit from ongoing optimization as they
the practice setting within the larger ecological system. are applied in different contexts [37]. The evidence that
The DSF suggests that optimal fit requires that charac- supports the benefits of health interventions arises from
teristics of the intervention, practice setting, and eco- trials that represent a very small slice of the diversity of
logical system be consistently tracked, using valid, reliable demographics, preferences, and health status of the
and relevant measures, and expects that interventions, population at large [7,40], but we should not expect evi-
settings and the ecological system should change over dence collected in one set of narrow, relatively optimal
time, particularly where data can suggest improve- circumstances to apply perfectly in other, vastly different
ments for each to better meet the needs of patients, contexts [41]. A ‘corollary’ of this recommendation is
the skills and resources within the practice setting, and that, other things being equal, quality improvement ap-
the larger ecology. proaches that involve adjusting and refining program
The DSF is intended to suggest a new paradigm to should be more effective than ‘quality assurance’ proce-
consider the long-term use and ongoing improvement of dures that emphasize fidelity to an initial protocol.
interventions, recognizing the limitations of the evidence
base made available through efficacy and effectiveness trials, Interventions can be continually improved, boosting
and allowing that continuous exposure of the intervention sustainment in practice, and can enable ongoing learning
to new populations, new contexts, and new innovations among developers, interventionists, researchers
can result in continued improvement of resulting out- and patients
comes, thus minimizing the perils of ‘program drift’ and There is tremendous opportunity to aggregate evidence
‘voltage drop.’ Indeed, the DSF posits that ongoing quality on the real-world impact of interventions when used in
improvement of interventions is the ultimate aim, not qua- practice. We can apply models of continual refinement
lity assurance of them. To be clear, we see value in ensuring that have been the cornerstone of software development
Chambers et al. Implementation Science 2013, 8:117 Page 6 of 11
http://www.implementationscience.com/content/8/1/117

(e.g., Firefox, Reaper, iTunes), as well as Web 2.0 sites (e.g., could maintain or possibly increase over time. This echoes
Wikipedia, Facebook, etc.) [42]. By augmenting trial data the computer industry, where each new release of a hard-
with practice-based evidence, we can understand much ware or software line is expected to be better than the
more about what works for whom, the question under- prior version. It also finds consonance with the evolution
lying personalized medicine [43,44]. This articulation of of the flu vaccine, which is constantly refined in response
the DSF suggests the need for a long-term plan to commit to the changing nature of the influenza virus each season.
resources for training and ongoing improvement. One im- A culture of improvement is central to ongoing interven-
plication of the DSF is that intervention impact can also tion use and treats improvement of the intervention as
be enhanced through increases in efficiency. The field has central to the sustainability process [33].
developed a plethora of multi-component interventions,
often without studies that determine what the minimal set Programs should be more likely to be maintained when
of components are needed to ensure benefit [45]. The there is strong ‘fit’ between the program and the
DSF, congruent with the Consolidated Framework for implementation setting
Implementation Research [46], emphasizes the impor- The concept of ‘fit’ has been discussed by other authors
tance of streamlining interventions to peel away compo- (e.g., Estabrooks, Glasgow, Dzewaltowski), [50] and goes
nents that may not be central to improving outcomes or back at least as far as Rogers’ Diffusion of Innovations
to adapt intervention components to a particular context. [51], where the concept of reinvention evoked the notion
of departing from the original intervention concept to
Ongoing feedback on interventions should use practical, ‘create’ a new version suited to the preferences and con-
relevant measures of progress and relevance straints of the local context. Fit is a multi-level construct
Too often, intervention trials focus on markers that are and involves alignment along multiple dimensions [50].
psychometrically valid but of less relevance to patients and The DSF posits that fit will likely change over time, due
clinicians [3,47]. For example, very specific, intervention- to changes in the way in which an intervention is deli-
related symptomatic scales may be most sensitive to vered, the characteristics of patients, providers and set-
change, but there is little guarantee that the measured tings, and the broader ecological system within which
change translates into a tangible, functional benefit for the healthcare settings reside. Attention to this fit, through
patient [48]. The DSF thus suggests the use of measures ongoing assessment and quality improvement efforts,
such as checklists that are relevant to desired outcomes of should improve sustainment and ultimately identify oppor-
patients, as well as sensitive to the ‘fit’ between interven- tunities for intervention improvement.
tions and context, and can be feasibly implemented [49].
Across each of the changes in Figure 2, we see available Organizational learning should be a core value of the
streams of data that can offer leverage points for improving implementation setting
interventions. Environmental changes, for example, can be While training and ongoing analyses exist in many organi-
tracked via population surveys, and market and claims data. zations, the demands imposed by multiple levels of change
Practice changes can be captured through electronic health require learning to be central to organizational activity for
records, claims data and practice surveys. Evidence reviews interventions to become sustainable. The context both
can provide key information on knowledge changes, and within an organization and in the broader ecological sys-
policy changes can be tracked via available Federal and tem is constantly changing, and requires a ‘learning orga-
Non-Profit sources (e.g., CMS, Kaiser Family Foundation, nization’ [52] to engage in problem-solving capacity at
etc.). This is consistent with the CQI model, but with multiple levels [53]. Organizational learning should also
specific emphasis on the ongoing refinement of the inter- target appropriate adaptation of evidence-based interven-
vention to counteract the assumptions of ‘Program Drift.’ tions, possibly in rapid learning cycles [44,54], followed by
ongoing assessment and feedback loops. The DSF is con-
Voltage drop is NOT inevitable gruent with the concepts of the learning healthcare system,
We reject the assumption that the more diverse and again with the emphasis on learning how to better develop,
complex a patient population is, the smaller the benefit deliver and sustain interventions.
of intervention, referred to above as ‘voltage drop’ [11].
This stems from an expectation that intervention studies Ongoing stakeholder involvement throughout should
require control of the environment to isolate a treatment lead to better sustainability
effect [7]. If we embrace CQI of the specific health inter- Continuously engaging stakeholders throughout the plan-
vention, we expect that with more experience, we will ning, implementation and adaption processes should help
better be able to adapt interventions to contexts and pa- increase the fit between the intervention and the local
tients. As we learn more about what works and what context, and help address evolving issues that might inter-
doesn’t and adjust protocols accordingly, the ‘voltage’ fere with sustainability. Just as researchers have proposed
Chambers et al. Implementation Science 2013, 8:117 Page 7 of 11
http://www.implementationscience.com/content/8/1/117

intervention development processes [12] that focus on analysis, opportunities for knowledge development and in-
the ultimate site where interventions will be delivered, we corporation of the ‘noise’ within healthcare contexts. Ra-
argue that partnership among all relevant stakeholders is ther than seeking to simplify the phenomenon of study,
essential to maintaining and improving interventions within either by avoiding adaptation of interventions, or assuming
care settings. the context to be unchanging, the DSF embraces change as
As Figure 3 depicts, we view sustainability as akin to a central influence on sustainability. Adaptation is ex-
the challenge of fitting a puzzle piece within an evolving pected, and even encouraged. Assessment of care settings
large tableau. Without sensitivity to the characteristics and outcomes is ongoing and incorporated within practice,
of the intervention, practice setting and the larger sys- and staffing and policy changes are incorporated in sus-
tem, there is little expectation that the intervention will tainability planning. Perhaps the biggest contrast of the
fit well within the setting, and as the context changes static and dynamic views is that the static view limits
(noted by the changing shape in the figure), sustainment lessons that can in turn provide feedback to other areas of
will be harder and harder to achieve. However, by exa- science; the DSF views an abundance of ongoing evidence
mining and adapting the intervention to a changing con- that can be cycled to continuously improve intervention
text, we believe that sustainment is not only possible, design, testing, and ongoing system change.
but that the utility of the intervention can be optimized.
The experience of delivering the intervention in vivo Discussion
over time serves to inform the ongoing evolution of the Implications of the DSF
intervention (noted by the change in shape of the puzzle This initial formulation of the DSF has implications for
piece). By concentrating on the dynamic ‘fit’ between an future practice, research and policy. For practice, the
intervention and its delivery context as the core ingre- DSF highlights the need for continuous assessment of
dient underlying sustainability, we embrace opportu- the local context, not just prior to implementation. This
nities to refine and improve the intervention. enables care settings to better manage the fit between
their resources, needs and the interventions, including
Contrasting static and dynamic views of sustainability generating consistent feedback on how interventions are
Table 2 compares static views of sustainability with the delivered to diverse patients and how patients do as a
DSF, offering sharp contrasts in data collection and result. The collection and analysis of this information

Figure 3 Using the dynamic sustainability framework as an engine for quality improvement. The DSF depicts a dynamic view of
sustainability, which allows for the evolution of an intervention within a changing delivery system. The changes in the shape of the puzzle pieces
and of the contexts reflects the ongoing change to interventions, practice settings, and care systems, and shows the use of quality improvement
methods to optimize the ‘fit’ and improve the public health benefit of sustained use of interventions.
Chambers et al. Implementation Science 2013, 8:117 Page 8 of 11
http://www.implementationscience.com/content/8/1/117

Table 2 Contrasting static views of sustainability with the dynamic sustainability framework
Static view Dynamic sustainability view
Adaptation Bad; avoided/eliminated Inevitable; encouraged, monitored and guided by evidence
Context assessment Initial or during implementation Ongoing
Outcomes assessment During study by researchers Incorporated as part of organization
Review of evidence Initial- from efficacy studies Ongoing; from convergent sources including replications
Staffing issues (e.g., turnover) and variations Ignored/feared Planned for; investigated
Generates new knowledge No Yes, feedback to other areas of science and to earlier stages
Note. This table contrasts more traditional static views of sustainability, in which efforts are made to minimize change and retain the original form of an
intervention, from a dynamic sustainability view that we suggest in the DSF, in which change is inevitable and can lead to better fit and ultimately better impact
of interventions.

allows practitioners to make informed decisions about [57,58]. Instead of a small team of researchers developing
how best to utilize existing interventions, allows for po- a priori an ‘optimal,’ static product, a large and often vir-
tential enhancements to the interventions to be made tual community including users and consumers continu-
and shared, and offers better information on which to ously upgrades dynamic products.
make decisions to cease delivering interventions that do This initial formulation of the DSF also has implications
not have benefit. The intention is to recognize and support for policy. Incentives are needed to support ongoing adap-
rapid learning, real-time problem-solving organizations [54] tation of interventions, particularly where evidence is li-
that are full partners in the generation of knowledge, not mited, specifically including monitoring of progress and
just its application. Thus, the DSF promotes the use of mul- documentation of adaptations, using quality measures rele-
tiple methods of planning for sustainability, including simu- vant to stakeholders and patients. In addition, research
lation modeling of the impact of different decisions, pilot funders must determine how to support longer-term pro-
testing of adaptations within local contexts, and continued jects related to sustainability with flexible research designs,
experimentation. Perhaps an even greater benefit to prac- since the ultimate benefit of integrating and modifying
tice would come through pooling of data across a larger set interventions may not be evident for many years. In
of sites, practitioners and patients, something done with addition, infrastructure to support pooling of ‘practice-
success for chronic disease [55,56]. based evidence’ will be needed, in order to ensure suffi-
For research, the DSF dispels the notion that interven- cient information is available about long-term use and
tion development, refinement and improvement are com- adaptation of interventions. The DSF aligns directly with
pleted prior to real world implementation. In contrast, a number of existing policy initiatives, at national, state
we suggest that development and refinement is never and local levels, including the advance of Patient-Centered
complete. Rather, sustainability is the process of managing Medical Homes, Accountable Care Organizations, Pay for
and supporting the evolution of an intervention within a Performance initiatives, and support for local demonstra-
changing context. We recommend (and welcome testing tion projects.
of the idea) that programs that monitor context and adjust We recognize that this conceptualization of the DSF
accordingly do better long-term. In addition, we see re- model should, consistent with its internal logic, be refined
search studies testing whether settings and programs and improved over time. Whether this happens through
using ongoing CQI or other means of feedback improve- testing of the tenets laid out in the previous section or
ment perform better over time. More broadly, we see the contributions of others’ theoretical or empirical studies,
DSF as changing the notion of a linear transition from we offer the DSF as the beginning of a longer debate. For
research to practice into a shared process of continual ex- example, while Figure 2 shows three levels (intervention,
perimentation and analysis through the use of both prac- practice setting, ecological system), we appreciate that
tice settings and ecological systems to track changes and many more levels of the system exist than what we have
assess evolving fit between interventions and practice set- depicted in the figure. We see further specification of the
tings. Principles of ‘crowd sourcing’ made popular within interrelationships of those levels as a useful area for fur-
the IT industry and resulting in open source products like ther research and development. In addition, we see the
Firefox, Wikipedia and Reaper, blur the lines between re- utility of aligning the DSF with alternative methods of de-
search and practice. We see this in the evolution of the veloping and testing dissemination and implementation
electronic health record, and the rapidly evolving patient interventions from the prevailing linear model. In an effort
health record, which is dramatically changing the scenario to present examples of how the DSF might be used to
from one where the medical expert makes all the decisions consider the sustainability of various types of interven-
to one of collaborative care and shared decision-making tions, we have included Table 3.
Chambers et al. Implementation Science 2013, 8:117 Page 9 of 11
http://www.implementationscience.com/content/8/1/117

Table 3 Illustrative examples of the use of DSF for different types of interventions
Intervention example Applying DSF principles
Clinical guidelines for pharmacotherapy

• Clinical guidelines for pharmacotherapy to treat a range of chronic • Assessing appropriate fit between guidelines and the care setting will
diseases have been developed, implemented and refined as new require analysis of multiple streams of data, including administrative,
compounds have reached the market and new evidence has been clinical, organizational and epidemiologic.
gathered about the relative benefit of different medications.
• The DSF suggests that collecting benchmarks over time on patient
• The influences on prescribing practices exist at the patient level outcomes, adaptations in algorithms used, available evidence (from the
(preferences and predictors of response), clinician level (practice patterns, literature, healthcare systems and patient populations), and contextual
level of training), system level (formulary design, insurance coverage, factors could result in improvements to the guidelines, to the capacity of
adherence monitoring). the health system to more seamlessly integrate the guidelines, to the
ecological system that could improve access, quality and health outcomes.
• Each influence will impact guideline implementation and overall benefit
of care for patients served within the health system.
Psychotherapy for mood disorders

• Manualized evidence-based psychotherapies for mood disorders have • The DSF suggests manualized psychotherapy could be improved by
been tested in numerous studies. tracking variation in use and therapeutic response of patients, contextual
characteristics that influence delivery, and additional interventions that
• Many of these therapies are designed to be delivered by specific affect clinical and functional outcomes.
providers, over a set number of sessions, with a clear step-by-step
approach. • Systems could track how patients respond to varying doses of therapy,
modes of delivery, and clinician characteristics. Over time, decision-makers
• Given variation in access to therapy (e.g., number of sessions covered, could align available care to the needs of patient populations, and
availability of therapists, time), limited predictive ability of response (how clinicians could adapt practice patterns to data on patient preferences
many sessions are needed, what are active ingredients, who should and outcomes, and general needs of the patient population.
deliver therapy?), emergent options for mode of delivery (web-based,
face-to-face, self-guided, asynchronous), optimizing psychotherapy for • By assessing the fit of psychotherapy delivery with patients, the service
individuals and systems is still beyond our current knowledge base. setting and the broader ecological system, the DSF hypothesizes that new
insights about psychotherapy optimization could drive improvements in
patient care.
Care management for chronic diseases

• Studies have shown the effectiveness of care management strategies to • Care management is influenced by drivers at patient, provider,
assess, intervene and monitor for a range of chronic conditions. organization and system levels.

• Typical strategies involve initial screening, assessment, treatment • Care management requires coordination among multiple people,
planning, care and self-management strategies, and follow-up. organizational supports and capital resources, all of which will likely shift
over time.
• While general care management approaches seem to be durable,
specific approaches can have difficulty being implemented across many • Therefore, care management cannot be sustained without continual
clinical and community settings, because of limitations in resources (both assessment of fit within the local setting and the support of the
monetary and staffing), information systems, financing processes, and components of the model.
other barriers.
• The DSF hypothesizes that attention to local adaptations made by
• With new technologies, additional evidence about treatment and healthcare and community settings to fit the model; and feedback on
preventive interventions, and reconfiguration of care systems, care staffing levels, intensity of care management, emerging interventions and
management for chronic diseases can be sustained and improved patient outcomes could enhance long-term sustainability and model
in a large variety of care settings. improvement.

• Evidence about who benefits most from different variants of care

management, who is an ideal care manager, and what are the best ways
of coordinating across primary care and specialty practice could lead to
better uptake and improve patient health.
Note. This table offers examples of how the sustainability of different types of interventions can be enhanced by applying the principles of the DSF. Both the
descriptions of the interventions and the use of the DSF to improve them are exemplars to further the debate, and are not intended to comprehensively depict
the extensive levels of influence.

Summary dynamic process, and provide unparalleled opportunity to

It is time to embrace the culture of a learning healthcare test and refine the principles offered in this paper. With-
system [44,59] to promote sustainability of interventions out this emphasis, we reify the past by asserting the pri-
that are optimized and customized to the myriad of macy of evidence-based guidelines based on evidence
clinical and community settings. The enormous changes from rarified clinical trial settings, and give second-class
in health systems in the past few years give particular status to ongoing learning in real world settings. Allowing
salience to a conceptualization of sustainability as a researchers to be the only ones allowed to generate new
Chambers et al. Implementation Science 2013, 8:117 Page 10 of 11
http://www.implementationscience.com/content/8/1/117

knowledge limits the opportunities to consistently im- 11. Kilbourne AM, Neumann MS, Pincus HA, et al: Implementing
prove the care we provide. In the past, these limitations evidence-based interventions in health care: application of the
replicating effective programs framework. Implement Sci 2007, 2:42.
were technological; we lacked the tools and the sources of 12. Weisz JR, Kazdin AE: Evidence-based psychotherapies for children and
information to drive rapid improvements. In this era of adolescents. New York: Guildford Press; 2010.
‘crowd sourcing,’ of exponentially-expanding processing 13. Kessler R, Glasgow RE: A proposal to speed translation of healthcare
research into practice: dramatic change is needed. Am J Prev Med 2011,
power and global connectedness, we no longer need to 40(6):637–644.
adhere to a view that once created, interventions and 14. Kaitin KI: Deconstructing the drug development process: the new face of
healthcare settings must be ‘frozen’ to optimize effective- innovation. Clin Pharmacol Ther 2010, 87(3):356–361.
15. Craig P, Dieppe P, Macintyre S, et al: Developing and evaluating complex
ness. Instead, we propose the DSF as helping to reconfig- interventions: the new medical research council guidance. BMJ 2008,
ure the research-practice-policy interface, in which the 337:a1655.
best possible information is gathered and used in real- 16. Prochaska JO, Evers KE, Prochaska JM, et al: Efficacy and effectiveness
trials: examples from smoking cessation and bullying prevention.
time to inform policy, improve practice, and answer the J Health Psychol 2007, 112(1):170–178.
highest priority research questions. Only then will the 17. COMMIT Research Group: Community intervention trial for smoking
promise of a learning healthcare system be reached. cessation (COMMIT): summary of design and intervention. J Natl Cancer
Inst 1991, 83:1620–1628.
18. Cohen DJ, Crabtree BF, Etz RS, et al: Fidelity versus flexibility:
Competing interests translating evidence-base research into practice. Am J Prev Med 2008,
REG is a member of the Senior Advisory Board of Implementation Science; 35:S381–S389.
all decisions regarding this paper were made by the editors. The authors
19. Glisson C, Schoenwald SK, Hemmelgarn A, et al: Randomized trial of MST
declare no other competing interests.
and ARC in a two-level evidence-based treatment implementation
strategy. J Consult Clin Psychol 2010, 78(4):537–550.
Authors’ contributions 20. Brownson RC, Colditz GA, Proctor EK: Dissemination and Implementation
DAC conceptualized and wrote the manuscript. REG and KCS provided input Research in Health. New York: OUP; 2012.
on the conceptual framework and edited the manuscript. All authors read 21. Chambers DA: Advancing sustainability research: challenging existing
and approved the final manuscript. paradigms. Public Health Dent 2011, 1(71):S99–100.
22. Shediac-Rizkallah MC, Bone LR: Planning for the sustainability of
Author details community-based health programs: conceptual frameworks and future
1
Division of Services and Intervention Research, National Institute of Mental directions for research, practice and policy. Health Educ Res 1998,
Health, 6001 Executive Blvd, Rockville, MD, USA. 2Department of Family 13(1):87–108.
Medicine, University of Colorado School of Medicine, Denver, CO, USA. 23. Scheirer MA: Is sustainability possible? A review and commentary on
3
Department of Family Medicine, Case Western Reserve University, Cleveland, empirical studies of program sustainability. Am J Eval 2005,
OH, USA. 26(3):320–347.
24. Scheirer MA, Dearing JW: An agenda for research on the sustainability of
Received: 2 April 2013 Accepted: 20 September 2013 public health programs. Am J Public Health 2011, 101(11):2059–2067.
Published: 2 October 2013 25. DiMaggio PJ, Powell WW: The iron cage revisited: institutional
isomorphism and collective rationality in organizational fields. Am Sociol
References Rev 1983, 48(2):147–160.
1. Greenhalgh T, Robert G, Macfarlane F, et al: Diffusion of innovations in 26. Ory MG, Lee SM, Mier N, et al: The science of sustaining health behavior
service organizations: systematic review and recommendations. change: the health maintenance consortium. Am J Health Behav 2010,
Milbank Q 2004, 82(4):581–629. 34(6):647–659.
2. Glasgow RE, Chambers D, Developing Robust, Sustainable: Implementation 27. Swerissen H, Crisp BC: The sustainability of health promotion
systems using rigorous, rapid, and relevant science. Clin Transl Sci 2012, interventions for different levels of social organization. Health Promot
5(1):48–55. Interv 2004, 19(1):123–130.
3. Haines A, Kuruvilla S, Borchert M: Bridging the implementation gap 28. Greenhalgh J, Meadows K: The effectiveness of the use of patient-based
between knowledge and action for health. Bull World Health Organ 2004, measures of health in routine practice in improving the process and
82(10):724–732. outcomes of patient care: a literature review. J Eval Clin Pract 1999,
4. Flay BR: Efficacy and effectiveness trials (and other phases of research) in 5(4):401–416.
the development of health promotion programs. Prev Med 1986, 29. Duckers ML, Wagner C, Vos L, et al: Understanding organisational
15:451–474. development, sustainability, and diffusion of innovations within
5. Greenwald P, Cullen JW: The new emphasis in cancer control. J Natl hospitals participating in a multilevel quality collaborative.
Cancer Inst 1985, 74:543–551. Implementation science 2011, 6(1):18.
6. Stirman SW, Kimberly J, Cook N, et al: The sustainability of new programs 30. Glasgow RE, Green LW, Taylor M, et al: An Evidence integration triangle for
and innovations: a review of the empirical literature and aligning science with policy and practice. Am J Prev Med 2012, 42(6):646–654.
recommendations for future research. Implement Sci 2012, 7:17. 31. Wandersman A, Imm P, Chinman M, et al: Getting to Outcomes: Methods and
7. Rounsaville BJ, Carroll KM, Onken LS: A stage model of behavioral tools for planning, evaluation and accountability. Rockville, MD: Center for
therapies research: getting started and moving on from stage 1. Substance Abuse Prevention; 1999.
Clin Psychol 2001, 8(2):133–142. 32. Estabrooks PA, Smith-Ray RL, Dzewaltoski DA, et al: Sustainability of
8. Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C: evidence-based community-based physical activity programs for older
National institutes of health approaches to dissemination and adults: lessons from Active for Life. Transl Behav Med 2011, 1(2):1–8.
implementation science: current and future directions. Am J Public Health 33. Wilson KD, Kurz RS: Bridging implementation and institutionalization
2012, 102(7):1274–1281. within organizations: proposed employment of continuous quality
9. Rabin BA, Brownson RC, Haire J, Kreuter MW, Weaver NL: A glossary for improvement to further dissemination. J Public Health Manag Pract 2008,
dissemination and implementation research in health. Public Health 14(2):109–116.
Manag Pract 2008, 14(2):117–123. 34. McLaughlin CP, Kaluzny AD: Continuous Quality Improvement in Health Care.
10. Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of Sudbury, MA: Jones and Bartlett; 2004.
evidence-based practice implementation in public service sectors. 35. Homer JB, Hirsch GB: Systems dynamics modeling for public health:
Adm Policy Ment Health 2011, 38:4–23. background and opportunities. Am J Public Health 2006, 96(3):452–458.
Chambers et al. Implementation Science 2013, 8:117 Page 11 of 11
http://www.implementationscience.com/content/8/1/117

36. Plsek PE, Greenhalgh T: The challenge of complexity in health care.

BMJ 2001, 323:625–628.
37. Ebi K: Climate change and health risks: assessing and responding to
them through ‘adaptive management. Health Aff 2011, 30(5):924–30.
38. Bushe GR, Marshak RJ: Revisioning organizational development:
diagnostic and dialogic premises and patterns of practice. J Appl Behav
Sci 2009, 45(3):348–368.
39. Checkland P: Soft systems methodology: a thirty year retrospective.
Syst Res 2000, 17:S11–S58.
40. Stange KC, Breslau ES, Dietrich AJ, et al: State-of-the-art and future
directions in multilevel interventions across the cancer control
continuum. J Natl Cancer Inst Monogr 2012, 44:20–31.
41. Green LW: Making research relevant: if it is an evidence-based
practice, where’s the practice-based evidence? Fam Pract 2008,
25(Supp 1):i20–i24.
42. Bughlin J, Chui M, Johnson B: The next step in open innovation,
The McKinsey Quarterly; 2012. http://www.mckinseyquarterly.com/
next_step_in_open_innovation_2155. Accessed.
43. Khoury MJ, Gwinn M, Glasgow RE, et al: Am J Prev Med 2012, 42(6):639–45.
44. Etheredge LM: A rapid-learning health system: what would a
rapid-learning health system look like, and how might we get there.
Health Aff 2007. Web Exclusive Collection(26):w107-w118 (Published ahead
of print January 26, 2007, doi:10.1377/hlthaff.26.2.w107.
45. Meissner HI, Vernon SW, Rimer BK, Wilson KM, Rakowski W, Briss PA, et al:
The future of research that promotes cancer screening. Cancer 2004,
101(5 Suppl):1251–9.
46. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC:
Fostering implementation of health services research findings into
practice: a consolidated framework for advancing implementation
science. Implementation Science 2009, 4:50.
47. Glasgow RE, Kaplan R, Ockene J, Society of Behavioral Medicine Health
Policy Committee, et al: Patient-reported measures of psychosocial issues
and health behavior should be added to electronic health records.
Health Affairs 2012, 31(3):497–504.
48. Tanenbaum S: Evidence-based practice as mental health policy: three
controversies and a caveat. Health Aff 2005, 24(1):163–173.
49. Rabin BA, Purcell P, Naveed S, et al: Advancing the application, quality
and harmonization of implementation science measures. Implement Sci
2012, 7:119.
50. Estabrooks P, Glasgow RE, Dzewaltowski DA: Physical activity promotion
through primary care. JAMA 2003, 289(22):2913–2916.
51. Rogers EM: Diffusion of Innovations (4th Edition). New York: Free Press; 1995.
52. Senge PM, Kleiner A, Roberts C, et al: The dance of change: The challenges of
sustaining momentum in learning organizations. 1st edition. New York:
Currency/Doubleday; 1999.
53. Lorig KR, Holman HR: Self-management education: history, definition,
outcomes, and mechanisms. Ann Behav Med 2003, 26:1–7.
54. Institute of Medicine: A foundation for evidence-driven practice: A rapid
learning system for cancer care. Workshop Summary. Washington, D.C: NAS
Press; 2010.
55. Wagner EH, Austin BT, Von Korff M: Organizing care for patients with
chronic illness. Milbank Q 1996, 74:511–544.
56. Unutzer J, Powers D, Katon W: From establishing an evidence-based
practice to implementation in real-world settings: IMPACT as a case
study. Psychiatr Clin North Am 2005, 28(4):1079–1092.
57. Detmer D, Bloomrosen M, Raymond B, et al: Integrated personal health
records: Transformative tools for consumer-centric care. BMC Med Inform
Decis Mak 2008, 8(45):1–14.
58. PatientsLikeMe: PatientsLikeMe website; 2012. http://patientslikeme.com. Submit your next manuscript to BioMed Central
Accessed March 15, 2012. and take full advantage of:
59. Institute of Medicine: Best Care at Lower Cost: The Path to Continuously
Learning Health Care in America. Consensus Report. Washington, D.C: NAS • Convenient online submission
Press; 2012.
• Thorough peer review

doi:10.1186/1748-5908-8-117 • No space constraints or color figure charges

Cite this article as: Chambers et al.: The dynamic sustainability • Immediate publication on acceptance
framework: addressing the paradox of sustainment amid ongoing change.
Implementation Science 2013 8:117. • Inclusion in PubMed, CAS, Scopus and Google Scholar
• Research which is freely available for redistribution

Submit your manuscript at

www.biomedcentral.com/submit