APRIL 2012

Orthogonal Design: A Powerful Method for Comparative

Effectiveness Research with Multiple Interventions
by Jelena Zurovac and Randy Brown

There is a growing interest in new or modified research methods that can balance the needs for quick
results and statistical rigor in studies of intervention effectiveness. Orthogonal design, which has been
extensively used in manufacturing and other fields but not in published health care evaluations or comparative
effectiveness research (CER), affords an opportunity to design interventions in real-world settings and to
study intervention components that can be implemented in various ways. This brief introduces orthogonal
design, describes key design and implementation considerations, and illustrates how it can be applied in
CER studies by describing its use in an ongoing study of care coordination delivery by Special Needs Plans
(SNPs) to dually eligible individuals.

The Need for New Methods Experience shows that, even after tions depends critically on the features
traditional evaluation methods (such of the intervention, the methods used
The aim of comparative effectiveness as randomized trials) find a given to engage patients and providers, and
research (CER) is to assess the relative intervention effective, the programs the target population for the interven-
ability of two or more interventions to often do not produce comparably tion components.
influence a desired outcome, and to favorable results when the intervention
evaluate how their effectiveness differs is repeated in other settings (due in part
across patients and circumstances. The to differences in how the intervention
WHY IS ORTHOGONAL DESIGN
WELL-SUITED FOR CER?
ongoing focus on CER, and the desire was implemented). In the last decade,
to obtain findings more quickly than several models of transitional care have • It can identify unplanned
typically possible with randomized been shown to be effective in reducing variations occurring in real
controlled trials, have created an interest readmission rates (Naylor et al. 1999; world practice and test them to
in new or modified methods. Of particular Coleman et al. 2006; Jack et al. 2009). determine which is best.
interest are new methods amenable to However, less is known about how • It combines the rigor of experi-
multi-component interventions that best to implement the various compo- mental design with the ability to
can be implemented in various ways. nents. For example, evidence shows produce rapid results by testing
Learning how best to implement that post-discharge follow-up helps multiple components at once.
intervention components in a real- reduce readmission rates, but there is • It can be used to assess whether
world setting is important because little information about how quickly more resource-intensive ways of
existing evidence often provides no this follow-up visit needs to occur, implementing interventions yield
guidance on whether more intensive how many times, and which protocols sufficiently better patient out-
or more expensive ways to implement to use. Mahoney (2010) notes that the comes to warrant the investment.
interventions yield significantly success of multi-component interven-
better outcomes.

The Center on Health Care Effectiveness (CHCE) was founded in 2010 to inform health care policy and
improve patient outcomes. Building on more than four decades of Mathematica’s substantive expertise,
methodological skill, and corporate infrastructure, CHCE is at the forefront of comparative effectiveness
policy analysis and research.
 ORTHOGONAL DESIGN b, for each intervention. Half of the of their interactions) with far fewer
OVER THE YEARS units assigned to option a for inter- experimental units than it would take to
Sir Ronald Fisher developed orthogo- vention 1 are assigned to option a for exhaust all possible intervention
nal design, described in his seminal intervention 2 and half to option b, and combinations. This feature makes it
book Design of Experiments (1935), so on for all possible intervention pair- particularly valuable for testing the best
based on agricultural experiments in ings. However, in efficient orthogonal way to implement complex interventions
England. Over the years, this design designs, any single intervention is with many facets. For example, it can
has been used widely in the chemical confounded with two-way or higher be applied in testing ways to implement
and automotive industries, natural order interactions between other the numerous components or activities
science research, business and interventions. The degree of such involved in care management of
marketing, and operations research. confounding depends on the number of complex patients. Variations across units
Recently, health plans have used experimental units used relative to the in how interventions are implemented
orthogonal design for internal quality number of interventions being tested. will occur regardless of whether the
improvement initiatives, although variants are tested; explicit testing as
these studies are rarely in the public The effect of an individual intervention
part of the experimental design allows
domain (Moore 1994; Jones and is calculated by comparing the mean out-
the program operator to learn which
Moore 1995). come over all subjects for experimental
variant is best for each intervention.
units that provide one variant (a) to the
mean for subjects of those who provide This design yields useful results even if
the other (b). As in any other design, one the difference between the two variants
Orthogonal Design
can use regression analysis with subject- of any intervention is not statistically
Orthogonal design is an experimental level data to compute intervention significant. Often, one of the two tested
design used to test the comparative effects, controlling for any differences variants is more difficult, burdensome,
effectiveness of multiple intervention between units with respect to the charac- or expensive to implement; thus, a find-
components—referred to here as “inter- teristics of their subjects. The regression ing of no significant difference means
ventions”—each of which takes on two can also control for potentially important that the more burdensome variant does
or more variants. In a two-level design, characteristics of the experimental units, not produce better results than the less
the researcher specifies two variants such as the baseline value of the outcome burdensome variant.
(referred to here as options a and b) variable for the experimental units, if the
for each intervention—such as which number of units is substantially larger Design and Implementation
of two protocols are followed or which than the number of interventions being
of two levels of intensity are used. An tested. Box (2005) provides a compre- The most important step in designing
algorithm is then used to generate a hensive discussion of methods. a successful orthogonal design study
specific set of combinations of a’s and that yields credible, actionable results,
The critical advantage of orthogonal is to identify intervention components
b’s that constitute an orthogonal design
design relative to typical controlled and alternative options that are feasible
for the number of interventions to be
trials is that it allows the researcher to implement and can improve patient
tested. The researcher then randomly
to test the effectiveness of many outcomes and/or operational costs. The
assigns (without replacement) one of
interventions simultaneously in a single interventions should also be ones that
these predetermined combinations of
experiment (and possibly identify some the program implementer wishes to test.
variants to each experimental unit (for
example, nurses or classrooms), who
E X A M P L E O F A N O R T H O G O N A L D E S I G N M AT R I X
will then administer this combination
of variants to all of their subjects (such Experimental Intervention
Unit 1 2 3 4 5 6 7 8 9 10 11
as patients or students). The matrix on
this page shows a design in which 12 1 a a b a a a b b b a b
experimental units are each assigned to 2 b a a b a a a b b b a
implement a specific variant for each 3 a b a a b a a a b b b
of the 11 interventions to be tested. The 4 b a b a a b a a a b b
set of combinations ensure “orthogonal- 5 b b a b a a b a a a b
ity,” which refers to a property of the 6 b b b a b a a b a a a
assignments that ensures that the effect 7 a b b b a b a a b a a
of any one intervention is unconfounded 8 a a b b b a b a a b a
with the effects of any other single 9 a a a b b b a b a a b
intervention. These combinations ensure 10 b a a a b b b a b a a
that half of the experimental units are 11 a b a a a b b b a b a
assigned to option a, and half to option 12 b b b b b b b b b b b

2
Further, when designing interventions a sufficient number of experimental much confounding to tolerate and which
or choosing components, it is important units are available to obtain unconfounded interaction effects to estimate, the
to consider: main effects. (For example, the matrix limitation of designs that use few
shown on p. 2 requires at least 12 experimental units relative to the number
• The likelihood that the experimental experimental units to estimate main of interventions tested (such as the matrix
units will faithfully implement the effects of 11 interventions.) If more than on p. 2) is that the effect of any one
component variants to which they are the minimum number of experimental intervention is confounded with many
assigned without excessive oversight units is available to test the desired two-way interactions. Doubling the
beyond what would be expected to interventions, the researcher must number of experimental units in this
occur in an ongoing program. decide how to use the additional design to 24 would ensure that each of
• The likelihood that variants shown to experimental units. The two options the 11 intervention effects being tested
be effective will actually be adopted are to (1) test more interventions or in this “foldover” design is confounded
as tested. (2) improve “resolution,” that is, to only with three-way and higher order
reduce the extent of confounding of interactions, which are usually expected
• The number of interventions to test.
main effects with potentially important to be negligible. The larger number
Orthogonal design studies face imple- second-order interactions of other of experimental units also increases
mentation considerations similar to interventions. In other words, the study the statistical power of the tests for
those of other studies, but resistance can test more interventions with greater all interventions.
from participants might be stronger. confounding of main effects with
People who implement the interventions second-order interactions, or test fewer Opportunities and
often feel initially that testing many interventions with less confounding. If Recommendations
interventions is “too complicated.” the number of experimental units is not
Engaging these experimental agents in fixed, the researcher must perform Orthogonal design is potentially a very
selecting the intervention variants to power calculations to determine how useful tool for rapid-cycle compara-
be tested and using tools such as many experimental units are needed for tive effectiveness research. It provides
implementation guides and individualized the desired precision before finalizing an opportunity to test—rigorously and
assignment sheets can ease concerns the number of interventions. Because simultaneously—many aspects of deliv-
about fidelity and help the researcher to this is a clustered design in which ering multifaceted interventions. This
better understand whether (and how) the groups of research subjects all receive advantage is relevant because nearly all
interventions might be implemented on the same set of interventions, power policy research today focuses on whether
an ongoing basis. Requiring the agents depends almost entirely on the number a broad concept is effective in improving
to record the provision of interventions of experimental units. outcomes, without studying the opera-
can yield important data for measuring tional details of how the interventions
fidelity to interventions and interpreting are provided. The experimental agents’
Limitations and Caveats
findings. In addition, researchers should experience and understanding of current
assess facilitators and barriers to Two key assumptions embedded in processes of care is invaluable in identify-
implementation to understand whether orthogonal design are that the experi- ing the intervention components for which
the interventions were actually mental units (agents) are homogenous, there is the most uncertainty about the
implemented and the factors that are and that all third or higher order relative effectiveness of alternative ways
necessary for effective implementation. intervention interactions can be to implement the component. Further-
Researchers should also keep in mind considered to be negligible. Homogeneity more, variations across health care work-
that the health care environment differs simply means that the agents would ers in how interventions are implemented
from other disciplines in which have roughly similar outcomes if will occur in any case; by formalizing and
orthogonal design has historically been implementing the same interventions, testing the variations, program operators
used (such as manufacturing), which which can be assessed by comparing can learn which ones yield the greatest
generally have more controlled environ- average outcomes prior to the study for improvements in patient outcomes. There-
ments. The analyses should always take each agent. Outliers that have a high fore, to harness the most value from this
the “intent-to-treat” approach, in which likelihood of distorting the comparison powerful design, studies should focus on
effects of interventions are computed by of means can then be excluded from the developing interventions in collaboration
comparing outcomes of those assigned study. The degree to which an orthogonal with those who would be implementing
to two variants of a given intervention, design study is subject to confounding them, such as the care coordinators in
regardless of whether or how thoroughly depends primarily on the number of our study. The flexibility of orthogonal
the variants were actually delivered. interventions and experimental units designs allows researchers to choose
and somewhat on the selected design. whether to screen many interventions at
When deciding how many interventions
Although the researcher can choose how once, or to test a smaller number with
to test, researchers must first ensure that
greater statistical precision.
3
 A P P LI C ATI ON: S T U D Y OF CA RE C OORD I NAT I ON D ELI VERY References
Box, George E. P., Stuart J. Hunter, and
Special Needs Plans (SNPs) were established in 2003 as part of the Medicare
William G. Hunter. “Statistics for Experi-
Prescription Drug, Improvement, and Modernization Act, with the goal of menters: Design, Innovation, and Discovery.
improving care for three high-risk target populations of Medicare beneficiaries: 2nd. Sl.” 2005.
(1) dual eligibles (those enrolled in both Medicare and Medicaid),
Coleman, Eric. A., Carla Parry, Sandra
(2) beneficiaries with chronic conditions, and (3) beneficiaries residing in
Chalmers, and Sung-Joon Min. “The Care
nursing homes. SNPs contract with CMS to provide all covered Medicare Transitions Intervention: Results of a
services in return for a monthly risk-adjusted capitation payment. Thus, SNPs Randomized Controlled Trial.” Archives of
have the incentive—and the requirement, under their contract with CMS—to Internal Medicine, vol. 166, no. 17, 2006,
engage in care management to help patients reduce their need for expensive pp. 1822–1828.
services, especially hospitalizations and emergency room visits. Although Fisher, Ronald A. “The Design of Experi-
many protocols, interventions, and screening tools exist, SNPs still have many ments.” Edinburgh: Oliver and Boyd, 1935.
questions about the most efficacious and cost-effective ways to design and
Jack, Brian W, Veerappa K. Chetty, David
implement interventions to address the unique and varied needs of the high-
Anthony, Jeffrey L. Greenwald, Gail
risk populations they serve. M. Sanchez, Anna E. Johnson, et al. “A
In collaboration with three SNPs serving dual eligibles, we designed and are Reengineered Hospital Discharge Program
to Decrease Rehospitalization.” Annals of
conducting an orthogonal design study to help SNPs identify care coordination
Internal Medicine, vol. 150, no. 3, 2009,
strategies that work best for their members. The main outcomes we will analyze pp. 178–87.
are hospital admissions, readmissions, and emergency room visits. One part of
the study involves 25 care coordinators who are implementing either current Jones, Frederic G., and Colletta H. Moore.
“Designing and Executing Experiments in
practice or an enhanced practice for each of 11 interventions over a 12-month
Care—A Data-Driven, Scientific Approach
period. The studied intervention variants address how often an intervention to Quality Improvement.” In Measuring
is provided, or which procedures or protocols are used in conducting routine Clinical Care: A Guide for Physician
contacts with patients, screening for the risk of falls, depression screening, care Executives, edited by S. C. Schoenbaum.
planning, patient coaching, and management of care transitions. For example, Providence, RI: American College of
the study is testing a care transitions intervention that compares the current Physician Executives, 1995.
practice of conducting one follow-up visit with patients within three business Mahoney, Jane E. “Why Multifactorial Fall-
days of discharge from an inpatient setting versus an enhanced practice that Prevention Interventions May Not Work:
includes a second follow-up visit within a week of the first one. Another care Comment on “Multifactorial Intervention to
transitions intervention being tested compares the current practice to a more Reduce Falls in Older People at High Risk
structured follow-up process that uses a checklist and an instrument to assess of Recurrent Falls.” Archives of Internal
Medicine, vol. 170, no. 13, 2010, p. 1117.
the patient’s understanding of post-discharge instructions. One of the depression
screening interventions tests effectiveness of a longer screening instrument Moore, Colletta H. “Experimental Design
versus a shorter one and the other tests variants in the minimum frequency of in Health Care.” Quality Management in
routine screening contacts. The details of the interventions were developed Healthcare, vol. 2, no. 2, 1994, p. 13.
in collaboration with the experimental agents (the care coordinators) and Naylor, Mary D., Dorothy Brooten, Roberta
management of the participating plans. Campbell, Barbara S. Jacobsen, Mathy D.
Mezey, Mark V. Pauly, and J. S. Schwartz.
“Comprehensive Discharge Planning and
Home Follow-Up of Hospitalized Elders:
A Randomized Clinical Trial.” JAMA, vol.
281, no. 7, 1999, pp. 613-620.

For more information, please contact

Jelena Zurovac at (609) 275-2383,
[email protected], or
Randy Brown at (609) 275-2393,
[email protected].

Princeton, NJ • Ann Arbor, MI • Cambridge, MA • Chicago, IL • Oakland, CA • Washington, DC

Visit our website at www.mathematica-mpr.com Mathematica® is a registered trademark of Mathematica Policy Research, Inc.