Volume 11 • Number 5 • 2008

VA L U E I N H E A LT H

Optimal Cost-Effectiveness Decisions:The Role of the

Cost-Effectiveness Acceptability Curve (CEAC), the
Cost-Effectiveness Acceptability Frontier (CEAF), and
the Expected Value of Perfection Information (EVPI)

Garry R. Barton, PhD,1 Andrew H. Briggs, PhD,2 Elisabeth A. L. Fenwick, PhD2

1
School of Medicine, Health Policy and Practice, University of East Anglia, Norwich, UK; 2Public Health & Health Policy, University of
Glasgow, Glasgow, UK

A B S T R AC T

Objective: To demonstrate how the optimal decision and option to never have the highest probability of being
level of uncertainty associated with that decision, can be cost-effective; and 3) the EVPI to increase when the probabil-
presented when assessing the cost-effectiveness of multiple ity of making the wrong decision decreases. Changing the
options. To explore and explain potentially counterintuitive correlation structure between multiple options did not
results that can arise when analyzing multiple options. change the presentation of results on the cost-effectiveness
Methods: A template was created, based on the assumption plane.
of multivariate normality, in order to replicate a previous Conclusion: The cost-effectiveness plane has limited use in
analysis that compared the cost-effectiveness of multiple representing the uncertainty surrounding multiple options as
options. We used this template to explain some of the differ- it cannot represent correlation between the options. CEACs
ent shapes that the cost-effectiveness acceptability curve can represent decision uncertainty, but should not be used to
(CEAC), cost-effectiveness acceptability frontier (CEAF), and determine the optimal decision. Instead, the CEAF shows the
expected value of perfection information (EVPI) may take, decision uncertainty surrounding the optimal choice and this
with changing correlation structure and variance between the can be augmented by the EVPI to show the potential gains to
multiple options. further research.
Results: We show that it is possible for 1) an option that Keywords: cost-effectiveness acceptability curve (CEAC),
is subject to extended dominance to have the highest pro- cost-effectiveness acceptability frontier (CEAF), cost-
bability of being cost-effective for some values of the cost- effectiveness plane, expected value of perfect information
effectiveness threshold; 2) the most cost-effective (optimal) (EVPI), uncertainty.

Introduction when purportedly constructing the CEAF, some ana-

lysts [8,9] have plotted the probability of cost-
The cost-effectiveness acceptability curve (CEAC) was
effectiveness for many options, rather than just the
introduced as a method to represent uncertainty, and is
optimal option, and used the term CEAF to describe this
often used to present the results of cost-effectiveness
mode of presentation. In light of these issues, we seek to
analysis [1–4]. However, it is possible that the CEAC
demonstrate how the CEAC and CEAF should be con-
may be misinterpreted [5,6]. This possibility is high-
structed, what they should and should not be used for,
lighted by the fact that one might (incorrectly) infer,
and in doing so, how the optimal decision can be
from one paper [7], that CEACs show the value at
presented, along with levels of uncertainty.
which each option becomes the optimal choice, and
In order to illustrate the above, we present (and
from another paper [8], that they show the probability
make available online) a spreadsheet-based template
that one option dominates another option. There is
that can be used to represent uncertainty in the deci-
a similar possibility that the purpose of the cost-
sion among a number of mutually exclusive treatment
effectiveness acceptability frontier (CEAF) may also be
options. We then employ this template to explain two
misinterpreted. This is highlighted by the fact that,
entirely possible results that may nonetheless appear
counterintuitive. First, that an option that was subject
Address correspondence to: Garry R. Barton, Health Economics to extended dominance [10,11] can have the highest
Group, School of Medicine, Health Policy and Practice, Univer-
sity of East Anglia, Norwich, NR4 7TJ, UK. E-mail:
probability of being cost-effective. Second, that the
[email protected] optimal option (as it provided the greatest benefit for
10.1111/j.1524-4733.2008.00358.x a given cost) may not have the highest probability

886 © 2008, International Society for Pharmacoeconomics and Outcomes Research (ISPOR) 1098-3015/08/886 886–897
Optimal Decisions: The Role of the CEAC, CEAF, and EVPI 887

of being cost-effective for any value of the cost- highest level of expected benefit at different levels of
effectiveness threshold. Both of these situations were cost, it has been argued that the plane fails to fully
apparent in the results presented by Goeree et al. [1], represent the level of uncertainty associated with the
and this study serves as a practical example to show estimated cost-effectiveness of an option in a graphical
why such results are in fact valid, and how decision manner [21]. As a consequence of this, the CEAC was
uncertainty in such situations should be presented and developed. The CEAC is constructed by plotting (for
interpreted. each option) the proportion of the cost and effect pairs
The paper is structured as follows: First, we provide that are cost-effective for a range of values of the
an overview of the main principles of cost-effectiveness cost-effectiveness threshold [22]. Thus, uncertainty is
analyses and illustrate the importance of the CEAC, characterized by estimating the probability that an
CEAF, and expected value of perfect information option is cost-effective at different levels of the cost-
(EVPI). Second, we outline a template that can be used effectiveness threshold, where this is undertaken for all
by others and is used here to demonstrate how to possible options. Fenwick et al. [17], however, have
calculate CEACs, the CEAF, and the EVPI in the pres- shown that an option that had the highest probability
ence of multiple options. Finally, this template is used of being cost-effective need not necessarily have the
to illustrate how different assumptions about the highest expected net benefit, and for this reason, the
correlation structure and the variances of the model CEAC should not be used to identify the optimal treat-
outputs can influence the shape of the resulting ment option. Instead, they propose the use of the
CEACs, CEAF, and EVPI, thus demonstrating why the CEAF [17], which plots the uncertainty associated
two potentially counterintuitive results occurred in the with the optimal option, for different values of the
study by Goeree et al. [1]. cost-effectiveness threshold. This is equivalent to plot-
ting each CEAC over the range of values for the cost-
Principles of Cost-Effectiveness Analyses effectiveness threshold for which each option is
When making decisions about the allocation of scarce estimated to be the most cost-effective, i.e., has the
health-care resources, it has been argued that two highest ICER that falls below the threshold [17].
questions are fundamental [12–14]. First, which Finally, in order to inform the second fundamental
option is estimated to be cost-effective, on the basis of question concerning the decision of whether to under-
existing evidence? Second, should further research be take further research, the EVPI can be calculated. The
undertaken in order to reduce the level of uncertainty EVPI gives an upper bound on the value of undertak-
associated with that decision? ing further research in order to eliminate the uncer-
In order to answer the first question, one seeks to tainty surrounding the decision about which option is
identify the option that is expected to provide the optimal for different levels of the cost-effectiveness
highest level of benefit for a given level of cost, i.e., threshold [20,23]. The EVPI is dependent upon both
the aim is to maximize health subject to a budget the probability of a wrong decision being made (as
constraint [15–17]. In the case of mutually exclusive shown by the CEAF), but also the consequences of that
options, this involves the calculation of the incremen- wrong decision [23,24]. Further research costing less
tal cost-effectiveness ratio (ICER) of each option rela- than the EVPI only has the potential to be efficient;
tive to the next best treatment option, and the selection a sufficient condition for determining the worth of
of the option that has the highest ICER below the further research requires comparison of the expected
cost-effectiveness threshold as the optimal option. value of sample information and the costs (see [23] and
When calculating the ICER, it is important to identify, [25] for more details).
and exclude, those options that are dominated (more
costly and less effective than another option) and those
Methods
that are subject to extended dominance (combinations
of other options can provide a higher level of benefit Probabilistic methods can be used to describe the uncer-
for the same cost) [18]. tainty associated with input parameters in a model, and
One of the most common ways of presenting the in turn, can estimate the uncertainty in the model
results of cost-effectiveness analysis is on the cost- outputs of cost, effect, and cost-effectiveness [20]. This
effectiveness plane [19], where the effects are measured equates to a Bayesian approach to cost-effectiveness
on the horizontal axis and costs are measured on the analysis because the parameters of interest are ascribed
vertical axis. By plotting the mean cost and mean effect a distribution in order to reflect the uncertainty con-
of each option and connecting each cost-effective (i.e., cerning the true value of the parameter [26]. For the
nondominated) option with the next less effective purposes of exploring potential scenarios relating to the
option, it is possible to display the efficiency frontier costs and effects of multiple options, we developed a
for these options [1,20]. Thus, though the cost- template (in Excel, Microsoft, Redmond, WA) that
effectiveness plane can answer the first fundamental approximates the joint distributions of the costs and
question, by identifying the option that provides the effects of multiple treatment options under the assump-
888 Barton et al.

tion of multivariate normality. As such, we seek to In contrast to the CEACs, the CEAF plots only the
provide a generic characterization of the uncertainty probability that the optimal option is cost-effective (at
associated with model outputs for the purposes of different l-values). Thus, it is first necessary to identify
illustration and ease of manipulation. Readers are which option is optimal at each level of the cost-
directed to Briggs et al. [20] for further direction on effectiveness threshold. To do this, it is necessary to
how to characterize the uncertainty of input parameters identify the mean cost and mean effect for each option
using probability distributions. The template, which (across each of the 1000 simulations), and to calculate
can be downloaded at http://www.dph.gla.ac.uk/heat, which option is optimal (has the highest expected net
works with up to seven different options and requires benefit) at different levels of l. Thus, the range of
the user to input the means and standard errors for all l-values over which an option is optimal can be cal-
options. In addition, the user also specifies the correla- culated, and the “switch points” (when there is a
tion structure between the costs and effects both within change in the optimal option) correspond to the ICER
and between options. From this information, the between different options [17]. The lower value of the
variance-covariance matrix is generated and (through range of values for the cost-effectiveness threshold for
the use of the Cholesky decomposition technique [20]) which each option was optimal denotes the ICER for
the template draws 1000 simulations of both the cost that particular option, and the upper value denotes the
and effect parameters of each option from the specified ICER for the next most costly option.
multivariate normal distribution. Finally, using the Once the optimal option has been identified for
methods described later, the template provides esti- each level of l, the probability of this option being
mates of the CEACs, CEAF and the EVPI. cost-effective can then be plotted (y-axis) for different
l-values (x-axis). This probability of being cost-
CEAC effective is determined by the CEAC. One minus the
In the case of multiple options, a separate CEAC can probability given by the CEAF at any value of the
be plotted for each option, where each CEAC repre- cost-effectiveness threshold is equivalent to the prob-
sents the (essentially Bayesian) probability of each ability that a “wrong” decision will be made (the error
option being cost-effective at different levels of the probability) [13]. Finally, it should be noted that
cost-effectiveness threshold [20,27]. This probability because only optimal options are represented, the
was estimated in the following manner: In each of the CEAF may not always correspond to the options with
1000 iterations, the total cost (C) and total effect (E) the highest probability of being cost-effective.
was estimated for each option, and for a particular
cost-effectiveness threshold (l), the net monetary EVPI
benefit (NMB) was estimated: NMB = l ¥ E - C. In The EVPI (for an individual patient) is determined, for
each of the 1000 iterations, the option with the highest a given l, by the difference between the expected value
net benefit was then identified. The probability of with perfect information and the expected value with
being cost-effective was then equivalent to the propor- current information. After calculating the expected net
tion of the 1000 iterations for which each option had benefit for each option across all iterations, the first
the highest net benefit. CEACs were estimated by plot- step is to identify the optimal option based on the
ting these proportions (y-axis) for different l-values current level of information. This is the option with the
(x-axis). With regard to the claim that the CEAC maximum expected net benefit for the given level of
shows the probability that one option dominates the cost-effectiveness threshold, as identified by the
another option [8], it can be seen that this is untrue CEAF. The second step is to identify the decision with
because the CEAC is determined by the net benefit of perfect information. This is done for every possible
each option and it is possible for an option to have a realization of uncertainty (every iteration) by identify-
higher net benefit without dominating another option. ing the option that maximizes net benefit. Within each
iteration, the value of perfect information is equal to
CEAF the difference between the net benefit of the optimal
Fenwick et al. [17] have shown that the probability of option (from step one) and the maximum net benefit
being cost-effective cannot be used to determine the (from step two), and will equal zero when the optimal
optimal option, and that if the societal objective is to option has the highest net benefit. Finally, the EVPI is
maximize health gain, then decisions should be taken calculated by taking the expectation over the values of
on the basis of expected net benefit, regardless of the perfect information. This is equivalent to the value of
uncertainty (probability) associated with the decision. the decision when made with perfect information aver-
Thus, the option with the highest probability of being aged across all possible realizations of uncertainty
cost-effective (highest CEAC) at any value of the cost- [14]. The EVPI was then calculated for differing
effectiveness threshold need not be the optimal option l-values and plotted on the same diagram as the
[28]. The solution to the fact that CEACs cannot iden- CEAF; it shows the maximum amount that one would
tify the optimal option is to plot the CEAF [17]. be willing to pay to eliminate uncertainty.
Optimal Decisions: The Role of the CEAC, CEAF, and EVPI 889

Replication of Previous Analyses compared with F)—the efficiency frontier is presented

By replicating the comparison of multiple options by for the mean costs and mean effects, rather than for each
Goeree et al. [1], we sought to depict and explain both of the 1000 simulations.
the optimal decision and the level of uncertainty asso- The corresponding CEACs are also plotted in
ciated with that decision. Goeree et al. [1] estimated Figure 1b, where it can be seen that one of the optimal
the cost-effectiveness of seven different management options (A) never had the highest probability of being
strategies for gastroesophageal reflux disease, where cost-effective. Instead, B initially had the highest prob-
costs were estimated at year 2000 levels (US$), and ability of being cost-effective (l < $7970), followed by
effects were measured in terms of quality adjusted life D ($7970 = l < $12,540), C ($12,540 = l < $14,850),
years (QALYs). In order to replicate the results, for F ($14,850 = l < $112,120), and E (l = $112,120).
each of the seven options, we extracted the mean cost Thus, options that were subject to extended domi-
(and associated standard error) and mean effect (and nance (C and D) were also associated with the highest
associated standard error) from the results of their probability of being cost-effective at certain levels of ë.
probabilistic analysis (which were reported for a This provides an illustration of why CEACs should not
cohort of 1000 patients). From the results provided by and cannot be used to identify the optimal option, as
the original authors, the correlation matrix was deter- has been previously shown by Fenwick et al. [17].
mined and employed to construct the variance– In order to simultaneously present the optimal
covariance matrix for the analysis. Initially, we option and the level of uncertainty associated with that
undertook a replication of the analysis undertaken by option, the CEAF was plotted (Fig. 1c). The ICERs for
Goeree et al. [1] (the base-case analysis). By altering options A, F, and E are plotted as vertical lines in order
the assumptions underlying this analysis, we then ran a to demonstrate how the optimal option was identified.
series of sensitivity analyses in order to explain why Prior to the first vertical line (l < $7755), option B
the two potentially counterintuitive results occurred was optimal, between the first two vertical lines
(i.e., why an option that was subject to extended domi- ($7755 = l < $12,183), option A was optimal, fol-
nance had the highest CEAC while an optimal option lowed by option F ($12,183 = l < $110,845), and
never had the highest CEAC), and showed that those finally, option E (l = $110,845). It should also be noted
results were entirely appropriate. In the first of these that the CEAF may be disjointed. This can be seen at the
analyses, we assumed that the costs and effects of first vertical line (l = $7755), where the probability of
different options were independent, but that the rela- option B being cost-effective was 32.7%, whereas the
tionship between costs and effects within options corresponding probability for option A was 25.8%.
remained the same. In the second analysis, we used the Moreover, at this point, as well as the optimal option
original correlation matrix but made the assumption (A) not having the highest probability of being cost-
that the variance associated with the mean cost and effective, there was a 74.2% probability that option A
mean effect of each option was one-fifth of that in the was not the most cost-effective option. This arose
base-case (which may have occurred if the original because the optimal option was determined by the
analysis had involved a larger sample size). Finally, the highest expected net benefit, whereas the CEAC simply
third analysis repeated that of the second, but with the represents the proportion of iterations over which each
assumption that the variance was five times greater option had the highest net benefit.
than that in the base-case (representing a smaller By replicating the Goeree et al. [1] analysis, it was
sample size). also possible to estimate the EVPI for an individual
patient (Fig. 1c). The EVPI was relatively low when
there was a high probability that the optimal option
Results was cost-effective, with local maxima when the
Replication of Previous Analyses optimal option changed (and the probability of a
wrong decision was relatively high), e.g., at a l of
Base-case. After constructing the variance–covariance approximately $12,000 and $111,000. However, this
matrix for the analysis by Goeree et al. [1], the template was not always the case, as when the optimal option
was used to replicate their analysis. The cost and changed from B to A there was a point of inflexion, but
effect pairs for each of the 1000 iterations, for each the EVPI continued to increase. This arose, despite
of the seven options, were first plotted on the cost- there being a local maxima in the error probability,
effectiveness plane (Fig. 1a). After determining which because the falling probability of making an incorrect
options were dominated (none) and subject to extended decision was outweighed by the increasing conse-
dominance (options C, D, and G), it was possible to quences of a wrong decision. Finally, it should be noted
estimate the efficiency frontier. In Figure 1a, it can that at high levels of the cost-effectiveness threshold,
be seen that this is composed of options B, A the EVPI fell to virtually zero. This was because of
(ICER = $7755, compared with B), F (ICER = virtually nonexistent uncertainty at these levels of l,
$12,183, compared with A), and E (ICER = $110,845, where option E had a high probability of being
890 Barton et al.

(a)

1800

E
G
1400
C
D
A
Cost ($)

B 1000

0.875 0.885 0.895 0.905 0.915

600 F

200
Effect (QALYs)

(b)
1
F
Probability of being cost-effective

0.9 E
0.8 B
0.7

0.6
0.5

0.4 D
0.3 C
0.2 A
G
0.1

0
1000 10000 100000 1000000
Cost-effectiveness threshold ($)

(c) 1 140

0.9 F E
Probability of being cost-effective

120
EVPI ($) for an individual patient

0.8
B
0.7 100

0.6
80
0.5
60
0.4

0.3 A
40
0.2
EVPI 20
0.1

0 0
1000 10000 100000 1000000
Cost-effectiveness threshold ($)

Figure 1 Replication of the analysis by Goeree et al. [1]. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability
curves (b), and the cost-effectiveness acceptability frontier and the expected value of perfect information (EVPI) (c).
Optimal Decisions: The Role of the CEAC, CEAF, and EVPI 891

cost-effective because of strong evidence that it was the optimal option being cost-effective increased, e.g., at
most effective option. a l of $12,000; in the base-case the EVPI was 41.4
when option A had a 17.9% probability of being cost-
Sensitivity analysis. When the analyses by Goeree effective, yet it rose to 107.4 when the options were
et al. [1] were replicated, with the assumption that the assumed to be independent and the probability of
costs and effects between options were independent, option A being cost-effective increased to 19.8%. This
the cost-effectiveness plane (Fig. 2a) was visually the change in the EVPI can be explained by the fact that
same as in the base-case analyses (see Fig. 1a), because when option B, for example, had a high net benefit in
near identical cost and effect pairs (within options) a particular iteration in the base-case (independent)
were drawn in both of these probabilistic analyses. analysis, it was more (less) likely that other options
Thus, the ICERs were very similar to that in the base- would also have a high net benefit. Thus, in the base-
case analyses. Conversely, the CEACs changed, with case the consequences of the wrong decision (repre-
the curves tending to converge—those options that sented by the incremental net benefit) were estimated
previously had the highest probability of being cost- to be less than in the analysis, where the options were
effective, for a particular l, tended to have a lower assumed to be independent.
probability of being cost-effective, and other options The final two analyses had the same correlation
tended to have a higher probability of being cost- structure as the base-case, but had differing levels of
effective. Importantly, this meant that option A (which variance. In both analyses, the cost and effect pairs for
was the optimal option between a l of $7755 and each option were centered around the same means,
$12,183) now had the highest probability of being and thereby ICERs, but the pairs were more condensed
cost-effective for a small range of l-values, and option when the level of variation was reduced by a factor of
C (which was subject to extended dominance) no five (Fig. 3a) and more spread out when the variation
longer had the highest probability of cost-effectiveness was inflated by a factor of five (Fig. 4a). Looking at
over any threshold value. Here, for reasons of brevity, the CEACs, reducing the level of variation tended to
rather than plotting the CEACs in the traditional increase the probability of optimal options being cost-
manner (as in Fig. 1b), they have been plotted on the effective, at the expense of the probability associated
same diagram as the CEAF and EVPI. In Figure 2b, the with other options (Fig. 3b). Indeed, option A now had
CEACs are plotted, where the (bold) continuous the highest probability of being cost-effective over a
section of the CEAC denotes the range of l-values over reasonable range of l-values, and options C/D only
which each option was optimal, and the (lighter) briefly did so ($11,690 ⱕ l < $12,950).
dashed section of the CEAC denotes the range of In the Goeree et al. [1] analysis, the two poten-
l-values over which each option did not provide the tially counterintuitive results arose because options C
highest expected net benefit. and D, which were subject to extended dominance,
In the Goeree et al. [1] analysis, these two poten- had relatively high levels of variation in net benefit,
tially counterintuitive results arose because all the compared with option A. For example, at a l of
options were positively correlated. Consequently, $12,000, the mean net benefits for options A, C, and
when, for example, option B had a low net benefit in a D were $9844, $9841, and $9840, and the variances
particular iteration, it was likely that in the same itera- were $7426, $10,967, and $11,027. Thus, as the
tion, other options would also have a low net benefit, probability distributions surrounding the mean net
and at low levels of l, this meant that option B had benefit of options C/D were more spread out than for
the highest net benefit in a high proportion of itera- option A within a particular iteration, despite the
tions (and thereby a high CEAC). Conversely, when marginally lower mean values of net benefit for C/D,
the options were assumed to be independent, when there was a higher probability of a higher net benefit
option B had a low net benefit, then there was a higher being drawn for options C/D than for option A.
chance of another option having a higher net benefit, When the level of variation was reduced, the distri-
in the same iteration, and option B thereby had the butions became more tightly centered around the
highest net benefit in a lower proportion of iterations. mean, and within a particular iteration, option A had
The assumption that the costs and effects between a higher probability of having a higher net benefit
options were independent also changed the CEAF than options C/D (see CEACs in Fig. 3b). Increasing
(Fig. 2b), where the optimal options tended to have a the level of variation had the opposite effect on the
lower probability of being cost-effective, and the EVPI CEACs (Fig. 4b), as those options that had a rela-
(Fig. 2b), which also tended to be higher. Here, as in tively high level of variation (e.g., options C and D)
Figure 1c, the CEAF was again constructed by plotting had the highest net benefit in a greater proportion of
the CEAC, as a continuous line, over the range of iterations, and thereby a greater probability of being
l-values over which each option had the highest cost-effective than in the base-case, at the expense of
expected net benefit. It should also be noted that the the those options that had a relatively low level of
EVPI even increased when the probability of the variation (e.g., options A and B).
892 Barton et al.

(a)

1800

G
1400 C E
D
Cost ($)

1000
A
B
F
0.875 0.88 0.885 0.89 0.895 0.9 0.905 0.91 0.915
600

200

Effect (QALYs)

(b)
1 140

0.9
120
Probability of being cost-effective

EVPI ($) for an individual patient

0.8 E
0.7 100
F
0.6
B 80

0.5 EVPI
60
0.4

0.3 40
A G
0.2
D 20
0.1
C
0 0
10 00 1 0000 1 0000 0 1000 000

Cost-effectiveness threshold ($)

Figure 2 Replication of the analysis by Goeree et al. [1], with the assumption that the costs and effects between options were independent. Results are
presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability curves, cost-effectiveness acceptability frontier, and expected value
of perfect information (EVPI) (b).

Reducing the level of variation increased the pro- CEAF was more often equivalent to the uppermost
portion of iterations where optimal options had the CEAC, and the probability of making a wrong deci-
highest net benefit. Thus, as optimal options tended to sion tended to be lower (Fig. 3b). This, combined with
have the highest probability of being cost-effective, the the lower level of variation in net benefit, which
Optimal Decisions: The Role of the CEAC, CEAF, and EVPI 893

(a)

1800

1400
E
G

C
Cost ($)

D
A
1000
F

0.875 B 0.885 0.895 0.905 0.915

600

200

QALYs
(b)
1 140
E
F
0.9
120
Probability of being cost-effective

EVPI ($) for an individual patient

0.8 B
0.7 100

0.6 A
80

0.5

60
0.4
D
0.3
C
40

0.2
20
0.1
EVPI G
0 0
1000 100 00 10 0000 10000 00

Cost-effectiveness threshold ($)

Figure 3 Replications of the analysis by Goeree et al. [1] with the assumption that the variance associated with the mean cost and mean effect of each
option was one-fifth of that in the base-case analysis. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability
curves, cost-effectiveness acceptability frontier, and expected value of perfect information (EVPI) (b).

reduced the consequences of making a wrong decision, less frequently equivalent to the uppermost CEAC, the
meant that the EVPI was also lower when the variation probability of making the wrong decision tended to
was reduced (Fig. 3b). However, the converse occurred increase, as did the consequences of making a wrong
when the level of variation increased—the CEAF was decision, and in turn the EVPI also increased (Fig. 4b).
894 Barton et al.

(a)

1800
G

D C E

1400
A
Cost ($)

1000

0.875 0.885 0.895 0.905 0.915

600
B
F

200

QALYs
(b)
1 140

0.9 E
120
Probability of being cost-effective

EVPI ($) for an individual patient

0.8

0.7 F 100

0.6
80
B
0.5
60
0.4

0.3 EVPI
C 40

0.2 A
G
20
0.1 D

0 0
1 000 100 00 10 0000 1 000 000

Cost-effectiveness threshold ($)

Figure 4 Replications of the analysis by Goeree et al. [1] with the assumption that the variance associated with the mean cost and mean effect of each
option was five times that in the base-case analysis. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability
curves, cost-effectiveness acceptability frontier, and expected value of perfect information (EVPI) (b).

Discussion assessing multiple options, we replicated the analysis

of a previous paper and performed sensitivity analysis
In order to show how to present the optimal decision on the assumptions of that analysis. In doing so, we
and the uncertainty associated with that decision when demonstrated that CEACs can sometimes show 1) that
Optimal Decisions: The Role of the CEAC, CEAF, and EVPI 895

options that are subject to extended dominance have only a first step in determining the value of further
the highest probability of being cost-effective; or 2) research. Indeed, in order to determine the latter, the
that optimal options do not have the highest probabil- third implication of our research is that the EVPI
ity of being cost-effective for any value of the cost- should be calculated. Moreover, because the correla-
effectiveness threshold. Thus, we have shown that tion structure between options was shown to affect
CEACs should not be used to determine the optimal the consequences of making the wrong decision, this
decision, as previously suggested by Kamath et al. [7]. shows that the EVPI can provide further information
In addition, we have explained that these two effects that is not captured by the cost-effectiveness plane or
can arise because of the correlation structure and dif- the CEAF.
ferences in the level of variation between multiple
options. Furthermore, in order to present the optimal Strengths and Weaknesses
decision and the level of uncertainty associated with A crucial assumption within this paper is that optimal
that decision, we have demonstrated that the CEAF, decisions will be made when options with the highest
along with the EVPI, should be calculated. Finally, expected net benefit are implemented (i.e., when ben-
it has also been shown that it is, for example, possible efits are maximized for a given cost). However, Coast
for uncertainty to increase (as assessed by the EVPI) [5] has argued that such an objective does not comply
when the probability of the optimal option being cost- with how society would wish to allocate scarce health-
effective also increases. care resources, because for example, it ignores the
notion of equity. To reflect this, the objective could be
Implications adapted to include an equity weighting, but this would
We have demonstrated, as previously suggested not change the fact that it is possible for an optimal
[17,21,22], that instead of using the CEAC to estimate option (unless optimal is defined in terms of probabil-
which option is optimal, the sole purpose of presenting ity) not to have the highest probability of being cost-
the CEAC is to demonstrate the level of uncertainty effective, and thus the CEAC should not be used to
associated with an option. Moreover, we have also determine the optimal allocation of resources.
shown that if our objective is to maximize the expected We are only aware of one previous possible expla-
net benefit regardless of the level of uncertainty, as nation as to why an optimal option may not have the
argued previously by Claxton [15], then this can result highest probability of being cost-effective, namely, that
in choosing options where we know there is a high the incremental net benefit was positively skewed [17].
probability of making a wrong decision. For example, However, as this was not the case in the Goeree et al.
in the study by Goeree et al. [1], at a l of $12,000, [1] analysis, positive skewness could not account for
option A was optimal despite the fact that the prob- why it had previously been shown that an option that
ability of this being the wrong decision exceeded 80% was subject to extended dominance had, at certain
(see Fig. 1c). The first implication of our results, there- l-values, the highest probability of being cost-effective,
fore, is that when presenting the results of cost- nor could it account for why an option that was cost-
effectiveness analyses, it should be made clear that the effective never had the highest probability of being
CEACs are presented in order to depict the level of cost-effective [1]. Thus, the main strength of this article
uncertainty and not the optimal decision. is in demonstrating that these instances can be brought
We also demonstrated that though the cost- about by particular correlation structures, or by differ-
effectiveness plane can depict the optimal decision, it ences in the relative levels of variance between multiple
cannot depict the level of uncertainty, because it is options. Our results are also therefore supported by
insensitive to the correlation structure between others [29,30] who have shown that the correlation
options. Thus, the second implication of our results is structure between the costs and effects of different
that in order to simultaneously present the optimal options can greatly influence the level of uncertainty.
decision, along with the uncertainty associated with By constructing a template, we have further demon-
that decision, the CEAF should be plotted. Indeed, the strated that the shapes of the CEACs, the CEAF, and
only time it may be appropriate to present uncertainty the EVPI are highly dependent upon assumptions
on the cost-effectiveness plane is when two options are made about the correlation structure and the relative
being compared. This arises because in contrast to the levels of the variance.
situation for multiple options, the incremental cost and The template we created was based on the assump-
incremental effects can be plotted, thus directly reveal- tion of multivariate normality; thus, it may not be
ing the correlation structure. appropriate to use it in replicate studies where there is
The CEAF is, however, constructed from the CEAC substantial departure from this distribution. That said,
of the optimal option(s), and therefore only depicts it is possible to use the Cholesky decomposition tech-
uncertainty through the probability of not selecting the nique to incorporate correlations between other types
most cost-effective option. The CEAF does not portray of distributions, as has been explained by Briggs et al.
the consequences of the decision; as such, it provides [20]. Moreover, the template can be similarly adapted
896 Barton et al.

to extract the mean cost and mean effect from other symptomatic knee osteoarthritis. Value Health 2003;
probabilistic models, and thereby construct the 6:144–57.
CEACs, CEAF, and EVPI for other studies. 8 Elliott RA, Hooper L, Payne K, et al. Preventing non-
steroidal anti-inflammatory drug-induced gastrointes-
tinal toxicity: are older strategies more cost-effective
Conclusion in the general population? Rheumatology (Oxford)
2006;45:606–13.
The cost-effectiveness plane has limited use in repre- 9 Brown TJ, Hooper L, Elliott RA, et al. A comparison
senting uncertainty in the multiple-option case as it of the cost-effectiveness of five strategies for the
cannot represent correlation between the options, prevention of non-steroidal anti-inflammatory drug
which can play a very important role. CEACs can induced gastrointestinal toxicity: a systematic review
represent decision uncertainty, but should not be used with economic modelling. Health Technol Assess
to determine the optimal decision. Instead, the CEAF 2006;10:1–183.
shows the optimal choice and this can be augmented 10 Weinstein MC. Principles of cost-effective resource
using the EVPI plot to show the potential gains of allocation in health care organizations. Int J Technol
further research to reduce uncertainty. Assess Health Care 1990;6:93–103.
11 Cantor SB. Cost-effectiveness analysis, extended
dominance, and ethics: a quantitative assessment.
We thank the authors of a previous publication (Goeree R,
Med Decis Making 1994;14:259–65.
O’Brien B, Blackhouse G, Marshall J, Briggs A, and Lad R)
12 Claxton K, Sculpher M, Drummond M. A rational
for providing us with the results of their analyses, which were
framework for decision making by the National Insti-
used to inform some of the analyses in this paper.
tute For Clinical Excellence (NICE). Lancet 2002;
360:711–15.
Source of financial support: No specific funding was provided 13 Sculpher M, Claxton K. Establishing the cost-
for this piece of research, which was undertaken while Garry effectiveness of new pharmaceuticals under conditions
Barton undertook a PhD sponsored by the UK Economic & of uncertainty—when is there sufficient evidence?
Social Research Council (ESRC) (PTA-037-2004-00051) at Value Health 2005;8:433–46.
the University of Nottingham. The manuscript was written 14 Fenwick E, Palmer S, Claxton K, et al. An iterative
during an ESRC sponsored visit to the Health Economics Bayesian approach to health technology assessment:
Appraisal Team at the University of Glasgow. Andrew Briggs application to a policy of pre-operative optimization
is supported by the William R Lindsay Chair in Health Policy for patients undergoing major elective surgery. Med
& Economic Evaluation. Decis Making 2006;26:480–96.
15 Claxton K. The irrelevance of inference: a decision-
References making approach to the stochastic evaluation of
health care technologies. J Health Econ 1999;18:341–
1 Goeree R, O’Brien BJ, Blackhouse G, et al. Cost- 64.
effectiveness and cost-utility of long-term manage- 16 O’Hagan A, McCabe C, Akehurst R, et al.
ment strategies for heartburn. Value Health 2002; Incorporation of uncertainty in health economic
5:312–28. modelling studies. Pharmacoeconomics 2005;23:
2 Miller P, Chilvers C, Dewey M, et al. Counseling 529–36.
versus antidepressant therapy for the treatment of 17 Fenwick E, Claxton K, Sculpher MJ. Representing
mild to moderate depression in primary care: eco- uncertainty: the role of cost-effectiveness acceptability
nomic analysis. Int J Technol Assess Health Care curves. Health Econ 2001;10:779–87.
2003;19:80–90. 18 UK BEAM Trial Team. United Kingdom back pain
3 Ramsey SD, Berry K, Etzioni R, et al. Cost effective- exercise and manipulation (UK BEAM) randomised
ness of lung-volume-reduction surgery for patients trial: cost effectiveness of physical treatments for back
with severe emphysema. N Engl J Med 2003; pain in primary care. BMJ 2004;329:1381.
348:2092–102. 19 Black WC. The CE plane: a graphic representation of
4 Legood R, Gray A, Wolstenholme J, et al. Lifetime cost-effectiveness. Med Decis Making 1990;10:212–
effects, costs, and cost effectiveness of testing for 14.
human papillomavirus to manage low grade cytologi- 20 Briggs AH, Sculpher MJ, Claxton K. Decision Mod-
cal abnormalities: results of the NHS pilot studies. elling for Health Economic Evaluation. New York:
BMJ 2006;332:79–85. Oxford University Press, 2006.
5 Coast J. Is economic evaluation in touch with society’s 21 van Hout BA, Al MJ, Gordon GS, et al. Costs, effects
health values? BMJ 2004;329:1233–6. and C/E-ratios alongside a clinical trial. Health Econ
6 Koerkamp BG, Hunink MGM, Stijnen T, et al. Limi- 1994;3:309–19.
tations of acceptability curves for presenting uncer- 22 Fenwick E, O’Brien BJ, Briggs AH. Cost-effectiveness
tainty in cost-effectiveness analysis. Med Decis acceptability curves—facts, fallacies and frequently
Making 2007;27:101–11. asked questions. Health Econ 2004;13:405–15.
7 Kamath CC, Kremers HM, Vanness DJ, et al. The 23 Claxton K. Bayesian approaches to the value of infor-
cost-effectiveness of acetaminophen, NSAIDs, and mation: implications for the regulation of new phar-
selective COX-2 inhibitors in the treatment of maceuticals. Health Econ 1999;8:269–74.
Optimal Decisions: The Role of the CEAC, CEAF, and EVPI 897

24 Claxton K, Cohen JT, Neumann PJ. When is evidence 28 Palmer S, Smith PC. Incorporating option values into
sufficient? Health Affairs 2005;24:93–101. the economic evaluation of health care technologies.
25 Ades AE, Sculpher M, Sutton A, et al. Bayesian J Health Econ 2000;19:755–66.
methods for evidence synthesis in cost-effectiveness 29 Ades AE, Lu G. Correlations between parameters in
analysis. Pharmacoeconomics 2006;24:1–19. risk models: estimation and propagation of uncer-
26 Briggs AH. A Bayesian approach to stochastic cost- tainty by Markov Chain Monte Carlo. Risk Anal
effectiveness analysis. Health Econ 1999;8:257–61. 2003;23:1165–72.
27 Briggs AH, Goeree R, Blackhouse G, et al. Probabi- 30 Chessa AG, Dekker R, van Vliet B, et al. Correlations
listic analysis of cost-effectiveness models: choosing in uncertainty analysis for medical decision making:
between treatment strategies for gastroesophageal an application to heart-valve replacement. Med Decis
reflux disease. Med Decis Making 2002;22:290–308. Making 1999;19:276–86.