JOURNAL OF MULTI-CRITERIA DECISION ANALYSIS

J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)

Published online 6 June 2003 in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/mcda.320

Weight Approximations in Multi-attribute Decision Models

RON ROBERTSa and PAUL GOODWINb,*
a
Faculty of Computing, Engineering and Mathematical Sciences, University of theWest of England,
Frenchay, Bristol, UK
b
The Management School, University of Bath, Bath, UK

ABSTRACT
The use of surrogate weights based on rankings has been proposed as a method for avoiding difficulties associated
with the elicitation of weights in multi-attribute decision analysis. When the simple multiattribute rating technique
using swings (SMARTS) method is being employed it has been suggested that rank order centroid (ROC) weights
are the best surrogate weights to use. This study shows that ROC weights are appropriate to use as a substitute for
original weights that are constrained to sum to a fixed total (usually 1 or 100) as used in the point allocation method.
If, however, the original weights are determined without any initial restrictions, as in the direct rating method, and
are then normalized, which is the common procedure in SMARTS analysis, then the ROC weights do not provide
the best approximations to the original weights. This paper shows how to obtain rank order distribution (ROD)
weights that provide a better approximation than the ROC approach to unrestricted original weights. The paper also
shows that, as the number of attributes in a decision problem increases, the ROD weights approximate to the more
easily calculated rank sum weights. Copyright # 2003 John Wiley & Sons, Ltd.

KEY WORDS: multi-criteria decision making; weight elicitation; rank order centroid weights

1. INTRODUCTION O1 O2 . Om
Decision makers are often required to choose W1 V11 V21 . Vm1
between several alternatives or options where each
option exhibits a range of attributes. When W2 V12 V22 . Vm2
choosing a new home, for example, the different
dwellings (options) under consideration may well . . . . .
vary in such attributes as cost, size, location,
facilities and age. Each option may score well with Wn V1n V2n . Vmn
respect to certain attributes and less well with
others. The relative importance to the decision Totals 1 V(O1) V(O2) . V(Om)
maker of each attribute will, to a large extent,
determine which option is regarded as best overall.
There are many procedures described in the Here Vij is the value assigned to the ith option
literature for formalizing the decision-making and jth attribute, 04Vij 4100. The Wj > 0 are
process under these conditions. Examples are weights reflecting the relative importance Pn of the
found in Keeney and Raiffa (1976), Von Winter- ranges of the attribute values, with j¼1 wj ¼ 1.
feldt and Edwards (1986), Green and Srinivasan In
Pn addition, w1 5w 2 5


5wn > 0 and vðOi Þ ¼
(1990) and Saaty (1994). This paper focuses on j¼1 wj vij where v is the aggregate multi attribute
multi-attribute value evaluation processes that value in the range 1005v50:
involve identifying single attributes and then Because the elicitation of weights can be
evaluating alternatives on them to form an difficult, several methods have been proposed for
‘attributes by options’ matrix. reducing the burden of the process. Many of these
methods involve asking the decision maker simpler
questions about the relative importance of the
attributes and using the responses to identify
*Correspondence to: University of Bath, The Manage- weights that are intended to approximate the
ment School, Bath BA2 7AY, UK. E-mail: mnspg@ decision maker’s ‘true’ weights. One of the more
management.bath.ac.uk recent proposals has been based on the use of rank

Copyright # 2003 John Wiley & Sons, Ltd.

292 R. ROBERTS AND P. GOODWIN

order centroid (ROC) weights (Barron and Bar- rating, (ii) point allocation and (iii) ranking. These
rett, 1996a; Edwards and Barron, 1994). In a approaches are outlined below.
series of simulation experiments, designed to assess (i) Direct rating method: The direct rating
the accuracy of ROC weights as approximations to method, discussed by von Winterfeldt and Ed-
‘true’ weights, Barron and Barrett found that wards (1986), uses ‘direct numerical ratio judge-
‘ROC captures. . . a substantial portion of the ments of relative attribute importance’. There are
information content of totally precise weights’ a number of ways of implementing this method. In
and, compared to other approximation methods, one approach the decision maker assigns an
‘ROC is clearly and overwhelmingly the most arbitrary importance of (say) 10 to the least
efficacious’. However, we argue below that the important attribute. Then the next most important
accuracy reported by Barron and Barrett (and by attribute is identified and a decision is made about
Edwards and Barron in a similar study) is based how much more important it is than the previous
on particular assumptions about how decision attribute and so on. The raw weights are P then
makers’ ‘true’ weights are formed and these normalized using the formula wi ¼ wi = ni¼1 wi
assumptions may not be valid in certain circum- where wi is the ith raw ratio weight and wi is the
stances. In the light of this, we propose an ith normalized weight.
alternative method that can be used when the An alternative approach involves arbitrarily
assumptions underlying the ROC method are not assigning a raw weight of 100 to the attribute
applicable. where switching from the worst to the best option
The paper is structured as follows. After a brief on that attribute is most desirable. The desirability
review of weight elicitation methods, we examine of making similar worst-to-best switches on each
the assumptions underlying ROC weights. The of the other attributes is then assessed relative to
alternative method for approximating weights is this, yielding raw weights on a scale with a
then described and formulae are derived to maximum of 100. Finally the weights are normal-
operationalize this method. Finally, the paper ized to sum to either 1 or 100 (see Goodwin and
reports the results of a simulation study which Wright, 1998).
was used to assess the relative efficacy of the alter- (ii) Point allocation method: In the point alloca-
native approximation method when the assump- tion method the decision maker has a ‘budget’
tions underpinning ROC weights do not apply. of points to allocate between the attributes in a
way that reflects their relative importance. For
example, the decision maker may be asked to
2. WEIGHT ELICITATION allocate 100 points between the five attributes
that are relevant to a particular decision. Clearly,
When obtaining the weights (Wj ) that will be in this method there is no need to normalize
associated with the ‘attributes by options’ matrix, the weights since the sum of 100 is already
it is important that they should take into prescribed.
consideration the decision maker’s relative pre- Doyle et al. (1997) declare the point allocation
ference for swings from the worst to the best method is a more difficult task since it is easier to
options on each attribute (Edwards and Barron, take 100 as the weight for the most important
1994; Goodwin and Wright, 1998). This is because attribute and then to allocate weights relative to
the value scale measuring the relative performance this 100 starting point as the weight of successive
of the options on a given attribute will usually attributes. The decision maker has no need to
range from 0 (for the worst option on that worry about the constraint that the total must be
attribute) to 100 (the best). Since similar 0–100 some specified value. Because of this Doyle et al.
scales will be used for all attributes, the weights argue that ‘Since the set of cognitive operations
will need to reflect the relative importance of these required to use the two methods is different, there
0–100 swings to the decision maker. However, in is every possibility that different decision weights
the following discussion, for brevity, we will will result’. Indeed, as Doyle et al. point out,
simply refer to ‘relative importance of the ranges although this method of determining weights and
of attribute values’ as ‘relative importance of the the direct rating method would seem to be minor
attributes’. There are several methods described in variants of each other, in practise they produce
the literature for determining weights. Most different profiles of decision weights, a result that
methods fall into one of three categories: (i) Direct surprised Jia et al. (1998).

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
 WEIGHT APPROXIMATION 293

(iii) Rank ordering of attributes: Barron and written as

Barrett (1996a) catalogue deficiencies in the
methods used by decision makers to determine wi ðRSÞ ¼ 2ðn þ 1  Ri Þ=nðn þ 1Þ; i ¼ 1; . . . ; n
exact weights. They cite evidence in studies by
Schoemaker and Waid (1982), Borcherding and where the ith rank is denoted by Ri .
(c) RR weights: This method uses the reciprocal
von Winterfeldt (1988), Epple (1990) and Jaccard
of the ranks which are normalized by dividing
et al. (1986) that the weights generated are highly
influenced by the method used to produce them. each term by the sum of the reciprocals. The
Since there is no agreed method of eliciting the formula is
weights there is no way of directly identifying 1=i
the ‘true’ set of weights. They also believe that the wi ðRRÞ ¼ Pn 1
rank i ¼ 1; . . . ; n:
weights produced may be more precise than those j¼1 j
envisaged by the decision maker who may be more option j ¼ 1; . . . ; n
comfortable and confident with a simple ranking
of the importance of each attribute swing, So for four options the weight for the third ranked
especially if it represents the considered outcome attribute is 13=ð1 þ 12 þ 13 þ 14Þ ¼ 13=25
12 ¼ 0:1600.
of a group of decision makers. This has led to
the development of a number of methods that
enable the rankings to be translated into ‘surro-
ogate’ weights that represent an approximation of 3. PAST COMPARISONS OF
the ‘‘true’’ weights (e.g. see Stillwell et al. (1981). APPROXIMATION METHODS
These methods include the ROC, the rank sum
(RS) and the rank reciprocal (RR) weights. For Studies by Edwards and Barron (1994) and Barron
each of these methods, the formulae are shown and Barrett (1996a), have investigated ‘What
below while the corresponding weights for each surrogate weights formula best represents the rank
rank, for numbers of attributes ranging from ordering of swing weights (and no other in-
n ¼ 2210 are listed in Tables I–III. formation)?’ and ‘How efficacious are the formu-
(a) ROC weights:
P In this case the weights lae}both relatively and absolutely?’ The
wi ðROCÞ ¼ 1=n nj¼i 1=j; i ¼ 1; . . . ; n. So, for researchers undertook simulations comparing the
example, when n ¼ 5 and i ¼ 3 and 4, three types of surrogate weights referred to above
w3 ¼ ð0 þ 0 þ 1=3 þ 1=4 þ 1=5Þ=5 ¼ 0:1567 and (RS, RR and ROC) with randomly generated
w4 ¼ ð0 þ 0 þ 0 þ 1=4 þ 1=5Þ=5 ¼ 0:0900. ‘real’ weights. They concluded that, although the
(b) RS weights: In the RS procedure the weights, surrogate weights they compared were surprisingly
wi ðRSÞ, are the individual ranks normalized by efficacious, the ROC weights were the most
dividing by the sum of the ranks. The formula accurate. As a consequence Edwards and Barron
producing the weights, in its simplest form can be (1994) incorporated ROC weights into their

Table I. (ROC) weights

Attributes

Rank 2 3 4 5 6 7 8 9 10

1 0.7500 0.6111 0.5208 0.4567 0.4083 0.3704 0.3397 0.3143 0.2929

2 0.2500 0.2778 0.2708 0.2567 0.2417 0.2276 0.2147 0.2032 0.1929
3 0.1111 0.1458 0.1567 0.1583 0.1561 0.1522 0.1477 0.1429
4 0.0625 0.0900 0.1028 0.1085 0.1106 0.1106 0.1096
5 0.0400 0.0611 0.0728 0.0793 0.0828 0.0846
6 0.0278 0.0442 0.0543 0.0606 0.0646
7 0.0204 0.0334 0.0421 0.0479
8 0.0156 0.0262 0.0336
9 0.0123 0.0211
10 0.0100

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
294 R. ROBERTS AND P. GOODWIN

Table II. (RS) weights

Attributes

Rank 2 3 4 5 6 7 8 9 10

1 0.6667 0.5000 0.4000 0.3333 0.2857 0.2500 0.2222 0.2000 0.1818

2 0.3333 0.3333 0.3000 0.2667 0.2381 0.2143 0.1944 0.1778 0.1636
3 0.1667 0.2000 0.2000 0.1905 0.1786 0.1667 0.1556 0.1455
4 0.1000 0.1333 0.1429 0.1429 0.1389 0.1333 0.1273
5 0.0667 0.0952 0.1071 0.1111 0.1111 0.1091
6 0.0476 0.0714 0.0833 0.0889 0.0909
7 0.0357 0.0556 0.0667 0.0727
8 0.0278 0.0444 0.0545
9 0.0222 0.0364
10 0.0182

Table III. (RR) weights

Attributes

Rank 2 3 4 5 6 7 8 9 10

1 0.6667 0.5455 0.4800 0.4379 0.4082 0.3857 0.3679 0.3535 0.3414

2 0.3333 0.2727 0.2400 0.2190 0.2041 0.1928 0.1840 0.1767 0.1707
3 0.1818 0.1600 0.1460 0.1361 0.1286 0.1226 0.1178 0.1138
4 0.1200 0.1095 0.1020 0.0964 0.0920 0.0884 0.0854
5 0.0876 0.0816 0.0771 0.0736 0.0707 0.0682
6 0.0680 0.0643 0.0613 0.0589 0.0569
7 0.0551 0.0525 0.0505 0.0488
8 0.0460 0.0442 0.0427
9 0.0393 0.0379
10 0.0341

simplified alternative to SMARTS called SMAR- ble values of ‘true’ weights it is necessary to
TER (SMART Exploiting Ranks) because consider how the ‘true’ weights might be distrib-
uted. Barron and Barrett (1996a) argue the case
(the) results of extensive simulations . . .suggest for sampling from a uniform distribution on the
that the SMARTS and SMARTER will agree basis that no weight is more likely than another.
on which option has the highest aggregate Consider, first, the simplest example involving
benefit in 75%–87% of cases. Even when they only two attributes. The most important attribute
did not agree, the option identified by SMAR- must be given a weight (w1 ) which is greater than
TER as having the highest aggregate benefit 0.5 and less than 1. As the weights in the points
tended to have a very similar score to the ‘best’ allocation method sum to 1 the weight of the other
option identified by SMARTS, suggesting that attribute must be (1  w1 ). Assuming that the
‘true’ weights are uniformly distributed, the means
an option which was ‘not too bad’ was being
of the two distributions of the ranks would be
picked by SMARTER. 0.75 and 0.25, respectively, and these correspond
with the ROC weights for two attributes. (See
Figure 1.)
4. THE RATIONALE FOR ROC WEIGHTS For problems involving more than two attri-
butes a mathematical approach is required to
In order to investigate how effective the surrogate determine the theoretical distribution of the
weights perform as approximations for the possi- ranked weights following a points allocation

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
 WEIGHT APPROXIMATION 295

Distribution of Ranks Point Allocation n = 2 length then the exact probability distribution for
5 the kth largest section is given by Rao and Sobel
(1980) to be
4
n X
nk  
j nk
Rank 1
fwk ðxÞ ¼ kðn  1Þ ð1Þ
Rank 2 k j¼0 j
3
f (x)

n2
 h1  ðk þ jÞxi ð1Þ
2
where k ¼ 1; 2; . . . ; n and where hai ¼ maxða; 0Þ.
This is equivalent to the formula given by Holst
1 (1980) for the length of the jth longest piece of
stick when a stick of uniform length is broken at
0 random into n pieces and the pieces are then
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ranked by size. Appendix A lists the equations of
(a) x the piecewise curves which (1) produces for n ¼ 2,
3 and 4. The corresponding distributions for n ¼ 2,
Distribution of Ranks Point Allocation n = 3.
7 3 and 4 are shown in Figure 1. The mean values of
Rank 2
the distributions of the ranks are equivalent to the
6 ROC weights i.e. 0.75, and 0.25 for n ¼ 2, 0.6111,
5
Rank 3 0.2778 and 0.1111 for n ¼ 3, and 0.5208, 0.2708,
Rank 1 0.1458 and 0.0625 for n ¼ 4.
4 Note that the underlying assumption of ROC
f (x)

weights is that the decision maker’s ‘true’ weights,

3
which we are attempting to approximate, will
2 naturally sum to a fixed total of 1 or 100. This, in
turn, implies that weights elicited through the
1
point allocation method are likely to be a valid
0 representation of the decision maker’s true pre-
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ferences. This assumption does not appear to be
(b) x reasonable for all circumstances, particularly given
Doyle et al.’s argument that this method places
Distribution of Ranks Point Allocation n = 4 greater cognitive complexity on the decision maker
12
than the direct rating method.
Rank 4 In the next section, we develop a weight
10
Rank 3 Rank 2 approximation method that assumes that valid
8 weights can be elicited through direct rating. We
refer to these weights as rank order distribution or
6 ROD weights.
Rank 1
f (x)

4
5. RANK ORDER DISTRIBUTION (ROD)
2 WEIGHTS
0 Recall that in the direct rating method the most
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
(c) x
important attribute is commonly assigned a weight
of 100 and the importance of the other attributes
Figure 1. (a) Distribution of Ranks Point Allocation are then assessed relative to this benchmark. The
n ¼ 2, (b) Distribution of Ranks Point Allocation n ¼ 3, ‘raw’ weights (wi ) obtained are then normalized to
(c) Distribution of Ranks Point Allocation n ¼ 4. sum to 1. Assuming that all attributes have some
importance, this means that the ranges of the
procedure. If the interval 0–1 is randomly cut into possible raw weights will be: w1 ¼ 100,
n sections by choosing n  1 random points 05w2 4100, 05w3 4w2 and, in general, 05wi 4
between 0 and 1 and ranking the sections by wi1 (where ia1). These ranges can be approxi-

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
296 R. ROBERTS AND P. GOODWIN

mated by representing all of the inequalities by If the independent variables come from the
less-than-or-equal-to expressions. We can then, U(0,100) distribution, then f ðv1 Þ ¼ f ðv2 Þ ¼


¼
1
like Edwards and Barron (1994) and Barron and f ðvn Þ ¼ 100 in which case, the joint distribution
1
Barrett (1996a), represent our uncertainty about f ðv1 ; v2 ; . . . ; vn Þ ¼ n! 100 n.
the ‘true’ weights by assuming uniform distribu- To illustrate the use of this distribution we will
tions for them. However, in the case of the direct again use the case where n ¼ 2. Consider attribute
rating method these uniform distributions will values v1 and v2 such that v1 5v2 50.
apply across the ranges specified above for wi As a result of the normalization process, the
rather than those assumed by the ROC method. distribution of the larger value, v1 , after normal-
We know of no empirical evidence that suggests ization can be obtained as follows,
that weights (or errors in the estimation of      
v1 1x
weights) follow a uniform distribution, but in the Pr 4x ¼ Pr v2 5 v1
absence of other information, this assumption is v1 þ v2 x
consistent with Bayes’ criterion (e.g. Kwon, 1978, But the expression ðð1  xÞ=xÞv1 50 is true if and
p. 142). In addition to Barron and Barrett, the only if 04x41 and the expression ðð1  xÞ=xÞv1
assumption has also been made by other research- 4v1 is true if and only if 124x41.
ers in this area such as Jia et al. (1998) and For the interval x 2 ½12; 1 it is always true that
Lahdelma and Salminen (2002). ðð1  xÞ=xÞv1 50 and so
Before examining the general problem of   Z 100 Z v1
determining approximate weights, assuming that v1 2
Pr 4x ¼ 2
dv2 dv1
direct rating is appropriate, it may be useful to re- v1 þ v2 0 ðð1xÞ=xÞv1 100
consider the simple problem involving two attri- Z 100  
2 1
butes. In this case, for simplicity and without loss ¼ 2 v1 dv1
of generality, we will assume that w1 ¼ 100. We 1002 0 x
 
therefore assume that w2  Uð0; 100Þ. Thus the 1
normalized weights will be ¼ 2
x
  
w1 ¼ 100=ð100 þ w2 Þ and w2 ¼ w2 =ð100 þ w2 Þ Differentiating gives the probability density func-
tion: fw1 ðxÞ ¼ x12 for x 2 ½12; 1.
so that The distribution of the smaller value v2 after nor-
Z 100 malization is given by Prðv2 =ðv1 þ v2 Þ4xÞ ¼ Prðv2 4
 
Eðw1 Þ ¼ 1=ð100 þ w2 Þ dw2 ðx=ð1xÞÞv1 Þ and the expression ðx=ð1xÞ=xÞv1 4v1
0 is true if and only if 04x412.
ðgiven the assumption of Therefore,
   Z 100 Z ðx=ð1xÞÞv1
uniformity for w2 Þ v2 2
h i Pr 4x ¼ dv2 dv1
 100 v1 þ v2 0 0 1002
¼ loge ð100 þ w2 Þ ¼ 0:693 Z 100 Z v1
0 2
þ dv2 dv1
Similarly, it can be shown that Eðw2 Þ ¼ 0:307. 0 0 100 2
These values are to be contrasted with the ROC ¼ x=ð1  xÞ þ 1 ¼ 1=ð1  xÞ
weights of 0.75 and 0.25.
To determine ROD weights for general pro- Differentiating gives fw2 ðxÞ ¼ 1=ð1  xÞ2 for x 2
blems we need to consider the probability dis- ½0; 12
tributions for the normalized weights that follow So for n ¼ 2 the distribution of the normalized
from our assumptions about the distributions of rank 1 is 1=x2 and is only defined between 12 and
the raw weights. The equations of the probability 1. The distribution of the normalized rank 2 is
density functions can be found from the joint given by 1=ð1  xÞ2 for the region between 0 and 12
distribution of n order statistics (see, for example, The means of these probability density func-
the theoretical text by Rice (1988)). tions are Lnð2Þ ¼ 0:6932 and 1  Lnð2Þ ¼ 0:3078,
If v1 ; v2 ; . . . ; vn are n independent random respectively.
variables ranked in order such that v1 5v2 5


5 A similar but more involved mathematical
vn then the joint distribution f ðv1 ; v2 ; . . . ; vn Þ ¼ process can be applied to produce the probability
n!f ðv1 Þf ðv2 Þ


f ðvn Þ. density functions for n ¼ 3, 4, etc. For n > 2 the

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
 WEIGHT APPROXIMATION 297

Distribution of Ranks. Direct Rating n = 2. The means of each rank order distribution
5 (ROD) for n ¼ 2 to 10 have been found mathe-
matically and are displayed in Table IV. These
4 weights can be compared with the ROC weights
Rank 2 Rank 1 (Table I), the RS weights (Table II), and the RR
3 weights (Table III).
f (x)

A graphical comparison of the ROD, ROC and

2 RS weights for 3 and for 9 attributes can be seen in
Figures 3 and 4, respectively (for clarity RR
1 weights have been omitted from the graph).
There is a very close match between the ROD and
0 RS weights in both cases. This matching is found
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
(a) x whatever the number of attributes. Indeed, in general,
the ROD weights tend towards the RS weights as the
Distribution of Ranks Direct Rating n = 3 number of attributes increases. Thus, given that
7
ROD weights are difficult to calculate when the
Rank 2
6 number of attributes is large, a practical solution is
5
Rank 3 to use RS weights for large attribute problems.
Rank 1
The ROC weights, on the other hand, depart
4
markedly from both the RS and ROD weights.
f (x)

3 Figures 3 and 4 also demonstrate another poten-

tial benefit of using ROD, rather than ROC,
2
weights. Belton and Stewart (2002, p. 142) have
1 pointed out that ROC weights are ‘extreme’ in that
0 the ratio of the highest to the lowest weights is so
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 large that the lowest ranked attribute will only
(b) x have a very marginal influence on the decision. As
Distribution of Ranks Direct Rating n = 4. we mentioned earlier, in practice, attributes with a
12 relative importance as low as this would usually be
Rank 4 eliminated from the decision model. The use of
10
Rank 3 ROD weights goes some way to reducing this
Rank 2
8 extreme value problem. For example, for 3
attributes the ratio of w1 to w3 is 3.4 (not 5.5 as
f (x)

6
Rank 1
in the ROC weights). For five attributes the ratio
of w1 to w5 is 5.6 (not 11.1 as in ROC), though
4
Belton and Stewart argue that ratios exceeding 5
2 usually indicate that the least-important attribute
is almost totally dominated by the other attribute.
0 In summary, there is clear theoretical evidence to
(c)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 show that the ROC weights are the best surrogate
x
weights to use for the point allocation method of
Figure 2. (a) Distribution of Ranks. Direct Rating assigning raw weights and the ROD weights are
n ¼ 2, (b) Distribution of Ranks. Direct Rating n ¼ 3, the best for direct rating methods of assigning raw
(c) Distribution of Ranks. Direct Rating n ¼ 4. weights. We next test this theoretical case empiri-
cally by following Barron and Barrett’s procedure
density functions are a series of piecewise equa- of applying the different weight approximation
tions. These equations, for each of the rank methods to a large set of simulated decisions.
distributions for n ¼ 3 to 5, are set out in the
Appendix B. The equations for n ¼ 6 to 10 are
increasingly more cumbersome but can be pro- 6. EFFICACY OF ROD WEIGHTS
vided by the first author on request. Figure 2
shows the shape of the probability density func- The simulation study was used to compare the
tions for n ¼ 2, 3 and 4. efficacy of the new ROD, the ROC, the RS and the

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
298 R. ROBERTS AND P. GOODWIN

Table IV. (ROD) weights

Attributes

Rank 2 3 4 5 6 7 8 9 10

1 0.6932 0.5232 0.4180 0.3471 0.2966 0.2590 0.2292 0.2058 0.1867

2 0.3068 0.3240 0.2986 0.2686 0.2410 0.2174 0.1977 0.1808 0.1667
3 0.1528 0.1912 0.1955 0.1884 0.1781 0.1672 0.1565 0.1466
4 0.0922 0.1269 0.1387 0.1406 0.1375 0.1332 0.1271
5 0.0619 0.0908 0.1038 0.1084 0.1095 0.1081
6 0.0445 0.0679 0.0805 0.0867 0.0893
7 0.0334 0.0531 0.0644 0.0709
8 0.0263 0.0425 0.0527
9 0.0211 0.0349
10 0.0173

0.7 efficacy of the surrogate weights (Barron and

Barrett, 1996b):
0.6
0.5
(i) The hit rate}this is the proportion of
ROD decisions in which the surrogate weights
Weight

0.4 ROC lead to the selection of the same best

0.3 RS decision option as the true weights.
(ii) The average value loss}this is the mean
0.2 absolute difference between the multi-attri-
0.1 bute value of the option selected by the true
weights and that of the option selected by
0
1 2 3 the surrogate weights (the multiattribute
Rank value of this latter option is calculated using
the true weights).
Figure 3. Comparison of weights for 3 attributes.
In the following simulation the true weights
were found by giving the most favoured attribute a
0.35 raw weight of 100 and then randomly selecting
0.3 from a U(0, 100) distribution of weights for the
remaining attributes. The raw weights were then
0.25
ROD ranked and normalized to sum to 100. (It should
Weight

0.2 be noted that pilot trials early in the study

ROC
0.15
RS established that almost exactly the same distribu-
0.1 tion of ranked true weights were obtained if,
0.05 instead, the weights for all attributes were ran-
domly selected from a U(0, 100) distribution
0
1 2 3 4 5 6 7 8 9
before normalising.) As Belton and Stewart
Rank (2002) point out, in practice, attributes with
weights close to 0 would probably be eliminated
Figure 4. Comparison of weights for 9 attributes. from a decision model. However, in common with
other studies, we have sampled weights from the
whole 0–100 range because we have no empirical
RR weights. The decision outcomes using the data on where the weight cut-off point, below
selected surrogate weights were compared with which attributes would be eliminated, lies.
decisions that would have been obtained using the The simulated decision problems had either 2, 4,
true weights elicited through the direct rating 6, 8 or 10 options and, similarly, 2, 4, 6, 8 or 10
approach. Two criteria were used to measure the attributes (this yielded 25 decision ‘structures’) and

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
 WEIGHT APPROXIMATION 299

Table V. (Hit rate)

2 Options 4 Options 6 Options 8 Options 10 Options

Attributes ROD ROC RS RR ROD ROC RS RR ROD ROC RS RR ROD ROC RS RR ROD ROC RS RR

2 100.0 100.0 100.0 100.0 88.0 87.2 87.9 85.1 86.4 85.0 86.3 77.3 84.5 83.4 85.0 75.7 84.0 82.1 84.0 75.5
4 92.0 83.2 97.7 97.4 88.4 83.8 88.5 73.9 85.6 81.3 85.9 69.6 84.9 79.7 85.0 66.4 83.0 78.6 83.3 65.2
6 94.4 89.9 94.4 92.0 89.1 82.7 89.2 72.7 87.6 80.6 87.4 68.3 85.9 78.6 86.1 65.1 84.7 76.6 84.9 62.6
8 94.4 88.4 87.1 93.5 90.8 82.3 90.6 72.2 88.1 78.8 88.1 66.7 86.6 76.6 86.7 63.5 85.9 75.8 86.1 61.9
10 95.2 88.0 95.2 93.5 90.8 81.2 90.7 72.2 89.2 77.9 89.2 66.6 88.2 76.5 88.3 63.4 87.6 75.0 87.6 62.6

Table VI. (Average value loss)

2 Options 4 Options 6 Options 8 Options 10 Options

Attributes ROD ROC RS RR ROD ROC RS RR ROD ROC RS RR ROD ROC RS RR ROD ROC RS RR

2 0.00 0.00 0.00 0.00 0.95 1.08 1.00 3.23 0.99 1.16 1.04 3.66 1.00 1.16 1.04 3.09 0.93 1.11 0.99 2.70
4 0.97 3.11 0.39 0.78 0.73 1.29 0.75 3.57 0.71 1.14 0.70 3.55 0.68 1.11 0.72 3.61 0.70 1.10 0.70 3.46
6 0.40 0.95 0.40 0.54 0.48 1.13 0.49 3.02 0.45 1.05 0.47 2.94 0.47 1.03 0.46 3.05 0.49 1.05 0.49 3.11
8 0.31 1.03 0.15 0.66 0.30 1.05 0.32 2.59 0.32 1.05 0.32 2.74 0.35 1.02 0.35 2.81 0.35 1.00 0.34 2.76
10 0.20 0.96 0.20 0.66 0.25 0.95 0.26 2.31 0.24 1.00 0.25 2.48 0.24 0.95 0.25 2.47 0.24 0.97 0.25 2.39

10 000 simulation trials were undertaken for each 8. AVERAGE VALUE LOSS
of these structures. For every attribute, values
were assigned to each option by giving at random The average value losses of the surrogate weight
one option a value of 0 another a value of 100 and methods are shown in Table VI. It can be seen that
the rest a value randomly selected from a U(0, 100) once again the ROD and RS weights outperform
distribution. the ROC and RR weights in every decision
As the results displayed below show, using the structure. There is little to distinguish between
ROD and RS surrogate weights, the outcome the ROD and RS weights but the RR weights
match with the true weights was impressive and produce relatively high average value losses.
consistently better than the ROC and RR weights
for the two measures of efficacy chosen.
9. CONCLUSIONS

7. HIT RATE The theoretical means of the distribution of the

ranks when using the points allocation method
Table V shows the percentage of times the various correspond precisely with the ROC weights. This
surrogate weights produced the same best alter- paper acknowledges the use of ROC approxi-
native as the true weights. The match using ROD mation weights as an alternative to the points
weights is shown in here. It will be seen that the allocation method of determining raw weights in
range is from 100% (for the 2  2 case) to a multivariate analysis.
minimum of 83.0% (for the case involving 10 However, in SMARTS analysis the direct rating
options and 4 attributes). method of selecting raw weights is normally used
By comparison the ROC weights range from as it is cognitively simpler and therefore is assumed
100% (for the 2  2 case) to a minimum of 75% to yield more consistent and accurate judgements
(for the 10  10 case) and case by case are from the decision maker. These raw weights are
consistently poorer than the ROD weights. The then normalized and it has been shown here that
RS percentage match was almost indistinguishable this normalization process yields different theore-
from the ROD weights. By far the poorest match tical distributions for the ranks. The means of
was found when using the RR weights. these distributions are the ROD weights. In the

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
300 R. ROBERTS AND P. GOODWIN

light of this, we believe that serious consideration Waid, 1982}though they did not assess the direct
should be given to the use of ROD weights as rating method). This may be because the decision
surrogate weights in direct rating cases. The ROD maker’s initial weights are ill formed and the
weights for n ¼ 2 to 10 have been published in particular elicitation procedure in some way
Table IV and can therefore be easily implemented becomes part of the process of weight formation
by decision analysts. or because the structure inherent in some proce-
The formulae for the distribution of the ROD dures prevents the decision maker from expressing
weights become progressively more complex as the his or her true weights and serves to distort them.
number of attributes increase. Since the RS In the absence of an external, objective, analogue
weights are so easy to calculate and closely match for preferences the accuracy of the weights
the ROD weights for higher numbers of attributes determined through different procedures can only
it is also recommended that serious consideration be assessed on indirect criteria such as the user’s
should be given to the use of RS weights when it is trust in the method or the method’s cognitive
intended to approximate a distribution of weights complexity. Nevertheless, there may be occasions
for problems involving large numbers of attributes when an analyst has good reason to believe that a
and where it can be assumed that the appropriate given decision maker more naturally perceives the
alternative method for eliciting the ‘true’ weights’ weights in a decision problem to be structured as a
would have been the direct rating method. series of swings with an importance that is
Of course, ultimately the true test of a weight perceived to be relative to that of the most
approximation procedure is the extent to which it important (or least important) attribute, rather
accurately represents the decision maker’s true than as an allocation of points across the
preferences. The empirical research suggests that attributes. If the analyst seeks to approximate
the type of elicitation procedure adopted will have the decision maker’s weights on the basis of the
an effect on the attribute weights that are formally ranking of these swings, the method described here
assessed (e.g. Doyle et al. 1997; Schoemaker and is likely to provide the appropriate approximation.

APPENDIX A

The equations of the piecewise curves for each of the rank distributions (point allocation) produced by
formula 1 for n ¼ 2, 3 and 4 are as follows:

n=2
Rank 1 8
0 0
< 2ðð1  xÞ  ð1  2xÞ Þ ¼ 0; 04x41=2
fw1 ðxÞ ¼ 0 1=24x41
2ðð1  xÞ ¼ 2;
:
0 elsewhere
Rank 2

2!ð1  2xÞ0 ¼ 2; 04x41=2
fw2 ðxÞ ¼
0 elsewhere

For n=3
Rank 1 8
>
> 6ðð1  xÞ  2ð1  2xÞ þ ð1  3xÞÞ ¼ 0; 04x41=3
<
6ðð1  xÞ  2ð1  2xÞÞ ¼ 18x  6; 1=34x41=2
fw1 ðxÞ ¼
>
> 6ðð1  xÞ ¼ 6  6x; 1=24x41
:
0 elsewhere
Rank 2 8
< 12ðð1  2xÞ þ ð1  3xÞÞ ¼ 12x; 04x41=3
fw2 ðxÞ ¼ 12ð1  2xÞ; 1=34x41=2
:
0 elsewhere
Rank 3

6ð1  3xÞ; 04x41=3
fw3 ðxÞ ¼
0 elsewhere

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
 WEIGHT APPROXIMATION 301

APPENDIX A (Continued)

Similarly for n=4 the expressions are

8
>
> 12ðð1  xÞ2  3ð1  2xÞ2 þ 3ð1  3xÞ2  ð1  4xÞ2 Þ; 04x41=4
>
Rank 1 > > 2 2 2
< 12ðð1  xÞ  3ð1  2xÞ þ 3ð1  3xÞ Þ; 1=44x41=3
2 2
fw1 ðxÞ ¼ 12ðð1  xÞ  3ð1  2xÞ Þ; 1=34x41=2
>
>
>
> 12ðð1  xÞ2 Þ; 1=24x41
>
:
0 elsewhere
Rank 2
8
>
>
> 36ðð1  2xÞ2  2ð1  3xÞ2  ð1  4xÞ2 Þ; 04x41=4
<
36ðð1  2xÞ2  2ð1  3xÞ2 Þ; 1=44x41=3
fw2 ðxÞ ¼
>
>
> 36ðð1  xÞ2 Þ; 1=34x41=2
:0 elsewhere
Rank 3 8
2 2
< 36ðð1  3xÞ þ ð1  4xÞ Þ; 04x41=4
fw3 ðxÞ ¼ 36ðð1  3xÞ Þ;2 1=44x41=3
:
0 elsewhere
Rank 4

12ð1  4xÞ2 ; 04x41=4
fw4 ðxÞ ¼
0 elsewhere

APPENDIX B

The piecewise curves for each of the rank distributions (direct rating) for n ¼ 2 to 5:

n=2
Rank 1
(
1=x2 ; 1=24x41
fw1 ðxÞ ¼
0 elsewhere

Rank 2
(
1=ð1  xÞ2 ; 04x41=2
fw2 ðxÞ ¼
0 elsewhere

n=3
Rank 1
8 3 2
< 1=x þ 3=x ;
> 1=34x41=2
fw1 ðxÞ ¼ þ1=x3  1=x2 ; 1=24x41
>
:
0 elsewhere

Rank 2
8 2 2
< 1=ðx  1Þ þ 1=ð2x  1Þ ;
> 04x41=3
2
fw2 ðxÞ ¼ þ1=x2  1=ðx  1Þ ; 1=34x41=2
>
:
0 elsewhere

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
302 R. ROBERTS AND P. GOODWIN

APPENDIX B (Continued)

Rank 3
(
þ4=ðx  1Þ2  1=ð2x  1Þ2 ; 04x41=3
fw3 ðxÞ ¼
0 elsewhere

Similarly for n=4 the expressions are

Rank 1
8
> þ1=2x4  4=x3 þ 8=x2 ; 1=44x41=3
>
>
< 1=x4 þ 5=x3  11=2x2 ; 1=34x41=2
fw1 ðxÞ ¼
> 4 3 2
> þ1=2x  1=x þ 1=2x ;
>
:
1=24x41
0 elsewhere

Rank 2
8
>
> þ1=2ðx  1Þ2  1=ð2x  1Þ2 þ 1=2ð3x  1Þ2 ; 04x41=4
>
>
< 1=x3 þ 9=2x2 þ 1=2ðx  1Þ2  1=ð2x  1Þ2 ; 1=44x41=3
fw2 ðxÞ ¼
> þ1=x3  5=2x2 þ 1=2ðx  1Þ2 ;
> 1=34x41=2
>
>
: elsewhere
0

Rank 3
8 2 2 2
< 8=ðx  1Þ þ 9=ð2x  1Þ  1=ð3x  1Þ ;
> 04x41=4
fw3 ðxÞ ¼ þ1=x2  8=ðx  1Þ2 þ 1=ð2x  1Þ2 ; 1=44x41=3
>
:
0 elsewhere

Rank 4
(
þ27=2ðx  1Þ2  8=ð2x  1Þ2 þ 1=2ð3x  1Þ2 ; 04x41=4
fw4 ðxÞ ¼
0 elsewhere

For n=5 the expressions are

Rank 1
8
>
> 1=6x5 þ 5=2x4  25=2x3 þ 125=6x2 ; 1=54x41=4
>
>
> 5 4 3 2
< þ1=2x  11=2x þ 39=2x  131=6x ;
> 1=44x41=3
5 4 3 2
fw1 ðxÞ ¼ 1=2x þ 7=2x  15=2x þ 31=6x ; 1=34x41=2
>
>
> þ1=6x5  1=2x4 þ 1=2x3  1=6x2 ;
> 1=24x41
>
>
:
0 elsewhere

Rank 2
8
>
> 1=6ðx  1Þ2 þ 1=2ð2x  1Þ2  1=2ð3x  1Þ2 þ 1=6ð4x  1Þ2 ; 04x41=5
>
>
>
> 4 3 2 2 2 2 1=54x41=4
< þ1=2x  16=3x þ 43=3x  1=6ðx  1Þ þ 1=2ð2x  1Þ  1=2ð3x  1Þ ;
>
fw3 ðxÞ ¼ 1=x4 þ 23=3x3  85=6x2  1=6ðx  1Þ2 þ 1=2ð2x  1Þ2 ; 1=44x41=3
>
>
>
> 1=34x41=2
> þ1=2x4  7=3x3 þ 17=6x2  1=6ðx  1Þ2 ;
>
>
: elsewhere
0

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)
 WEIGHT APPROXIMATION 303

APPENDIX B (Continued)
Rank 3
8
>
> þ8=ðx  1Þ2  33=2ð2x  1Þ2 þ 9=ð3x  1Þ2  1=2ð4x  1Þ2 ; 04x41=5
>
>
< 1=x3 þ 13=2x2 þ 8=ðx  1Þ2  33=2ð2x  1Þ2 þ 1=ð3x  1Þ2 ; 1=54x41=4
fw3 ðxÞ ¼
>
> þ1=x3  9=2x2 þ 8=ðx  1Þ2  1=2ð2x  1Þ2 ; 1=44x41=3
>
>
: elsewhere
0

Rank 4
8 2 2 2 2
< 81=2ðx  1Þ þ 113=2ð2x  1Þ  33=2ð3x  1Þ þ 1=2ð4x  1Þ ;
> 04x41=5
2 2 2
fw4 ðxÞ ¼ þ1=x  81=2ðx  1Þ þ 16=ð2x  1Þ  1=2ð3x  1Þ ;
2 1=54x41=4
>
:
0 elsewhere

Rank 5
(
þ128=3ðx  1Þ2  81=2ð2x  1Þ2 þ 8=ð3x  1Þ2  1=6ð4x  1Þ2 ; 04x41=5
fw5 ðxÞ ¼
0 elsewhere

REFERENCES Jaccard J, Breinberg D, Ackerman LJ. 1986. Assessing

attribute importance: a comparison of six methods.
Barron FH, Barrett BE. 1996a. Decision quality using Journal of Consumer Research 12: 463–468.
ranked attributes weights. Management Science. Jia J, Fischer GW, Dyer JS. 1998. Attribute weighting
42(11): 1515–1523. methods and decision quality in the presence of
Barron FH, Barrett BE. 1996b. The Efficacy of response error: a simulation study. Journal of
SMARTER–Simple Multi-Attribute Rating Techni- Behavioral Decision Making 11(2): 85–105.
que Extended to Ranking. Acta Psychologica 93: Keeney RL, Raiffa H. 1976. Decisions with Multiple
23–36. Objectives: Preferences and Value Trade-offs. Wiley:
Belton V, Stewart TJ. 2002. Multiple Criteria Decision New York.
Analysis: An Integrated Approach. Kluwer Academic Kwon I-W. 1978. Statistical Decision Theory with
Publishers: Norwell, MA. Business and Economic Applications: A Bayesian
Borcherding K, Von Winterfeldt D. 1988. The effect of Approach. Petrocelli/Charter: New York.
varying value trees on multiattribute evaluations. Lahdelma R, Salminen P. 2002. Classifying alternatives
Acta Psychologica 68:153–170. in SMA using cross confidence factors. TICS Tech-
Doyle JR, Green RH, Bottomley PA. 1997. Judging nical Report, No. 472, Turku Centre for Computer
relative importance: direct rating and point allocation Science, Finland.
are not equivalent. Organisational Behavior and Rao JS, Sobel M. 1980. Incomplete dirichlet integrals
Human Decision Processes. 70(1): 55–72. with applications to ordered uniform spacings.
Edwards W, Barron HF. 1994. SMARTS and SMAR- Journal of Multivariate Analysis 10: 603–610.
TER: improved simple methods for multiattribute Rice JA. 1988. Mathematical Statistics and Data
utility measurement. Organisational Behavior and Analysis. Duxbury Press: Belmont, CA.
Human Decision Processes. 60: 306–325. Saaty T. 1994. The Analytical Hierarchy Process. Wiley:
Epple, 1990. Eliciting and reconciling multiattribute New York.
utility weights. Dissertation, University of Southern Schoemaker PHJ, Waid CD. 1982. An experimental
California, Los Angeles, CA. comparison of different approaches to determining
Goodwin P, Wright G. 1998. Decision Analysis for weights in additive utility models. Management
Management Judgment, (2nd edn). Wiley: Chichester. Science 28: 182–196.
Green PE, Srinivasan K. 1990. Conjoint analysis Stillwell WG, Seaver DA, Edwards W. 1981. A
in marketing: new developments with implications comparison of weight approximation techniques in
for research and practice. Journal of Marketing 54: multiattribute utility decision making. Organisational
3–19. Behavior and Human Performance 28: 62–77.
Holst L. 1980. On the lengths of the pieces of a stick Von Winterfeldt D, Edwards W. 1986. Decision Analysis
broken at random. Journal of Applied Probability 17: and Behavioral Research. Cambridge University Press:
623–634. Cambridge, MA.

Copyright # 2003 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 11: 291–303 (2002)