Assessing Quality of Policy Properties in

Verification of Access Control Policies

Evan Martin JeeHyun Hwang Tao Xie Vincent Hu

Computer Science Department National Institute of Standards
North Carolina State University and Technology
Raleigh, NC Gaithersburg, MD, USA
{eemartin, jhwang4, txie}@ncsu.edu [email protected]

Abstract cation and validation to ensure that the policy specifications

truly encapsulate the desires of the policy authors.
Access control policies are often specified in declarative Property verification [9, 11, 13, 14, 19, 24] consumes a
languages. In this paper, we propose a novel approach, policy and a property, and determines whether the policy
called mutation verification, to assess the quality of proper- satisfies the property. Policy verification, while useful, re-
ties specified for a policy and, in doing so, the quality of the quires the policy authors to write a property set, which can
verification itself. In our approach, given a policy and a set be verified against the policy under verification to ensure its
of properties, we first mutate the policy to generate various correctness. With a property set of higher quality (covering
mutant policies, each with a single seeded fault. We then a larger portion of a policy’s behavior), the policy authors
verify whether the properties hold for each mutant policy. If are more likely to detect policy faults (if any) and increase
the properties still hold for a given mutant policy, then the the confidence of its correctness.
quality of these properties is determined to be insufficient
In this paper, we propose a novel approach that assesses
in guarding against the seeded fault, indicating that more
the quality of properties for a policy based on mutation ver-
properties are needed to augment the existing set of proper-
ification, a counterpart of mutation testing [7] in verifica-
ties to provide higher confidence of the policy correctness.
tion. We have implemented our approach in a tool called
We have implemented Mutaver, a mutation verification tool
Mutaver. To the best of our knowledge, there is no previous
for XACML, and applied it to policies and properties from a
work to assess the quality of a property set and guide how
real-world software system.
to write a property set. The results of our mutation veri-
fication approach can be immediately used to aid property
elicitation and serve as a general quality metric for a set of
1. Introduction properties that ultimately check for faults in the policy.
In our approach, we propose mutation verification as a
Access control is one of the most fundamental and means to determine which properties in the given property
widely used security mechanisms for resources. It controls set interact with rules in a policy during policy verifica-
which principals such as users or processes have access to tion. In particular, given a policy, our approach automati-
which resources in a system. A growing trend has emerged cally seeds it with faults to produce various mutant policies,
toward writing access-control-policy specifications in stan- each containing one fault. Then given a property set for this
dardized, declarative languages such as XACML [1] and policy, our approach conducts property verification on this
Ponder [6]. Implementing and maintaining these policies policy (called the original policy) and each mutant policy. If
are important and yet challenging tasks, especially as access the property set holds for the original policy but fails to hold
control policies become more complex and are used to man- for the mutant policy, then the mutant is said to be killed by
age a large amount of distributed and sensitive information. the property set. The ratio of the number of killed mutants
Identifying discrepancies between policy specifications and to the total number of mutants serves as a metric to quantify
their intended function is crucial because correct implemen- comprehensiveness of the elicited property set. By analyz-
tation and enforcement of policies by applications are based ing the verification results, we can determine what rules, if
on the premise that the policy specifications are correct. As any, fail to interact with the given property set and thus help
a result, policy specifications must undergo rigorous verifi- guide property elicitation by targeting not-covered rules.
 Different from previous research [17, 21] on policy mu- Property P r1 is intuitive since we certainly do not
tation testing, instead of assessing the quality of a request want students to assign grades. Property P r2 is to ensure
set (by using a policy evaluation engine) in policy testing, that indeed faculty members can assign grades (otherwise
we assess the quality of a property set by using a policy who would assign them?). Finally, P r3 is an example of
verification approach. A property set can effectively sum- separation-of-duty since we do not want anyone to assign
marize various complex behaviors of a policy. In practice, their own grade, an apparent conflict of interest.
a property is often intuitive and expressed in various ways
1 If role = Faculty
(implicitly or explicitly) for a policy. Furthermore, as the 2 and resource =
behavior characterized by a property cannot be easily char- 3 (ExternalGrades or InternalGrades)
4 and action = (View or Assign)
acterized by one or multiple policy requests, we cannot as- 5 Then
sess the property set with the previous approaches [17, 21] 6 Permit
7 If role = Student
on policy mutation testing. 8 and resource = ExternalGrades
This paper makes the following main contributions: 9 and action = Receive
10 Then
11 Permit
• We propose a novel approach for assessing the quality
of properties for a policy in policy verification.
• We implement the proposed approach with an auto- Figure 1. Rules in an example XACML policy.
matic tool that facilitates automated mutation verifica-
tion of access control policies written in XACML [1].
1 If role = Faculty
• We present a case study on an access control policy 2 and resource =
3 (ExternalGrades or InternalGrades)
from a real-world software system to demonstrate the 4 and action = (View or Assign)
feasibility of this approach. 5 Then
6 Deny
7 If role = Student
The rest of the paper is organized as follows. Section 2 8 and resource = ExternalGrades
presents an example, Section 3 offers some background in- 9 and action = Receive
10 Then
formation, Section 4 presents our approach, Section 5 de- 11 Permit
scribes our experiences of applying our approach on a real-
world policy, Section 6 discusses issues in the approach,
Section 7 presents related work, and Section 8 concludes. Figure 2. The first mutant XACML policy.

2. Example 1 If role = Faculty

2 and resource =
3 (ExternalGrades or InternalGrades)
4 and action = (View or Assign)
This section illustrates our approach to mutation verifi- 5 Then
cation through a simple example. The example and corre- 6 Permit
7 If role = Student
sponding properties come from an example used by Fisler 8 and resource = ExternalGrades
et al. [9]. This access control policy formalizes a univer- 9 and action = Receive
10 Then
sity’s policy on assigning and accessing grades. It is a role- 11 Deny
based access control [8] policy with two roles specified in
the subject attribute: FACULTY and S TUDENT, two pos-
sible resource attributes: I NTERNAL G RADES and E XTER - Figure 3. The second mutant XACML policy.
NAL G RADES , and three possible action attributes: A SSIGN ,
V IEW, and R ECEIVE. For this example, we expect the fol- Figure 1 shows the example XACML policy. To keep the
lowing properties to hold: example readable and concise, we write the policy as simple
I F -T HEN statements. This representation over-simplifies
P r1 There do not exist members of S TUDENT who can the complexity of XACML policies but suffices for illustra-
A SSIGN E XTERNAL G RADES. tive purposes.
The first step of mutation verification is to generate mu-
P r2 All members of FACULTY can A SSIGN both I NTER - tant policies. For this example, we show only the mutants
NAL G RADES and E XTERNAL G RADES . produced by the Change Rule Effect (CRE) mutation oper-
ator [17]. The CRE mutation operator simply inverts each
P r3 There exists no combination of roles such that a user rule’s E FFECT by changing P ERMIT to D ENY, or D ENY to
with those roles can both R ECEIVE and A SSIGN the P ERMIT (one at a time for each mutant policy). The number
resource E XTERNAL G RADES. of mutant policies created by this operator is equal to the
number of rules in the policy. This operator should never consisting of each counterexample produced for each fail-
create equivalent mutants, which are mutant policies with ing property in the form of a concrete access request and an
the same behavior as the original policy, unless a rule is expected response. The quality of this test suite is directly
unreachable. The example policy has only two rules and dependent on the quality of the property set.
thus only two mutant policies are generated. Figures 2 and
3 show these two mutant policies for the first and second 3. Background
rules, respectively.
The second step of mutation verification is to determine
This section presents background information including
which properties hold for the original policy and each mu-
a description of XACML, policy mutation testing, and Mar-
tant policy. The mutant is said to be killed by a property if
grave, a policy verification tool used in our approach.
the property holds for the original policy but does not hold
for the mutant policy. In other words, the property reveals
3.1. XACML
the fault seeded in the mutant policy. Similar to mutation
testing [17, 21], the greater the number of mutants killed,
The eXtensible Access Control Markup Language
the more comprehensive the properties are in covering pol-
(XACML) is an XML-based syntax used to express policies,
icy behaviors, and thus the more effective the properties are
requests, and responses. This general-purpose language for
at interacting with the rules in the policy.
access control policies is an OASIS (Organization for the
The original policy (Figure 1) satisfies all three proper-
Advancement of Structured Information Standards) stan-
ties; therefore, if any property does not hold for a mutant
dard [1] that describes both a language for policies and a
policy, then that mutant policy is killed by the property.
language for requests or responses of access control deci-
The first mutant policy in Figure 2 does not satisfy P r2
sions. The policy language is used to describe general ac-
and thus the first mutant is killed. Recall P r2 seeks to en-
cess control requirements and is designed to be extended to
sure that all faculty members can assign grades. Since the
include new functions, data types, combining logic, etc.
fault in Figure 2 is precisely the rule that grants this access,
the property is apparently violated. The output of the prop-
erty verification is a list of specific access requests that vi-
3.2. Policy Mutation Testing
olate some property. The output from the property verifica-
tion on the first mutant policy yields two requests: a request Mutation testing [7] has historically been applied to
for a FACULTY to A SSIGN I NTERNAL G RADES and another general-purpose programming languages. The program un-
request for a FACULTY to A SSIGN E XTERNAL G RADES. der test is iteratively mutated to produce numerous mutants,
Access is denied for both requests, indicating a violation each containing one fault. A test input is independently ex-
of property P r2 . ecuted on the original program and each mutant program. If
The second mutant policy in Figure 3 is not killed by any the output of a test input executed on a mutant differs from
of the three properties, reflecting that the properties are not the output of the same test input executed on the original
comprehensive and do not completely “cover” the policy. program, then the seeded fault is detected and the mutant is
This realization leads to the elicitation of our fourth prop- said to be killed.
erty, which was not originally specified by Fisler et al. [9]: Policy mutation testing [17, 21] has been used to mea-
sure the fault-detection capability of a request set. In our
P r4 All members of S TUDENT can R ECEIVE E XTERNAL - previous work [17], we proposed a fault model for access
G RADES. control policies and defined a set of mutation operators that
implement that model with the goal of programmatically
Property P r4 fails to hold for the second mutant policy creating mutant policies in order to evaluate test generation
in Figure 3, thus killing the mutant, revealing its fault, and techniques and coverage criteria in terms of fault-detection
increasing the mutant-killing ratio. capability. Similarly, Traon et al. [21] adapt mutation anal-
In general, mutation verification serves two main pur- ysis and define mutation operators to quantify the effective-
poses: (1) to quantify how thoroughly a set of properties ness of a test set. Unfortunately, there are various expenses
interacts with or covers the policy behavior and (2) to fa- and barriers associated with mutation testing [17]. Primar-
cilitate property elicitation such that a property set interacts ily the generation and execution of a large number of mu-
with or covers all rules defined in the policy. In particular, tants on a large test set. Fortunately, policy mutation test-
the CRE mutation operator is useful in identifying specific ing is not as expensive as program mutation testing simply
rules that are not covered by the property set. The CRE because policy specification languages are far simpler than
mutation operator and other mutation operators together are general-purpose programming languages. Similarly and for
useful in quantifying the overall quality of the property set. the same reason, formal verification of policy specification
As a by-product of this process, a test suite is generated is less costly. This distinction is one of the primary reasons
that policy mutation verification is feasible. We use a vari-
ant of the policy mutation testing framework from our previ-
ous work [17] to facilitate the implementation of our policy
mutation verification approach presented in Section 4.
Mutation analysis has been applied to model-based test-
ing as well. Generally a model checker accepts a state-based
model and a property, and outputs a counterexample if that
property is not satisfied. The counterexample is essentially
a test input that can then be executed on the concrete imple-
mentation of the model. Specification mutation [5] is a way
to measure the effectiveness of a test input by mutating the
specification. The specification is a set of properties that de-
scribe how the model should behave. By mutating the spec-
ification (i.e., properties), one can determine the adequacy
of the model and its corresponding implementation. Con-
versely one can mutate the model [4, 10] to determine the Figure 4. Property verification.
adequacy of the specification. Our approach presented in
Section 4 is analogous to model mutation where the model
mutant is lengthy and semantically quite different from the
is the policy and the specification are the properties.
original one, their comparison is often costly.

4. Mutation Verification 4.2. Property Verification

This section presents our approach for policy mutation
Given a policy, a set of properties, and a set of mutant
verification to assess the quality of policy properties. We
policies, the next step is to determine which properties hold
next describe the details of each step in the approach: mu-
and which properties do not hold for both the original policy
tant generation, property verification, mutant-killing deter-
and each mutant policy as illustrated in Figure 4. We lever-
mination, and property generation.
age Margrave [9] to perform property verification. Mar-
grave represents XACML policies as multi-terminal binary
4.1. Mutant Generation decision diagrams (MTBDDs). MTBDDs are a type of deci-
sion diagram that maps bit vectors over a set of variables to
Given a policy, the first step is to generate a set of mu- a finite set of results. Margrave is implemented on top of the
tant policies. In our previous work [17], we presented a CUDD package [20], which provides an efficient implemen-
fault model for access control policies and a mutation test- tation of MTBDDs. Margrave can verify various properties
ing framework to investigate the fault model. The frame- (that are represented in Margrave’s specific format) against
work includes mutation operators used to implement the a given policy. Property P ri in Section 2 can be converted
fault model, mutant generation, equivalent-mutant detec- to Margrave’s specific format and verified whether P ri is
tion, and mutant-killing determination. Previously, we used satisfied by the policy.
mutation testing to measure the quality of a request set in
To perform property verification on a policy using Mar-
terms of fault-detection capability. In our new approach,
grave, a Scheme program is written that leverages the Mar-
we generate mutants not to measure the quality of a request
grave API. This program must load the policy, optionally
set, but to measure the quality of a set of properties used
specify any environment constraints, and define the set of
for property verification. We use the Change Rule Effect
properties that the policy must satisfy. In order to perform
(CRE) [17] mutation operator to help guide property elici-
property verification programmatically for each mutant pol-
tation to improve upon the existing property set.
icy, we generate an executable script and Scheme program
The inputs to this step are the policy under test and a
for both the original policy and each mutant policy. We
set of mutation operators. The mutator then generates a
specified and generated an EMF1 (Eclipse Modeling Frame-
set of mutant policies, each with a single fault. Among
work) model that encapsulates the necessary information
generated mutants, semantically equivalent mutants to the
to generate the executable scripts and Scheme programs.
original policy may exist. Such mutants and the original
Given an instance of this EMF model, we use the Model To
policy produce the same policy decisions for all possible
Text (M2T2 ) framework and a set of Java Emitter Templates
requests. We leverage a change-impact analysis tool such
as Margrave [9] to detect equivalent mutants by comparing 1 http://www.eclipse.org/modeling/emf/

the original policy and each mutant policy. Note that if a 2 http://www.eclipse.org/modeling/m2t/
(JET) to generate executable shell scripts and Scheme pro-
grams for each policy under test and its corresponding set Table 1. Policies used in the case-study.
Subject # Set # Policy # Rule # Property
of mutants. The generated scripts pipe a generated Scheme
program to a command-line Scheme interpreter. The out- C ONTINUE - A 111 266 298 9
C ONTINUE - B 111 266 306 9
put of the Scheme interpreter is then piped to a trace file for
S IMPLE -P OLICY 1 2 2 3
further processing. These trace files contain the information
necessary for determining which properties hold and which
properties do not hold for the original policy and each mu-
tant policy. “living” CRE mutant. Each CRE mutant corresponds to a
specific policy rule. When a given rule changes effect (e.g.,
deny to permit), the corresponding property P ri may de-
4.3. Mutant-Killing Determination
tect a semantic change and by definition P ri kills the CRE
mutant. To elicit P ri , we manually inspect the rule’s con-
The next step is to compute the mutant-killing ratio. The text. Let S, O, and A denote the set of subjects, objects,
mutant-killing ratio is the ratio of the number of mutants and actions in a rule, respectively. A rule is in the form
killed to the total number of mutants. This ratio serves as [S ∩O ∩A → Ef f ect] where the effect is either P ermit or
a metric to quantify the quality of a set of properties with Deny. If an effect is P ermit (Deny), we generate a prop-
respect to covering a given policy. A higher mutant-killing erty that a subject S can (cannot) take an action A on re-
ratio indicates the property set interacts with or covers a source O. For a mutant policy, when the effect is changed to
higher number of rules defined in the policy. Deny (P ermit), the property may detect a semantic fault
The trace files generated by the property verification de- in the mutated policy and cover the not-covered rule being
scribed earlier are parsed in order to divide the property set mutated.
into four subsets for each mutant. A property is either true
or false (i.e., the property is satisfied or is not satisfied) with
respect to a given policy. Let P denote the set of all proper- 5. Case Study
ties. Let the set of properties satisfied by the original policy
be denoted by OT and let the set of properties not satisfied
by the original policy be denoted OF . We have applied our mutation verification tool to an ac-
cess control policy for C ONTINUE [15]. C ONTINUE is a
P = OT ∪ OF (1) web-based conference management system that supports
the submission, review, discussion, and notification phases
Similarly, given a mutant policy M , let the set of properties of conferences. The C ONTINUE policy was used as a case
satisfied by the mutant policy be denoted by MT and the set study to explore property verification and change-impact
of properties not satisfied by the mutant policy be denoted analysis for Margrave by Fisler et al. [9]. The conference
MF and by Equation 2 all properties fall into one of these management system itself has been used to manage sev-
sets. eral conferences. Table 1 lists the policies used in our case
P = MT ∪ MF (2) study. Each row corresponds to a policy and Columns 2,
3, and 4 denote the number of P OLICY S ET, P OLICY, and
For our case study described in Section 5, all properties are RULE elements in each policy, respectively. Column 5 de-
intended to be satisfied by the original policy (i.e., holding notes the number of properties used for each policy. We
true). A property that holds true for both the original policy selected these policies in our case study because these poli-
and the mutant policy cannot expose the fault in the mutant cies are available with their formal properties. The S IM -
policy because the property does not apply to the portion of PLE policy was presented in Section 2 and C ONTINUE - A
the policy that contains the fault. On the other hand, if at and C ONTINUE - B are two versions of the C ONTINUE pol-
least one property holds true for the original policy but fails icy. All three policies and property sets are available at the
to hold true for the mutant policy as formalized in Equa- Margrave web site3 .
tion 3, then the mutant is killed.
We divide the case study into two parts: the first part per-
∃p ∈ P : p ∈ OT ∩ MF (3) forms mutation verification with the CRE mutation operator
for property assessment while the second part performs mu-
tation verification with several mutation operators and the
4.4. Property Generation
additional manually specified properties for property aug-
mentation.
To increase property coverage and investigate property
verification, we manually generate additional properties for 3 http://www.cs.brown.edu/research/plt/software/

a given policy. A property P ri is elicited by considering a margrave/

 Table 2. Mutant-killing ratios.
Subject mutant-kill ratio # mutants # killed
CONTINUE - A 24.16% 298 72
CONTINUE - B 24.84% 306 76
SIMPLE - POLICY 50.00% 2 1

5.1. Property Assessment

Table 2 shows the results of mutation verification using

only the CRE mutation operator, specifically the number
of mutants (Column 3), number of killed mutants (Column
4), and the mutant-killing ratio (Column 2) for each pol-
icy. As discussed in Section 2, the S IMPLE policy has only
two rules and thus two mutant policies are generated with
the CRE mutation operator. One mutant is killed so the Figure 6. C ONTINUE - B property failures for
mutant-killing ratio is simply 21 or 50%. The complexity each policy-property pair.
of the C ONTINUE policies makes them far more interest-
ing. Each version of C ONTINUE has approximately 300
rules and roughly 14 of the mutants generated from these More specifically, each integer value along the horizontal
rules are killed. This result shows that a single property can axis denotes a single property and each integer value along
cover multiple rules (which can be inferred by comparing the vertical axis denotes a single policy. Furthermore, the
the number of rules to the number of properties). policy at y = 0 is the original (un-mutated) policy. These
scatter plots allow us to quickly determine which properties
interact with which rules in the policy.
Property P r0 fails to hold for the original version of
C ONTINUE - A indicated by a data point at (P r0 , P0 ) in Fig-
ure 5. As a result, P r0 also fails to hold for any mutant poli-
cies as indicated by the numerous data points along x = 0.
If a property fails to hold for the original policy, then the
property is not expected to (and most likely will not) hold
for any mutant policies. Furthermore, this property is not
capable of killing mutants because in order for a mutant to
be killed by a property, the property must be satisfied by
the original policy as described in Section 4. The natural
language for this property is as follows:
P r0 If the subject is a pc-member, it is not the discussion
phase, and unsubmitted for the review for a paper de-
Figure 5. C ONTINUE - A property failures for spite being assigned it, then the subject cannot see all
each policy-property pair. parts of other’s reviews for that paper.
This property fails simply because C ONTINUE - A is an ear-
lier version of the policy. All properties including P r0 do
To further visualize and discuss the results, let each prop- hold for the revised version in Figure 6.
erty and each mutant be identified by an integer number. Another readily noticeable peculiarity of both Figures 5
For example, let the original policy be denoted P0 , each and 6 is the absence of P r8 . Recall that nine properties are
mutant policy be denoted P1 , P2 , . . . Pm , and each property verified against each policy, implying one property, P r8 ,
P r0 , P r1 , . . . P rp−1 where m and p are the number of mu- does not appear to interact with any rule explicitly defined in
tants and properties, respectively. A property-policy pair the policy. The natural language for the “missing” property
(P rx , Py ) is mapped to a point (x, y) in Figures 5 and 6. is:
A data point is plotted on the chart at (x, y) if the property
P rx fails to hold for Policy Py . Therefore, Figures 5 and 6 P r8 No legal request is mapped to Not Applicable, that is
illustrate all property failures for each property-policy pair. every legal request is decided by either deny or permit.
P r8 is an excellent example of a valid property that is not of data points. This property in natural language states that
explicitly specified in the policy itself. A policy should cer- if the subject role attribute is empty and the resource class is
tainly be written such that every legal request returns a deny not conference info, then return deny. This property effec-
or permit response. This property, however, is a generic tively identifies information leakage introduced through the
property potentially applicable to a wide range of policies. mechanism described earlier. This result indicates that this
Although the property is quite relevant, it is not (and ar- property set is effective at identifying information leakage
guably should not) be specified explicitly in the policy it- in the policy.
self. An argument against its inclusion in the policy itself On the other hand, the mutants that are not killed are
is that the property is generic; in particular, it is unrelated generally those that mutate a permitting rule to deny. For
to the access control logic of the system but is rather one example, when the rule that allows the admin to read
of the best practices. This type of generic property is not the pcMember-info-isChairFlag is switched from permit to
accounted for in this implementation of mutation verifica- deny, no property identifies the restricted access. Similar to
tion. Further investigation is needed to determine how to having general properties for ensuring that sensitive infor-
incorporate such properties. For instance, mutation opera- mation remains protected, one also wants to have properties
tors that consider not only the policy but also the properties for ensuring access is granted when appropriate. The fact
may account for these types of properties. that the un-killed mutants are generally of this type indicates
The un-mutated C ONTINUE - B (P0 in Figure 6) satisfies that the property set can be improved by adding properties
all properties as indicated by the lack of data points along for ensuring that access is granted when appropriate.
P0 = y = 0. Again, P r8 (i.e., x = 8) is not plotted be-
cause this generic property does not interact directly with 5.2. Property Augmentation
any rules specified in the policy. Properties P r5 and P r7 are
interesting because they fail for only a single rule for both
The low mutant-killing ratios in Section 5.1 imply that
C ONTINUE - A and C ONTINUE - B. The natural language for
the existing property set can be improved by augmenting
these properties are:
the existing property set. Motivated by these mutation veri-
P r5 If a subject is not a pc-chair or admin, then he/she may fication results, we manually generate properties to kill the
not set the meeting flag. “living” CRE mutants. We do so by creating a property that
mirrors a not-covered rule. Because each mutant created
P r7 If someone is not a pc-chair or admin, then he/she with the CRE mutation operator is associated with a single
can never see paper-review-rc for which he/she is con- rule in the policy, we can identify which rules in the policy
flicted. are not covered by the existing property set. After manu-
By manual inspection, we determine that the mutant ally constructing these properties, we then perform muta-
killed by P r5 is the same for both versions of the pol- tion verification using various mutation operators to com-
icy. The killed mutant corresponds to the last rule in the pare the quality of the existing property sets with the aug-
P OLICY S ET that specifies access to the meeting flag. More mented property sets (i.e., the existing property set plus the
specifically, once all permitted combinations of subjects and manually specified properties).
actions are specified, the final rule ensures all other requests Unfortunately, the size of the C ONTINUE policy makes
for the meeting flag are denied. Because the mutant policy manually specifying properties impractical. As a result, for
changed this rule’s decision to permit, the mutant was killed this case study, we investigate the first nine, commonly used
by P r5 . In a similar fashion, the killed mutant for P r7 is rules in the C ONTINUE - A and C ONTINUE - B policies. Fur-
identical for both versions and corresponds to precisely the thermore, because the difference between the C ONTINUE -
rule that ensures the denial of requests for paper reviews A and C ONTINUE - B policies is small and the rules that we
when the isConflicted flag is set. consider for manual property generation are identical, the
The C ONTINUE policy heavily uses the first-applicable results are the same for both policies. For brevity, we refer
combining algorithm. As a result, it is often the case that, to the results of both policies simply as C ONTINUE.
for a given resource, all permitting rules are specified first The manually generated properties are elicited by con-
followed by more general denying rules. When these types sidering the “living” CRE mutants since each CRE mutant
of denying rules are mutated to permit, the policy leaks corresponds to a single rule in the policy. Since the S IMPLE
sensitive information (i.e., access is granted when it should policy has only one “living” mutant, only one additional
not). A general property for ensuring that sensitive informa- property is specified. Furthermore, the property mirrors the
tion remains protected is effective at identifying these leaks. policy rule. For example, the property “A student can write
For example, properties P r1 in Figure 5 and P r3 in Figure 6 grades” is generated based on the policy rule “A student is
are in fact the same property. This property interacts with a permitted to write grades”.
large number of policy rules indicated by the large number Table 3 summarizes the number of generated mutants,
 Table 3. Policy mutation and mutants killed by property sets.
S IMPLE C ONTINUE
Mutants # mutants # kill kill % # new kill new kill % # mutants # kill kill % # new kill new kill %
PSTT 0 0 0 0 0 2 2 100 2 100
PSTF 1 1 100 1 100 6 3 50 3 50
PTT 2 2 100 2 100 7 3 43 7 100
PTF 2 2 100 2 100 9 3 33 3 33
RTT 2 2 100 2 100 0 0 0 0 0
RTF 2 2 100 2 100 9 0 0 0 0
CPC 0 0 0 0 0 7 0 0 2 17
CRC 0 0 0 0 0 0 0 0 0 0
CRE 2 1 50 2 100 9 1 11 7 78
Total 11 10 90.91 11 100 49 12 24.49 24 48.98

the number of killed mutants, and the mutant-killing ratio augmented property set Pnew increases mutant-killing ra-
for each policy for both the existing property set P and the tios only in the CRE mutants since the remaining mutants
augmented property set Pnew . Each row of the table cor- cannot be killed. For the C ONTINUE policy, we manually
responds to a particular mutation operator and the column specify six properties and the augmented property set Pnew
groups 1 and 2 correspond to the S IMPLE policy and the kills 78% of the CRE mutants. The two not-covered rules
C ONTINUE policy, respectively. (i.e., “living” CRE mutants) cannot be killed. Pnew also
In the case study, we do not use some mutation operators increases the mutant-killing ratios for other mutation oper-
if they result in equivalent mutants for the given policy or ators.
if the property verification tool cannot handle a particular We observed that some types of mutants cannot be killed
XACML feature. For example, we omit RCT (Rule Condi- with Pnew . For example, PSTT, CPC, and CRC mutants
tion True) and RCF (Rule Condition False) operations since and RTT, RTF, and CRC mutants are not killed. These mu-
the policies do not use that feature of rule conditions due tants may not be semantically equivalent to the original pol-
to a limitation in the current Margrave. In addition, Mar- icy. However, the property set Pnew is not sufficient to kill
grave sometimes reports errors during property verification these mutants.
of some mutants. For instance, some properties require that
certain elements exist in a given policy. Mutation opera- 6. Discussion
tions such as PSTT (Policy Set Target True) may remove
elements that are necessary in order to verify a particular We believe that our approach can be applied to assess
property. These errors correctly indicate semantic faults in the quality of a property set against policies written in lan-
mutant policies and so we consider them killed. guages other than XACML. Previous approaches converted
The PTT (Policy Target True) and PTF (Policy Target policies in one language (such as XACML) to other lan-
False) operators delete or modify the top-level policy ele- guages (such as Alloy [12], RW [23], and Description Log-
ment in the S IMPLE policy effectively; these operators re- ics [14]) that are equipped with verification tools. As our
move one of the two rules resulting in drastic semantic dif- approach requires property verification (against a policy and
ferences that are immediately detected by the both P and its mutants) provided by these verification tools, such con-
Pnew . On the other hand, the CPC (Change Policy Combin- version enables our approach to also be applicable to poli-
ing Algorithm) and CRC (Change Rule Combining Algo- cies languages other than XACML and with verification
rithm) mutation operators generate equivalent mutants (i.e., tools other than Margrave.
mutants that are semantically equivalent to the original pol- Our approach to mutation verification provides a qual-
icy) that cannot be killed. For the C ONTINUE policy, the ity assessment of a property set for a policy. If a property
CRC generates equivalent mutants. Such semantic equiv- set achieves a mutant-killing ratio of 100%, can we say that
alent mutants are detected and not considered in our case the property set is exhaustive or complete? This situation is
study. similar to statement coverage in software testing. If a test
Table 3 shows that the existing property set P can kill suite achieves 100% statement coverage for a given pro-
50% and 11% of CRE mutants in S IMPLE and C ONTINUE gram, can we say the test suite can detect all faults in the
policies, respectively. By manually specifying a property program? The answer, of course, is absolutely not. While
that mirrors the not-covered rule for the S IMPLE policy, mutation verification serves as a quality assessment for a
we can kill all CRE mutant policies. As expected, the property set and, with the CRE mutation operator, identifies
which properties interact with which rules in the policy, it oped the FIREMAN tool to detect misconfiguration of fire-
may not consider more abstract, generic properties. For ex- wall policies.
ample, P r1 of the illustrative example in Section 2 ensures There are also several verification tools available for
that a student cannot assign grades. While this property is XACML policies [1]. Hughes and Bultan [11] translated
an intuitive one of the problem domain, it is not explicitly XACML policies to the Alloy language [12], and checked
expressed in the policy itself. This particular policy con- their properties using the Alloy Analyzer. Schaad and Mof-
tains only rules that allow access whereas this property is fett [19] also leverage Alloy to check that role-based access
concerned with denying access. The fact that this property control policies do not allow roles to be assigned to users
does not interact with the rules in the policy does not im- in ways that violate separation-of-duty constraints. Zhang
ply that it is not needed. A better example is discussed in et al. [24] developed a model-checking algorithm and tool
Section 5 where a property serves as more of the best prac- support to evaluate access control policies written in RW
tice that is not related to the problem domain of the access languages, which can be converted to XACML [23]. Ko-
control. laczek [13] proposes to translate role-based access control
Future work shall investigate a means of automatically policies into Prolog for verification. Kolovski et al. [14] for-
generating various types of properties to cover more rules malize XACML policies with description logics (DL), which
and entities in an access control policy. In our case study, are a decidable fragment of the first-order logic, and exploit
we manually generated properties to cover not-covered existing DL verifiers to conduct policy verification. Fisler
rules based on the mutation verification results for the CRE et al. [9] developed Margrave, which can verify XACML
mutation operator. As we extracted these properties from policies against properties, if properties are specified, and
(explicitly expressed) not-covered rules, each of the proper- perform change-impact analysis on two versions of poli-
ties is specifically effective to kill the (previously un-killed) cies when properties are not specified. When Margrave de-
rule. But these properties may not kill other mutants. As tects property violations during policy verification, it auto-
existing properties often describe more general behaviors matically generates concrete counterexamples in the form
of a policy, further exploration of mutation operators for of specific requests that illustrate violations of the speci-
mutation verification is needed to investigate how to reflect fied properties. Similarly, when Margrave detects semantic
relevant properties (that are not necessarily specified in the differences during change-impact analysis, it automatically
policy itself) in the mutation verification process. generates specific requests that reveal semantic differences
between two versions of a policy. Most of these approaches
7. Related Work require user-specified properties to be verified. Our new ap-
proach complements these existing policy verification ap-
To the best of our knowledge, no metric has been de- proaches because our approach helps assess the quality of
fined to quantify the coverage of a policy or model by some the properties during policy verification.
property set. Our related previous approach on policy mu- Our previous work [16] proposed an approach to policy
tation testing [17] defined a fault model and corresponding property inference via machine learning. Such properties
automated mutator in order to quickly assess the quality of are often not available in practice and their elicitation is a
a test suite; the assessment results can be further used to challenging and tedious task. Furthermore, once proper-
assess test-generation and test-selection techniques in terms ties are defined, it is difficult to assess their effectiveness
of fault-detection capability. Such policy mutation testing is and identify potential problematic areas that need improve-
related to the approach proposed by Ammann et al. [3] that ment. Our mutation verification approach intends to help al-
mutates a model (corresponding to a policy in our work) leviate that challenge. Our implementation leverages Mar-
and then uses the model mutants to assess the quality of a grave’s property verification feature to verify mutant poli-
test suite. Our new approach leverages a variation of an au- cies against properties.
tomated mutator [17] in our implementation of the mutation Although various coverage criteria [25] for software pro-
verification approach. However, different from these previ- grams exist, only recently have coverage criteria for access
ous approaches on assessing the quality of a test suite, our control policies been proposed [18]. Policy coverage crite-
new approach focuses on assessing the quality of a property ria are needed to measure how well policies are tested and
set based on mutating a policy. which parts of the policies are not covered by the existing
To help ensure the correctness of policy specifications, test inputs. Martin et al. [18] defined policy coverage and
researchers and practitioners have developed formal verifi- developed a policy coverage measurement tool. Because it
cation tools for policies. Several policy verification tools is tedious for developers to manually generate test inputs
are developed specifically for firewall policies. Al-Shaer for policies, and manually generated test inputs are often
and Hamed [2] developed the Firewall Policy Advisor to not sufficient for achieving high policy coverage, they de-
classify and detect policy anomalies. Yuan et al. [22] devel- veloped several test generation techniques. Different from
these policy testing approaches, our new approach focuses [7] R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test
on assessing the quality of properties in policy verification. data selection: Help for the practicing programmer. IEEE
Computer, 11(4):34–41, April 1978.
[8] D. F. Ferraiolo, D. R. Kuhn, and R. Chandramouli. Role-
8. Conclusion Based Access Control. Artech House, Inc., 2003.
[9] K. Fisler, S. Krishnamurthi, L. A. Meyerovich, and M. C.
The need for carefully controlling access to sensitive Tschantz. Verification and change-impact analysis of
information is increasing as the amount and availability access-control policies. In Proc. ICSE, pages 196–205,
of data are growing. In order to separate the semantics 2005.
of access control from the system itself, policy authors [10] G. Fraser and F. Wotawa. Using model-checkers for
mutation-based test-case generation, coverage analysis and
increasingly specify access control policies in declarative
specification analysis. In Proc. ICSEA, pages 16–21, 2006.
languages such as XACML. Doing so facilitates managing, [11] G. Hughes and T. Bultan. Automated verification of access
maintaining, and analyzing policies. To increase confidence control policies. Technical Report 2004-22, Department of
in the correctness of specified policies, policy authors can Computer Science, University of California, Santa Barbara,
formally verify policies against a property set. Policy veri- 2004.
fication is an important technique for high assurance of the [12] D. Jackson, I. Shlyakhter, and M. Sridharan. A micromodu-
correct specification of access control policies. Since the ef- larity mechanism. In Proc. ESEC/FSE, pages 62–73, 2001.
fectiveness of the verification process is directly dependent [13] G. Kolaczek. Specification and verification of constraints in
on the quality of the properties, we have proposed a novel role based access control for enterprise security system. In
approach called mutation verification to assess the quality Proc. WETICE, pages 190–195, 2003.
[14] V. Kolovski, J. Hendler, and B. Parsia. Analyzing web ac-
of a property set in verification of access control policies.
cess control policies. In Proc. WWW, pages 677–686, 2007.
We have implemented a tool for the approach being applied [15] S. Krishnamurthi. The CONTINUE server (or, how i admin-
on XACML policies. We applied our mutation verification istered PADL 2002 and 2003). In Proc. PADL, pages 2–16,
tool to policies and properties from a real-world software 2003.
system. Our experiences show that the performance of the [16] E. Martin and T. Xie. Inferring access-control policy proper-
property verification is encouraging and mutation verifica- ties via machine learning. In Proc. POLICY, pages 235–238,
tion can scale to sufficiently large access control policies. 2006.
Furthermore, mutation verification is a complementary ap- [17] E. Martin and T. Xie. A fault model and mutation testing
proach to property verification by aiding in the elicitation of of access control policies. In Proc. WWW, pages 667–676,
2007.
properties.
[18] E. Martin, T. Xie, and T. Yu. Defining and measuring policy
coverage in testing access control policies. In Proc. ICICS,
Acknowledgment pages 139–158, 2006.
[19] A. Schaad and J. D. Moffett. A lightweight approach to
specification and analysis of role-based access control ex-
This work is supported in part by NSF grant CNS-
tensions. In Proc. SACMAT, pages 13–22, 2002.
0716579 and its NIST supplement. [20] F. Somenzi. CUDD: CU Decision Diagram Package.
http://vlsi.colorado.edu/˜fabio/CUDD/.
References [21] Y. L. Traon, T. Mouelhi, and B. Baudry. Testing security
policies: Going beyond functional testing. In Proc. ISSRE,
[1] OASIS eXtensible Access Control Markup Language pages 93–102, 2007.
(XACML). http://www.oasis-open.org/ [22] L. Yuan, J. Mai, Z. Su, H. Chen, C.-N. Chuah, and P. Mo-
committees/xacml/, 2005. hapatra. FIREMAN: A toolkit for FIREwall Modeling and
[2] E. Al-Shaer and H. Hamed. Discovery of policy anomalies ANalysis. In Proc. S&P, pages 199–213, May 2006.
in distributed firewalls. In Proc. INFOCOM, pages 2605– [23] N. Zhang, M. Ryan, and D. P. Guelev. Synthesising verified
2616, 2004. access control systems in XACML. In Proc. FMSE, pages
[3] P. Ammann and P. E. Black. A specification-based coverage 56–65, 2004.
metric to evaluate test sets. In Proc. HASE, pages 239–248, [24] N. Zhang, M. Ryan, and D. P. Guelev. Evaluating access
1999. control policies through model checking. In Proc. InfoSec,
[4] P. E. Ammann, P. E. Black, and W. Majurski. Using model pages 446–460, 2005.
checking to generate tests from specifications. In Proc. [25] H. Zhu, P. A. V. Hall, and J. H. R. May. Software unit test
ICFEM, pages 46–54, 1998. coverage and adequacy. ACM Comput. Surv., 29(4):366–
[5] T. A. Budd and A. S. Gopal. Program testing by specifica- 427, 1997.
tion mutation. Computer Languages, 10(1):63–73, 1985.
[6] N. Damianou, N. Dulay, E. Lupu, and M. Sloman. The Pon-
der policy specification language. In Proc. POLICY, pages
18–38, 2001.