2016 IEEE Symposium on Security and Privacy

Algorithmic Transparency via

Quantitative Input Influence:
Theory and Experiments with Learning Systems

Anupam Datta Shayak Sen Yair Zick

Carnegie Mellon University, Pittsburgh, USA
{danupam, shayaks, yairzick}@cmu.edu

Abstract—Algorithmic systems that employ machine learning The call for algorithmic transparency has grown in in-
play an increasing role in making substantive decisions in modern tensity as public and private sector organizations increas-
society, ranging from online personalization to insurance and ingly use large volumes of personal information and complex
credit decisions to predictive policing. But their decision-making
processes are often opaque—it is difficult to explain why a certain data analytics systems for decision-making [6]. Algorithmic
decision was made. We develop a formal foundation to improve transparency provides several benefits. First, it is essential
the transparency of such decision-making systems. Specifically, to enable identification of harms, such as discrimination,
we introduce a family of Quantitative Input Influence (QII) introduced by algorithmic decision-making (e.g., high interest
measures that capture the degree of influence of inputs on outputs credit cards targeted to protected groups) and to hold entities
of systems. These measures provide a foundation for the design
of transparency reports that accompany system decisions (e.g., in the decision-making chain accountable for such practices.
explaining a specific credit decision) and for testing tools useful This form of accountability can incentivize entities to adopt
for internal and external oversight (e.g., to detect algorithmic appropriate corrective measures. Second, transparency can
discrimination). help detect errors in input data which resulted in an adverse
Distinctively, our causal QII measures carefully account for decision (e.g., incorrect information in a user’s profile because
correlated inputs while measuring influence. They support a
of which insurance or credit was denied). Such errors can then
general class of transparency queries and can, in particular,
explain decisions about individuals (e.g., a loan decision) and be corrected. Third, by explaining why an adverse decision
groups (e.g., disparate impact based on gender). Finally, since was made, it can provide guidance on how to reverse it (e.g.,
single inputs may not always have high influence, the QII by identifying a specific factor in the credit profile that needs
measures also quantify the joint influence of a set of inputs to be improved).
(e.g., age and income) on outcomes (e.g. loan decisions) and the
marginal influence of individual inputs within such a set (e.g.,
income). Since a single input may be part of multiple influential Our Goal. While the importance of algorithmic transparency
sets, the average marginal influence of the input is computed is recognized, work on computational foundations for this
using principled aggregation measures, such as the Shapley value, research area has been limited. This paper initiates progress
previously applied to measure influence in voting. Further, since in that direction by focusing on a concrete algorithmic trans-
transparency reports could compromise privacy, we explore the
parency question:
transparency-privacy tradeoff and prove that a number of useful
transparency reports can be made differentially private with very
little addition of noise. How can we measure the influence of inputs (or features) on
Our empirical validation with standard machine learning algo- decisions made by an algorithmic system about individuals or
rithms demonstrates that QII measures are a useful transparency groups of individuals?
mechanism when black box access to the learning system is
available. In particular, they provide better explanations than
standard associative measures for a host of scenarios that we Our goal is to inform the design of transparency reports,
consider. Further, we show that in the situations we consider, which include answers to transparency queries of this form.
QII is efficiently approximable and can be made differentially To be concrete, let us consider a predictive policing system
private while preserving accuracy. that forecasts future criminal activity based on historical data;
individuals high on the list receive visits from the police.
I. I NTRODUCTION An individual who receives a visit from the police may seek
a transparency report that provides answers to personalized
Algorithmic decision-making systems that employ machine transparency queries about the influence of various inputs
learning and related statistical methods are ubiquitous. They (or features), such as race or recent criminal history, on the
drive decisions in sectors as diverse as Web services, health- system’s decision. An oversight agency or the public may
care, education, insurance, law enforcement and defense [1], desire a transparency report that provides answers to aggregate
[2], [3], [4], [5]. Yet their decision-making processes are often transparency queries, such as the influence of sensitive inputs
opaque. Algorithmic transparency is an emerging research area (e.g., gender, race) on the system’s decisions concerning the
aimed at explaining decisions made by algorithmic systems. entire population or about systematic differences in decisions

© 2016, Anupam
2375-1207/16 $31.00
Datta.
© Under
2016 IEEE
license to IEEE. 598
DOI 10.1109/SP.2016.42
among groups of individuals (e.g., discrimination based on transparency about the relative importance of individual inputs
race or age). These reports can thus help identify harms and within the set (e.g., age vs. income) in the system’s decision.
errors in input data, and provide guidance on what input We achieve the first desideratum by formalizing a notion
features to work on to modify the decision. of a quantity of interest. A transparency query measures the
influence of an input on a quantity of interest. A quantity of
Our Model. We focus on a setting where a transparency interest represents a property of the behavior of the system for
report is generated with black-box access to the decision- a given input distribution. Our formalization supports a wide
making system1 and knowledge of the input dataset on which range of statistical properties including probabilities of various
it operates. This setting models the kind of access available outcomes in the output distribution and probabilities of output
to a private or public sector entity that pro-actively publishes distribution outcomes conditioned on input distribution events.
transparency reports. It also models a useful level of access Examples of quantities of interest include the conditional
required for internal or external oversight of such systems probability of an outcome for a particular individual or group,
to identify harms introduced by them. For the former use and the ratio of conditional probabilities for an outcome for
case, our approach provides a basis for design of transparency two different groups (a metric used as evidence of disparate
mechanisms; for the latter, it provides a formal basis for impact under discrimination law in the US [7]).
testing. Returning to our predictive policing system, the law We achieve the second desideratum by formalizing causal
enforcement agency that employs it could proactively publish QII measures. These measures (called Unary QII) model the
transparency reports, and test the system for early detection difference in the quantity of interest when the system operates
of harms like race-based discrimination. An oversight agency over two related input distributions—the real distribution and a
could also use transparency reports for post hoc identification hypothetical (or counterfactual) distribution that is constructed
of harms. from the real distribution in a specific way to account for
correlations among inputs. Specifically, if we are interested in
Our Approach. We formalize transparency reports by introduc-
measuring the influence of an input on a quantity of interest of
ing a family of Quantitative Input Influence (QII) measures
the system behavior, we construct the hypothetical distribution
that capture the degree of influence of inputs on outputs of
by retaining the marginal distribution over all other inputs and
the system. Three desiderata drove the definitions of these
sampling the input of interest from its prior distribution. This
measures.
choice breaks the correlations between this input and all other
First, we seek a formalization of a general class of
inputs and thus lets us measure the influence of this input
transparency reports that allows us to answer many useful
on the quantity of interest, independently of other correlated
transparency queries related to input influence, including but
inputs. Revisiting our moving company hiring example, if the
not limited to the example forms described above about the
system makes decisions only using the weightlifting ability of
system’s decisions about individuals and groups.
applicants, the influence of gender will be zero on the ratio of
Second, we seek input influence measures that appropriately
conditional probabilities of being hired for males and females.
account for correlated inputs—a common case for our target
We achieve the third desideratum in two steps. First, we
applications. For example, consider a system that assists in
define a notion of joint influence of a set of inputs (called
hiring decisions for a moving company. Gender and the
Set QII) via a natural generalization of the definition of the
ability to lift heavy weights are inputs to the system. They
hypothetical distribution in the Unary QII definition. Second,
are positively correlated with each other and with the hiring
we define a family of Marginal QII measures that model the
decisions. Yet transparency into whether the system uses the
difference on the quantity of interest as we consider sets with
weight lifting ability or the gender in making its decisions (and
and without the specific input whose marginal influence we
to what degree) has substantive implications for determining if
want to measure. Depending on the application, we may pick
it is engaging in discrimination (the business necessity defense
these sets in different ways, thus motivating several different
could apply in the former case [7]). This observation makes
measures. For example, we could fix a set of inputs and ask
us look beyond correlation coefficients and other associative
about the marginal influence of any given input in that set on
measures.
the quantity of interest. Alternatively, we may be interested in
Third, we seek measures that appropriately quantify input the average marginal influence of an input when it belongs
influence in settings where any input by itself does not have to one of several different sets that significantly affect the
significant influence on outcomes but a set of inputs does. quantity of interest. We consider several marginal influence
In such cases, we seek measures of joint influence of a set aggregation measures from cooperative game theory originally
of inputs (e.g., age and income) on a system’s decision (e.g., developed in the context of influence measurement in voting
to serve a high-paying job ad). We also seek measures of scenarios and discuss their applicability in our setting. We
marginal influence of an input within such a set (e.g., age) also build on that literature to present an efficient approximate
on the decision. This notion allows us to provide finer-grained algorithm for computing these measures.
1 By “black-box access to the decision-making system” we mean a typical
Recognizing that different forms of transparency reports
setting of software testing with complete control of inputs to the system and may be appropriate for different settings, we generalize our QII
full observability of the outputs. measures to be parametric in its key elements: the intervention

599
used to construct the hypothetical input distribution; the quan- strongly correlated with gender (with men having better overall
tity of interest; the difference measure used to quantify the lifting ability than woman). One particular question that an
distance in the quantity of interest when the system operates analyst may want to ask is: “What is the influence of the input
over the real and hypothetical input distributions; and the Gender on positive classification for women?”. The analyst
aggregation measure used to combine marginal QII measures observes that 20% of women are approved according to his
across different sets. This generalized definition provides a classifier. Then, he replaces every woman’s field for gender
structure for exploring the design space of transparency re- with a random value, and notices that the number of women
ports. approved does not change. In other words, an intervention on
Since transparency reports released to an individual, reg- the Gender variable does not cause a significant change in
ulatory agency, or the public might compromise individual the classification outcome. Repeating this process with Weight
privacy, we explore the possibility of answering transparency Lifting Ability results in a 20% increase in women’s hiring.
queries while protecting differential privacy [8]. We prove Therefore, he concludes that for this classifier, Weight Lifting
bounds on the sensitivity of a number of transparency queries Ability has more influence on positive classification for women
and leverage prior results on privacy amplification via sam- than Gender.
pling [9] to accurately answer these queries. By breaking correlations between gender and weight lifting
We demonstrate the utility of the QII framework by de- ability, we are able to establish a causal relationship between
veloping two machine learning applications on real datasets: the outcome of the classifier and the inputs. We are able to
an income classification application based on the benchmark identify that despite the strong correlation between a negative
adult dataset [10], and a predictive policing application classification outcome for women, the feature gender was not
based on the National Longitudinal Survey of Youth [11]. a cause of this outcome. We formalize the intuition behind
Using these applications, we argue, in Section VII, the need such causal experimentation in our definition of Quantitative
for causal measurement by empirically demonstrating that Input Influence (QII).
in the presence of correlated inputs, observational measures We are given an algorithm A. A operates on inputs (also
are not informative in identifying input influence. Further, referred to as features for ML systems), N = {1, . . . , n}.
we analyze transparency reports of individuals in our dataset Everyi ∈ N , can take on various states, given by Xi . We let
to demonstrate how Marginal QII can provide insights into X = i∈N Xi be the set of possible feature state vectors, let
individuals’ classification outcomes. Finally, we demonstrate Z be the set of possible outputs of A. For a vector x ∈ X
that under most circumstances, QII measures can be made and set of inputs S ⊆ N , x|S denotes the vector of inputs in
differentially private with minimal addition of noise, and can S. We are also given a probability distribution π on X , where
be approximated efficiently. π(x) is the probability of the input vector x. We can define a
In summary, this paper makes the following contributions: marginal probability of a set of inputs S in the standard way
• A formalization of a specific algorithmic transparency as follows:

problem for decision-making systems. Specifically, we πS (x|S ) = π(x ) (1)
define a family of Quantitative Input Influence metrics {x ∈X |x |S =x|S }
that accounts for correlated inputs, and provides answers
to a general class of transparency queries, including the When S is a singleton set {i}, we write the marginal
absolute and marginal influence of inputs on various probability of the single input as πi (x).
behavioral system properties. These metrics can inform Informally, to quantify the influence of an input i, we
the design of transparency mechanisms and guide pro- compute its effect on some quantity of interest; that is, we
active system testing and posthoc investigations. measure the difference in the quantity of interest, when the
• A formal treatment of privacy-transparency trade-offs, feature i is changed via an intervention. In the example above,
in particular, by construction of differentially private the quantity of interest is the fraction of positive classification
answers to transparency queries. of women. In this paper, we employ a particular interpretation
• An implementation and experimental evaluation of the of “changing an input”, where we replace the value of every
metrics over two real data sets. The evaluation demon- input with a random independently chosen value. To describe
strates that (a) the QII measures are informative; (b) they the replacement operation for input i, we first define an
remain accurate while preserving differential privacy; and expanded probability space on X × X , with the following
(c) can be computed quite quickly for standard machine distribution:
learning systems applied to real data sets.
II. U NARY QII π̃(x, u) = π(x)π(u). (2)
Consider the situation discussed in the introduction, where The first component of an expanded vector (x, u), is just
an automated system assists in hiring decisions for a moving the original input vector, whereas the second component repre-
company. The input features used by this classification system sents an independent random vector drawn from the same dis-
are : Age, Gender, Weight Lifting Ability, Marital Status and tribution π. Over this expanded probability space, the random
Education. Suppose that, as before, weight lifting ability is variable X(x, ui ) = x represents the original feature vector.

600
The random variable X−i Ui (x, u) = x|N \{i} ui , represents the In the example above, for a classifier A, the quantity of
random variable with input i replaced with a random sample. interest, the fraction of women (represented by the set W ⊆
Defining this expanded probability space allows us to switch X ) with positive classification, can be expressed as follows:
between the original distribution, represented by the random
variable X, and the intervened distribution, represented by
QA (·) = E(A(·) = 1 | X ∈ W),
X−i Ui . Notice that both these random variables are defined
from X ×X , the expanded probability space, to X . We denote
the set of random variables of the type X × X → X as R(X ). and the influence of input i is:
We can now define probabilities over this expanded space.
For example, the probability over X remains the same:
ι(i) = E(A(X) = 1 | X ∈ W)−E(A(X−i Ui ) = 1 | X ∈ W).

Pr(X = x) = π̃(x , u ) When A is clear from the context, we simply write Q rather
{(x ,u )|x =x} than QA . We now instantiate this definition with different
⎛ ⎞ quantities of interest to illustrate the above definition in three
 
=⎝ π(x )⎠ 
π(u ) different scenarios.
{x |x =x} u

= π(x) A. QII for Individual Outcomes

Similarly, we can define more complex quantities. The One intended use of QII is to provide personalized trans-
following expression represents the expectation of a classifier parency reports to users of data analytics systems. For exam-
c evaluating to 1, when i is randomly intervened on: ple, if a person is denied a job application due to feedback
from a machine learning algorithm, an explanation of which
 factors were most influential for that person’s classification
E(c(X−i Ui ) = 1) = π̃(x, ui ). can provide valuable insight into the classification outcome.
{(x,u)|c(xN \i ui )=1} For QII to quantify the use of an input for individual
outcomes, we define the quantity of interest to be the classifi-
cation outcome for a particular individual. Given a particular
Observe that the expression above computes the probability individual x, we define Qxind (·) to be E(c(·) = 1 | X = x).
of the classifier c evaluating to 1, when input i is replaced The influence measure is therefore:
with a random sample from its probability distribution πi (ui ).

 ιxind (i) = E(c(X) = 1 | X = x) − E(c(X−i Ui ) = 1 | X = x)

π̃(x, ui ) (3)
{(x,u)|c(xN \i ui )=1}
   When the quantity of interest is not the probability of
= π(x) π(u) positive classification but the classification that x actually
x {ui |c(xN \i ui )=1} {u|ui =ui } received, a slight modification of the above QII measure is
  more appropriate:
= π(x) πi (ui )
x {ui |c(xN \i ui )=1}

We can also define conditional distributions in the usual ιxind-act (i) = E(c(X) = c(x) | X = x)
way. The following represents the probability of the classifier −E(c(X−i Ui ) = c(x) | X = x)
(4)
evaluating to 1 under the randomized intervention on input i = 1 − E(c(X−i Ui ) = c(x) | X = x)
of X, given that X belongs to some subset Y ⊆ X : = E(c(X−i Ui ) = c(x) | X = x)

The above probability can be interpreted as the probability

E(c(X−i Ui ) = 1 ∧ X ∈ Y) that feature i is pivotal to the classification of c(x). Computing
E(c(X−i Ui ) = 1 | X ∈ Y) = .
E(X ∈ Y) the average of this quantity over X yields:
Formally, for an algorithm A, a quantity of interest QA (·) :
R(X ) → R is a function of a random variable from R(X ).

x∈X Pr(X = x)E(i is pivotal for c(X) | X = x)

Definition 1 (QII). For a quantity of interest QA (·), and an = E(i is pivotal for c(X)).
input i, the Quantitative Input Influence of i on QA (·) is (5)
defined to be We denote this average QII for individual outcomes as
defined above, by ιind-avg (i), and use it as a measure for
ιQA (i) = QA (X) − QA (X−i Ui ). importance of an input towards classification outcomes.

601
B. QII for Group Outcomes III. S ET AND M ARGINAL QII
As in the running example, the quantity of interest may In many situations, intervention on a single input variable
be the classification outcome for a set of individuals. Given a has no influence on the outcome of a system. Consider, for
group of individuals Y ⊆ X , we define QY grp (·) to be E(c(·) = example, a two-feature setting where features are age (A) and
1 | X ∈ Y). The influence measure is therefore: income (I), and the classifier is c(A, I) = (A = old ) ∧ (I =
high). In other words, the only datapoints that are labeled 1
are those of elderly persons with high income. Now, given
ιY
grp (i) = E(c(X) = 1 | X ∈ Y) − E(c(X−i Ui ) = 1 | X ∈ Y) a datapoint where A = young, I = low , an intervention on
(6) either age or income would result in the same classification.
However, it would be misleading to say that neither age nor
C. QII for Group Disparity
income have an influence over the outcome: changing both the
Instead of simply classification outcomes, an analyst may states of income and age would result in a change in outcome.
be interested in more nuanced properties of data analytics Equating influence with the individual ability to affect the
systems. Recently, disparate impact has come to the fore as a outcome is uninformative in real datasets as well: Figure 1 is a
measure of unfairness, which compares the rates of positive histogram of influences of features on outcomes of individuals
classification within protected groups defined by gender or for a classifier learnt from the adult dataset [13]2 . For most
race. The ‘80% rule’ in employment which states that the individuals, all features have zero influence: changing the state
rate of selection within a protected demographic should be of one feature alone is not likely to change the outcome of
at least 80% of the rate of selection within the unprotected a classifier. Of the 19537 datapoints we evaluate, more than
demographic. The quantity of interest in such a scenario is half have ιx (i) = 0 for all i ∈ N , Indeed, changes to outcome
the ratio in positive classification outcomes for a protected are more likely to occur if we intervene on sets of features.
group Y from the rest of the population X \ Y. In order to get a better understanding of the influence of a
feature i ∈ N , we should measure its effect when coupled
E(c(X) = 1 | X ∈ Y) with interventions on other features. We define the influence
of a set of inputs as a straightforward extension of the influence
E(c(X) = 1 | X ∈ Y)
of individual inputs. Essentially, we wish the influence of a set
However, the ratio of classification rates is unstable at at of inputs S ⊆ N to be the same as when the set of inputs is
low values of positive classification. Therefore, for the com- considered to be a single input; when intervening on S, we
putations in this paper we use the difference in classification draw the states of i ∈ S based on the joint distribution of the
rates as our measure of group disparity. states of features in S, πS (uS ), as defined in Equation (1).
We can naturally define a distribution over X × i∈S Xi ,
naturally extending (2) as:
QY
disp (·) = |E(c(·) = 1 | X ∈ Y) − E(c(·) = 1 | X ∈ Y)|
(7)
π̃(x, uS ) = π(x)πS (uS ). (9)
The QII measure of an input group disparity, as a result is:
We also define the random variable X−S US (x, uS ) =
ιY Y Y
disp (i) = Qdisp (X) − Qdisp (X−i Ui ). (8) x|N \S uS ; X−S (x, uS ) has the states of features in N \ S
fixed to their original values in x, but features in S take on
More generally, group disparity can be viewed as an as- new values according to uS .
sociation between classification outcomes and membership
in a group. QII on a measure of such association (e.g., Definition 2 (Set QII). For a quantity of interest Q, and an
group disparity) identifies the variable that causes the associ- input i, the Quantitative Input Influence of set S ⊆ N on Q
ation in the classifier. Proxy variables are variables that are is defined to be
associated with protected attributes. However, for concerns ιQ (S) = Q(X) − Q(X−S US ).
of discrimination such as digital redlining, it is important
to identify which proxy variables actually introduce group Considering the influence of a set of inputs opens up a
disparity. It is straightforward to observe that features with number of interesting questions due to the interaction between
high QII for group disparity are proxy variables, and also cause inputs. First among these is how does one measure the
group disparity. Therefore, QII on group disparity is a useful individual effect of a feature, given the measured effects of
diagnostic tool for determining discriminiation. The use of QII interventions on sets of features. One natural way of doing so
in identifying proxy variables is explored experimentally in is by measuring the marginal effect of a feature on a set.
Section VII-B. Note that because of such proxy variables,
simply ensuring that protected attributes are not input to 2 The adult dataset contains approximately 31k datapoints of users’ personal
the classifier is not sufficient to avoid discrimination (see attributes, and whether their income is more than $50k per annum; see
also [12]). Section VII for more details.

602
 10000 aggregation methods, which, given an influence measure v :
2N → R, output a vector φ ∈ Rn , whose i-th coordinate
8000 corresponds in some natural way to the aggregate influence,
or aggregate causal effect, of feature i.
Number of individuals

The original motivation for game-theoretic measures is

6000
revenue division [15, Chapter 18]: the function v describes
the amount of money that each subset of players S ⊆ N can
4000 generate; assuming that the set N generates a total revenue of
v(N ), how should v(N ) be divided amongst the players? A
2000 special case of revenue division that has received significant
attention is the measurement of voting power [16]. In voting
systems with multiple agents with differing weights, voting
00.0 0.2 0.4 0.6 0.8 1.0 power often does not directly correspond to the weights of the
Maximum Influence of some input
agents. For example, the US presidential election can roughly
Fig. 1: A histogram of the highest specific causal influence be modeled as a cooperative game where each state is an agent.
for some feature across individuals in the adult dataset. Alone, The weight of a state is the number of electors in that state (i.e.,
most inputs alone have very low influence. the number of votes it brings to the presidential candidate who
wins that state). Although states like California and Texas have
higher weight, swing states like Pennsylvania and Ohio tend
Definition 3 (Marginal QII). For a quantity of interest Q, and to have higher power in determining the outcome of elections.
an input i, the Quantitative Input Influence of input i over a A voting system is modeled as a cooperative game: players
set S ⊆ N on Q is defined to be are voters, and the value of a coalition S ⊆ N is 1 if S
can make a decision (e.g. pass a bill, form a government,
ιQ (i, S) = Q(X−S US ) − Q(X−S∪{i} US∪{i} ).
or perform a task), and is 0 otherwise. Note the similarity
Notice that marginal QII can also be viewed as a difference to classification, with players being replaced by features. The
in set QIIs: ιQ (S ∪ {i}) − ιQ (S). Informally, the difference game-theoretic measures of revenue division are a measure
between ιQ (S ∪ {i}) and ιQ (S) measures the “added value” of voting power: how much influence does player i have
obtained by intervening on S ∪ {i}, versus intervening on S in the decision making process? Thus the notions of voting
alone. power and revenue division fit naturally with our goals when
The marginal contribution of i may vary significantly based defining aggregate QII influence measures: in both settings,
on S. Thus, we are interested in the aggregate marginal one is interested in measuring the aggregate effect that a single
contribution of i to S, where S is sampled from some element has, given the actions of subsets.
natural distribution over subsets of N \ {i}. In what follows, A revenue division should ideally satisfy certain desiderata.
we describe a few measures for aggregating the marginal Formally, we wish to find a function φ(N, v), whose input
contribution of a feature i to sets, based on different methods is N and v : 2N → R, and whose output is a vector in
for sampling sets. The primary method of aggregating the Rn , such that φi (N, v) measures some quantity describing
marginal contribution is the Shapley value [14]. The less the overall contribution of the i-th player. Research on fair
theoretically inclined reader can choose to proceed to Section revenue division in cooperative games traditionally follows an
V without a loss in continuity. axiomatic approach: define a set of properties that a revenue
division should satisfy, derive a function that outputs a value
A. Cooperative Games and Causality
for each player, and argue that it is the unique function that
In this section, we discuss how measures from the theory of satisfies these properties.
cooperative games define measures for aggregating marginal Several canonical fair cooperative solution concepts rely
influence. In particular, we observe that the Shapley value [14] on the fundamental notion of marginal contribution. given a
is characterized by axioms that are natural in our setting. player i and a set S ⊆ N \ {i}, the marginal contribution of
However, other measures may be appropriate for certain input i to S is denoted mi (S, v) = v(S ∪ {i}) − v(S) (we simply
data generation processes. write mi (S) when v is clear from the context). Marginal QII,
Definition 2 measures the influence that an intervention on as defined above, can be viewed as an instance of a measure of
a set of features S ⊆ N has on the outcome. One can naturally marginal contribution. Given a permutation π ∈ Π(N ) of the
think of Set QII as a function v : 2N → R, where v(S) is the elements in N , we define Pi (σ) = {j ∈ N | σ(j) < σ(i)};
influence of S on the outcome. With this intuition in mind, this is the set of i’s predecessors in σ. We can now simi-
one can naturally study influence measures using cooperative larly define the marginal contribution of i to a permutation
game theory, and in particular, prevalent influence measures in σ ∈ Π(N ) as mi (σ) = mi (Pi (σ)). Intuitively, one can think
cooperative games such as the Shapley value, Banzhaf index of the players sequentially entering a room, according to some
and others. These measures can be thought of as influence ordering σ; the value mi (σ) is the marginal contribution that
i has to whoever is in the room when she enters it.

603
 Generally speaking, game theoretic influence measures Definition 6
(Efficiency (Eff)). A value satisfies the efficiency
specify some reasonable way of aggregating the marginal property if i∈N φi = v(N ).
contributions of i to sets S ⊆ N . That is, they measure a
player’s expected marginal contribution to sets sampled from All of these axioms take on a natural interpretation in the
some distribution D over 2N , resulting in a payoff of QII setting. Indeed, if two features have the same probabilistic
 effect, no matter what other interventions are already in place,
ES∼D [mi (S)] = Pr[S]mi (S). they should have the same influence. In our context, the
D
S⊆N dummy axiom says that a feature that never offers information
Thus, fair revenue division draws its appeal from the degree with respect to an outcome should have no influence. In the
to which the distribution D is justifiable within the context case of specific causal influence, the efficiency axiom simply
where revenue is shared. In our setting, we argue for the use states that the total amount of influence should sum to
of the Shapley value. Introduced by the late Lloyd Shapley, the
Shapley value is one of the most canonical methods of dividing Pr(c(X) = c(x) | X = x) − Pr(c(X−N ) = c(x) | X = x)
revenue in cooperative games. It is defined as follows: =1 − Pr(c(X) = c(x)) = Pr(c(X) = c(x)).
1 
ϕi (N, v) = Eσ [mi (σ)] = mi (σ) That is, the total amount of influence possible is the likelihood
n!
σ∈Π(N ) of encountering elements whose evaluation is not c(x). This is
Intuitively, the Shapley value describes the following process: natural: if the vast majority of elements have a value of c(x),
players are sequentially selected according to some randomly it is quite unlikely that changes in features’ state will have any
chosen order σ; each player receives a payment of mi (σ). The effect on the outcome whatsoever; thus, the total amount of
Shapley value is the expected payment to the players under influence that can be assigned is Pr(c(X) = c(x)). Similarly,
this regime. The definition we use describes a distribution if the vast majority of points have a value different from x,
over permutations of N , not its subsets; however, it is easy then it is likelier that a random intervention would result in a
to describe the Shapley value in terms of a distribution over change in value, resulting in more influence to be assigned.
subsets. If we define p[S] = n1 n−1 1
, it is a simple exercise In the original paper by [14], it is shown that the Shapley
( |S| ) value is the only function that satisfies (Sym), (Dum), (Eff),
to show that
 as well as the additivity (Add) axiom.
ϕi (N, v) = p[S]mi (S).
S⊆N
Definition 7 (Additivity (Add)). Given two games
N, v1
, N, v2
, we write N, v1 + v2
to denote the game
Intuitively, p[S] describes the following process: first, choose v  (S) = v1 (S) + v2 (S) for all S ⊆ N . A value φ satisfies the
a number k ∈ [0, n − 1] uniformly at random; next, choose a additivity property if φi (N, v1 ) + φi (N, v2 ) = φi (N, v1 + v2 )
set of size k uniformly at random. for all i ∈ N .
The Shapley value is one of many reasonable ways of
measuring influence; we provide a detailed review of two In our setting, the additivity axiom makes little intuitive
others — the Banzhaf index [17], and the Deegan-Packel sense; it would imply, for example, that if we were to multiply
index [18] — in Appendix A. Q by a constant c, the influence of i in the resulting game
should be multiplied by c as well, which is difficult to justify.
B. Axiomatic Treatment of the Shapley Value
[19] offers an alternative characterization of the Shapley
In this work, the Shapley value is our function of choice for value, based on the more natural monotonicity assumption,
aggregating marginal feature influence. The objective of this which is a strong generalization of the dummy axiom.
section is to justify our choice, and provide a brief exposition
of axiomatic game-theoretic value theory. We present the Definition 8 (Monotonicity (Mono)). Given two games
axioms that define the Shapley value, and discuss how they N, v1
, N, v2
, a value φ satisfies strong monotonicity if
apply in the QII setting. As we show, by requiring some mi (S, v1 ) ≥ mi (S, v2 ) for all S implies that φi (N, v1 ) ≥
desired properties, one arrives at a game-theoretic influence φi (N, v2 ), where a strict inequality for some set S ⊆ N
measure as the unique function for measuring information use implies a strict inequality for the values as well.
in our setting. Monotonicity makes intuitive sense in the QII setting: if a
The Shapley value satisfies the following properties: feature has consistently higher influence on the outcome in one
Definition 4 (Symmetry (Sym)). We say that i, j ∈ N are setting than another, its measure of influence should increase.
symmetric if v(S ∪ {i}) = v(S ∪ {j}) for all S ⊆ N \ {i, j}. For example, if a user receives two transparency reports (say,
A value φ satisfies symmetry if φi = φj whenever i and j are for two separate loan applications), and in one report gender
symmetric. had a consistently higher effect on the outcome than in the
other, then the transparency report should reflect this.
Definition 5 (Dummy (Dum)). We say that a player i ∈ N
is a dummy if v(S ∪ {i}) = v(S) for all S ⊆ N . A value φ Theorem 9 ([19]). The Shapley value is the only function that
satisfies the dummy property if φi = 0 whenever i is a dummy. satisfies (Sym), (Eff) and (Mono).

604
 To conclude, the Shapley value is a unique way of measur- is a measure of group disparity, and the natural intervention
ing aggregate influence in the QII setting, while satisfying a distribution is using the prior as the question does not suggest
set of very natural axioms. a particular bias. On the other hand, when the question is:
“Which features are most influential for person A’s classifica-
IV. T RANSPARENCY S CHEMAS tion as opposed to person B?”, a natural quantity of interest is
We now discuss two generalizations of the definitions person A’s classification, and a natural intervention distribution
presented in Section II, and then define a transparency schema is the constant intervention using the features of person B.
that map the space of transparency reports based on QII. A thorough exploration of other points in this design space
a) Intervention Distribution: In this paper we only con- remains an important direction for future work.
sider randomized interventions when the interventions are
V. E STIMATION
drawn independently from the priors of the given input.
However, depending on the specific causal question at hand, While the model we propose offers several appealing prop-
we may use different interventions. Formally, this is achieved erties, it faces several technical implementation issues. Several
by allowing an arbitrary intervention distribution π inter such elements of our work require significant computational effort;
that in particular, both the probability that a change in feature state
π̃(x, u) = π(x)π inter (u). would cause a change in outcome, and the game-theoretic
influence measures are difficult to compute exactly. In the
The subsequent definitions remain unchanged. One example following sections we discuss these issues and our proposed
of an intervention different from the randomized intervention solutions.
considered in the rest of the paper is one held constant at a
vector x0 : A. Computing Power Indices
 Computing the Shapley or Banzhaf values exactly is gen-
1 for u = x0 erally computationally intractable (see [21, Chapter 4] for a
πx0 (u) =
inter
general overview); however, their probabilistic nature means
0 o.w.
that they can be well-approximated via random sampling.
A QII measure defined on the constant intervention as More formally, given a random variable X, suppose that we
defined above, measures the influence of being different from are interested in estimating some determined quantity q(X)
a default, where the default is represented by x0 . (say, q(X) is the mean of X); we say that a random variable
b) Difference Measure: A second generalization allows q ∗ is an ε-δ approximation of q(X) if
us to consider quantities of interest which are not real numbers.
Pr[|q ∗ − q(X)| ≥ ε] < δ;
Consider, for example, the situation where the quantity of
interest is an output probability distribution, as in the case in other words, it is extremely likely that the difference
in a randomized classifier. In this setting, a suitable measure between q(X) and q ∗ is no more than ε. An ε-δ approximation
for quantifying the distance between distributions can be scheme for q(X) is an algorithm that for any ε, δ ∈ (0, 1) is
used as a difference measure between the two quantities of able to output a random variable q ∗ that is an ε-δ approxima-
interest. Examples of such difference measures include the tion of q(X), and runs in time polynomial in 1ε , log 1δ.
KL-divergence [20] between distribution or distance metrics [22] show that when N, v
is a simple game (i.e. a game
between vectors. where v(S) ∈ {0, 1} for all S ⊆ N ), there exists an ε-
c) Transparency Schema: We now present a transparency δ approximation scheme for both the Banzhaf and Shapley
schema that maps the space of transparency reports based on values; that is, for φ ∈ {ϕ, β}, we can guarantee that for any
QII measures. It consists of the following elements: ε, δ > 0, with probability ≥ 1 − δ, we output a value φ∗i such
• A quantity of interest, which captures the aspect of the that |φ∗i − φi | < ε.
system we wish to gain transparency into. More generally, [23] observe that the number of i.i.d.
• An intervention distribution, which defines how a coun- samples needed in order to approximate the Shapley value and
terfactual distribution is constructed from the true distri- Banzhaf index is parametrized in Δ(v) = maxS⊆N v(S) −
bution. minS⊆N v(S). Thus, if Δ(v) is a bounded value, then an ε-
• A difference measure, which quantifies the difference δ approximation exists. In our setting, coalitional values are
between two quantities of interest. always within the interval [0, 1], which immediately implies
• An aggregation technique, which combines marginal QII the following theorem.
measures across different subsets of inputs (features). Theorem 10. There exists an ε-δ approximation scheme for
For a given application, one has to appropriately instantiate the Banzhaf and Shapley values in the QII setting.
this schema. We have described several instances of each
schema element. The choices of the schema elements are B. Estimating Q
guided by the particular causal question being posed. For Since we do not have access to the prior generating the
instance, when the question is: “Which features are most data, we simply estimate it by observing the dataset itself.
important for group disparity?”, the natural quantity of interest Recall that X is the set of all possible user profiles; in this

605
case, a dataset is simply a multiset (i.e. possibly containing VI. P RIVATE T RANSPARENCY R EPORTS
multiple copies of user profiles) contained in X . Let D be a One important concern is that releasing influence measures
finite multiset of X , the input space. We estimate probabilities estimated from a dataset might leak information about in-
by computing sums over D. For example, for a classifier c, dividual users; our goal is providing accurate transparency
the probability of c(X) = 1. reports, without compromising individual users’ private data.

To mitigate this concern, we add noise to make the measures
x∈D 1(c(x) = 1) differentially private. We show that the sensitivities of the QII
ÊD (c(X) = 1) = . (10)
|D| measures considered in this paper are very low and therefore
very little noise needs to be added to achieve differential
Given a set of features S ⊆ N , let D|S denote the elements
privacy.
of D truncated to only the features in S. Then, the intervened
The sensitivity of a function is a key parameter in ensuring
probability can be estimated as follows:
that it is differentially private; it is simply the worst-case


change in its value, assuming that we change a single data
uS ∈D|S x∈D 1(c(x|N \S uS ) = 1) point in our dataset. Given some function f over datasets, we
ÊD (c(X−S ) = 1) = .
|D|2 define the sensitivity of a function f with respect to a dataset
(11) D, denoted by Δf (D) as
Similarly, the intervened probability on individual outcomes
can be estimated as follows: max

|f (D) − f (D )|
D


where D and D differ by at most one instance. We use the
uS ∈DS 1(c(x|N \S uS ) = 1)
ÊD (c(X−S ) = 1|X = x) = . shorthand Δf when D is clear from the context.
|D| In order to not leak information about the users used
(12) to compute the influence of an input, we use the standard
Finally, let us observe group disparity: Laplace Mechanism [8] and make the influence measure
  differentially private. The amount of noise required depends
 
ÊD (c(X−S ) = 1 | X ∈ Y) − ÊD (c(X−S ) = 1 | X ∈
/ Y) on the sensitivity of the influence measure. We show that
the influence measure has low sensitivity for the individuals
The term ÊD (c(X−S ) = 1 | X ∈ Y) equals
used to sample inputs. Further, due to a result from [9] (and
1   stated in [25]), sampling amplifies the privacy of the computed
1(c(x|N \S uS ) = 1),
|Y| statistic, allowing us to achieve high privacy with minimal
x∈Y uS ∈DS
noise addition.
Thus group disparity can be written as: The standard technique for making any function differ-
 1   entially private is to add Laplace noise calibrated to the
 1(c(x|N \S uS ) = 1)
|Y| sensitivity of the function:
x∈Y uS ∈DS
1    Theorem 11 ([8]). For any function f from datasets to R,
− 1(c(x|N \S uS ) = 1). (13)
the mechanism Kf that adds independently generated noise
|D \ Y|
x∈D\Y uS ∈DS
with distribution Lap(Δf (D)/
) to the k output enjoys
-
We write Q̂Ydisp (S) to denote (13).
differential privacy.
If D is large, these sums cannot be computed efficiently. Since each of the quantities of interest aggregate over a
Therefore, we approximate the sums by sampling from the large number of instances, the sensitivity of each function is
dataset D. It is possible to show using the According to the very low.
Hoeffding bound [24], partial sums of n random variables
Xi , within a bound Δ, can be well-approximated with the Theorem 12. Given a dataset D,
1
following probabilistic bound: 1) ΔÊD (c(X) = 1) = |D|
2
 n  2) ΔÊD (c(X−S ) = 1) ≤ |D|
1   
  −2nε2 3) ΔÊD (c(X−S ) = 1|X = x) = |D| 1
Pr  (Xi − EXi ) ≥ ε ≤ 2 exp  
n  Δ
i=1 4) Q̂Y 1 1
disp (S) ≤ max |D∩Y| , |D\Y|
Since all the samples of measures discussed in the paper
Proof. We examine some cases here. In Equation 10, if two
are bounded within the interval [0, 1], we admit an ε-δ
datasets differ by one instance, then at most one term of the
approximation scheme where the number of samples n can
summation will differ. Since each term can only be either 0
be chosen to be greater than log(2/δ)/2ε2 . Note that these
or 1, the sensitivity of the function is
bounds are independent of the size of the dataset. Therefore,
 
given an efficient sampler, these quantities of interest can be  0 1  1
ΔÊD (c(X) = 1) =   − = .
approximated efficiently even for large datasets. |D| |D|  |D|

606
 Similarly, in Equation 11, an instance appears 2|D| − 1 actually used. In Section VII-B, we illustrate the distinction
times, once each for the inner summation and the outer between different quantities of interest on which Unary QII
summation, and therefore, the sensitivity of the function is can be computed. We also illustrate the effect of discrimination
on the QII measure. In Section VII-C, we analyze transparency
2|D| − 1 2 reports of three individuals to demonstrate how Marginal QII
ΔÊD (c(X−S ) = 1) = ≤ .
|D|2 |D| can provide insights into individuals’ classification outcomes.
For individual outcomes (Equation (12)), similarly, only one Finally, we analyze the loss in utility due to the use of
term of the summation can differ. Therefore, the sensitivity of differential privacy, and provide execution times for generating
(12) is 1/|D|. transparency reports using our prototype implementation.
Finally, we observe that a change in a single element x of We use the following datasets in our experiments:
1
D will cause a change of at most |D∩Y| if x ∈ D ∩ Y, or • adult [10]: This standard machine learning benchmark
1
of at most |D\Y| if x ∈ D \ Y. Thus, the maximal change to dataset is a a subset of US census data that classifies
  the income of individuals, and contains factors such as
1 1
(13) is at most max |Y| , |D\Y| .
age, race, gender, marital status and other socio-economic
parameters. We use this dataset to train a classifier that
While the sensitivity of most quantities of interest is low predicts the income of individuals from other parameters.
2
(at most a |D| ), Q̂Y
disp (S) can be quite high when |Y| is either
Such a classifier could potentially be used to assist credit
very small or very large. This makes intuitive sense: if Y is decisions.
a very small minority, then any changes to its members are • arrests [11]: The National Longitudinal Surveys are a
easily detected; similarly, if Y is a vast majority, then changes set of surveys conducted by the Bureau of Labor Statistics
to protected minorities may be easily detected. of the United States. In particular, we use the National
We observe that the quantities of interest which exhibit Longitudinal Survey of Youth 1997 which is a survey of
low sensitivity will have low influence sensitivity as well: young men and women born in the years 1980-84. Re-
for example, the local influence of S is 1(c(x) = 1) − spondents were ages 12-17 when first interviewed in 1997
ÊD (c(X−S ) = 1] | X = x); changing any x ∈ D (where and were subsequently interviewed every year till 2013.
1
x = x will result in a change of at most |D| to the local The survey covers various aspects of an individual’s life
influence. such as medical history, criminal records and economic
Finally, since the Shapley and Banzhaf indices are normal- parameters. From this dataset, we extract the following
ized sums of the differences of the set influence functions, we features: age, gender, race, region, history of drug use,
can show that if an influence function ι has sensitivity Δι, history of smoking, and history of arrests. We use this
then the sensitivity of the indices are at most 2Δι. data to train a classifier that predicts history of arrests to
To conclude, all of the QII measures discussed above aid in predictive policing, where socio-economic factors
α
(except for group parity) have a sensitivity of |D| , with α are used to decide whether individuals should receive a
being a small constant. To ensure differential privacy, we need visit from the police. This application is inspired by a
only need add noise with a Laplacian distribution Lap(k/|D|) similar application in [26].
to achieve 1-differential privacy. The two applications described above are hypothetical ex-
Further, it is known that sampling amplifies differential amples of decision-making aided by machine learning that use
privacy. potentially sensitive socio-economic data about individuals,
and not real systems that are currently in use. We use these
Theorem 13 ([9], [25]). If A is 1-differentially private, then
classifiers to illustrate the subtle causal questions that our QII
for any
∈ (0, 1), A (
) is 2
-differentially private, where
measures can answer.
A (
) is obtained by sampling an
fraction of inputs and
We use the following standard machine learning classifiers
then running A on the sample.
in our dataset: Logistic Regression, SVM with a radial basis
Therefore, our approach of sampling instances from D to function kernel, Decision Tree, and Gradient Boosted Decision
speed up computation has the additional benefit of ensuring Trees. Bishop’s machine learning text [27] is an excellent
that our computation is private. resource for an introduction to these classifiers. While Logistic
Table I contains a summary of all QII measures defined in Regression is a linear classifier, the other three are nonlinear
this paper, and their sensitivity. and can potentially learn very complex models. All our ex-
periments are implemented in Python with the numpy library,
VII. E XPERIMENTAL E VALUATION
and the scikit-learn machine learning toolkit, and run on an
We demonstrate the utility of the QII framework by develop- Intel i7 computer with 4 GB of memory.
ing two simple machine learning applications on real datasets.
Using these applications, we first argue, in Section VII-A, A. Comparison with Observational Measures
the need for causal measurement by empirically demonstrat- In the presence of correlated inputs, observational measures
ing that in the presence of correlated inputs, observational often cannot identify which inputs were causally influential.
measures are not informative in identifying which inputs were To illustrate this phenomena on real datasets, we train two

607
 Name Notation Quantity of Interest Sensitivity
QII on Individual Outcomes (3) ιind (S) Positive Classification of an Individual 1/|D|
QII on Actual Individual Outcomes (4) ιind-act (S) Actual Classification of an Individual 1/|D|
Average QII (5) ιind-avg (S) Average Actual Classification 2/|D|
QII on Group Outcomes (6) ιY
grp (S) Positive Classification for a Group 2/|D ∩ Y|
QII on Group Disparity (8) ιY
disp (S) Difference in classification rates among groups 2 max(1/|D \ Y|, 1/|D ∩ Y|)

TABLE I: A summary of the QII measures defined in the paper

classifiers: (A) where gender is provided as an actual input, In both examples, features correlated with the sensitive
and (B) where gender is not provided as an input. For classifier attribute are the most influential for group disparity according
(B), clearly the input Gender has no effect and any correlation to the sensitive attribute instead of the sensitive attribute
between the outcome and gender is caused via inference from itself. It is in this sense that QII measures can identify
other inputs. In Table II, for both the adult and the arrests proxy variables that cause associations between outcomes and
dataset, we compute the following observational measures: sensitive attributes.
Mutual Information (MI), Jaccard Index (Jaccard), Pearson QII with artificial discrimination: We simulate discrimi-
Correlation (corr), and the Disparate Impact Ratio (disp) to nation using an artificial experiment. We first randomly assign
measure the similarity between Gender and the classifiers ZIP codes to individuals in our dataset. Then to simulate
outcome. We also measure the QII of Gender on outcome. systematic bias, we make an f fraction of the ZIP codes
We observe that in many scenarios the observational quantities discriminatory in the following sense: All individuals in the
do not change, or sometimes increase, from classifier A to protected set are automatically assigned a negative classifi-
classifier B, when gender is removed as an actual input cation outcome. We then study the change in the influence
to the classifier. On the other hand, if the outcome of the of features as we increase f . Figure 3a, shows that the
classifier does not depend on the input Gender, then the QII influence of Gender increases almost linearly with f . Recall
is guaranteed to be zero. that Marital Status was the most influential feature for this
classifier without any added discrimination. As f increases,
B. Unary QII Measures the importance of Marital Status decreases as expected, since
the number of individuals for whom Marital Status is pivotal
In Figure 2, we illustrate the use of different Unary QII
decreases.
measures. Figures 2a, and 2b, show the Average QII measure
(Equation 5) computed for features of a decision forest classi- C. Personalized Transparency Reports
fier. For the income classifier trained on the adult dataset, the To illustrate the utility of personalized transparency reports,
feature with highest influence is Marital Status, followed by we study the classification of individuals who received poten-
Occupation, Relationship and Capital Gain. Sensitive features tially unexpected outcomes. For the personalized transparency
such as Gender and Race have relatively lower influence. reports, we use decision forests.
For the predictive policing classifier trained on the arrests The influence measure that we employ is the Shapley value,
dataset, the most influential input is Drug History, followed by with the underlying cooperative game defined over the local
Gender, and Smoking History. We observe that influence on influence Q. In more detail, v(S) = ιQA (S), with QA being
outcomes may be different from influence on group disparity. E[c(·) = 1 | X = x]; that is, the marginal contribution of
QII on group disparity: Figures 2c, 2d show influences i ∈ N to S is given by mi (S) = E[c(X−S ) = 1 | X =
of features on group disparity for two different settings. The x] − E[c(X−S∪{i} ) = 1 | X = x].
figure on the left shows the influence of features on group We emphasize that some features may have a negative
disparity by Gender in the adult dataset; the figure on the Shapley value; this should be interpreted as follows: a feature
right shows the influence of group disparity by Race in the with a high positive Shapley value often increases the certainty
arrests dataset. For the income classifier trained on the that the classification outcome is 1, whereas a feature whose
adult dataset, we observe that most inputs have negative Shapley value is negative is one that increases the certainty
influence on group disparity; randomly intervening on most that the classification outcome would be zero.
inputs would lead to a reduction in group disparity. In other Mr. X: The first example is of an individual from the
words, a classifier that did not use these inputs would be fairer. adult dataset, who we refer to as Mr. X, and is described in
Interestingly, in this classifier, marital status and not sex has Figure 4a. He is is deemed to be a low income individual, by
the highest influence on group disparity by sex. an income classifier learned from the data. This result may be
For the arrests dataset, most inputs have the effect of surprising to him: he reports high capital gains ($14k), and
increasing group disparity if randomly intervened on. In only 2.1% of people with capital gains higher than $10k are
particular, Drug history has the highest positive influence on reported as low income. In fact, he might be led to believe that
disparity in arrests. Although Drug history is correlated with his classification may be a result of his ethnicity or country
race, using it reduces disparate impact by race, i.e. makes fairer of origin. Examining his transparency report in Figure 4b,
decisions. however, we find that the the most influential features that led

608
 logistic kernel svm decision tree random forest
adult arrests adult arrests adult arrests adult arrests
MI A 0.045 0.049 0.046 0.047 0.043 0.054 0.044 0.053
MI B 0.043 0.050 0.044 0.053 0.042 0.051 0.043 0.052
Jaccard A 0.501 0.619 0.500 0.612 0.501 0.614 0.501 0.620
Jaccard B 0.500 0.611 0.501 0.615 0.500 0.614 0.501 0.617
corr A 0.218 0.265 0.220 0.247 0.213 0.262 0.218 0.262
corr B 0.215 0.253 0.218 0.260 0.215 0.257 0.215 0.259
disp A 0.286 0.298 0.377 0.033 0.302 0.335 0.315 0.223
disp B 0.295 0.301 0.312 0.096 0.377 0.228 0.302 0.129
QII A 0.036 0.135 0.044 0.149 0.023 0.116 0.012 0.109
QII B 0 0 0 0 0 0 0 0

TABLE II: Comparison of QII with associative measures. For 4 different classifiers, we compute metrics such as Mutual
Information (MI), Jaccard Index (JI), Pearson Correlation (corr), Group Disparity (disp) and Average QII between Gender and
the outcome of the learned classifier. Each metric is computed in two situations: (A) when Gender is provided as an input to
the classifier, and (B) when Gender is not provided as an input to the classifier.

to his negative classification were Marital Status, Relationship to a loss in utility of the QII measure. To estimate the loss
and Education. in utility, we set a noise of 0.005 as the threshold of noise
Mr. Y: The second example, to whom we refer as Mr. Y at which the measure is no longer useful, and then compute
(Figure 5), has even higher capital gains than Mr. X. Mr. Y is fraction of times noise crosses that threshold when Laplacian
a 27 year old, with only Preschool education, and is engaged noise is added at
= 1. The results of this experiment are as
in fishing. Examination of the transparency report reveals that follows:
the most influential factor for negative classification for Mr. Y Count Loss in Utility
Y is his Occupation. Interestingly, his low level of education Race: White 27816 2.97 × 10−14
is not considered very important by this classifier. Race: Black 3124 5.41 × 10−14
Mr. Z: The third example, who we refer to as Mr. Z Race: Asian-Pac-Islander 1039 6.14 × 10−05
(Figure 6) is from the arrests dataset. History of drug use Race: Amer-Indian-Eskimo 311 0.08
and smoking are both strong indicators of arrests. However, Race: Other 271 0.13
Mr. X received positive classification by this classifier even Gender: Male 21790 3.3 × 10−47
without any history of drug use or smoking. On examining Gender: Female 10771 3.3 × 10−47
his classifier, it appears that race, age and gender were most We note that for most reasonably sized groups, the loss in
influential in determining his outcome. In other words, the utility is negligible. However, the Asian-Pac-Islander, and the
classifier that we train for this dataset (a decision forest) has Amer-Indian-Eskimo racial groups are underrepresented in this
picked up on the correlations between race (Black), and age dataset. For these groups, the QII on Group Disparity estimate
(born in 1984) to infer that this individual is likely to engage in needs to be very noisy to protect privacy.
criminal activity. Indeed, our interventional approach indicates
that this is not a mere correlation effect: race is actively being
used by this classifier to determine outcomes. Of course, in
this instance, we have explicitly offered the race parameter
E. Performance
to our classifier as a viable feature. However, our influence
measure is able to pick up on this fact, and alert us of
the problematic behavior of the underlying classifier. More We report runtimes of our prototype for generating trans-
generally, this example illustrates a concern with the black parency reports on the adult dataset. Recall from Section VI
box use of machine learning which can lead to unfavorable that we approximate QII measures by computing sums over
outcomes for individuals. samples of the dataset. According to the Hoeffding bound to
derive an (
, δ) estimate of a QII measure, at
= 0.01, and
D. Differential Privacy n = 37000 samples, δ = 2 exp(−n
2 ) < 0.05 is an upper
Most QII measures considered in this paper have very low bound on the probability of the output being off by
. Table III
sensitivity, and therefore can be made differentially private shows the runtimes of four different QII computations, for
with negligible loss in utility. However, recall that the sensi- 37000 samples each. The runtimes of all algorithms except
tivity of influence measure on group disparity ιYdisp depends on for kernel SVM are fast enough to allow real-time feedback
the size of the protected group in the dataset D as follows: for machine learning application developers. Evaluating QII
 metrics for Kernel SVMs is much slower than the other metrics
Y 1 1
ιdisp = 2 max , because each call to the SVM classifier is very computationally
|D \ Y| |D ∩ Y| intensive due to a large number of distance computations that
For sufficiently small minority groups, a large amount of it entails. We expect that these runtimes can be optimized
noise might be required to ensure differential privacy, leading significantly. We present them as proof of tractability.

609
 0.16 0.18
0.14 0.16
0.12 0.14

QII on Outcomes
QII on Outcomes

0.12
0.10
0.10
0.08
0.08
0.06 0.06
0.04 0.04
0.02 0.02
0.00 0.00

ce
x

ion
y

y
ce
pit ip

ati on
m

Wo Sex
pit ss
lat n

pe e

try
E ain

Co ss
cu s

ea
k

Se
tor

tor
Oc tatu
Re patio

g
ee

Ra
Ca onsh

-Nu

Ca kcla

Ra
Lo
uc ati

g
urs A

Y
un
G

His

His

Re
rw

th
Ed duc

al
lS

al

on

r
i

B ir

us
ug

ing
rita

ns
Dr

ok
Ma

Ce
Sm
Ho

Feature Feature
(a) QII of inputs on Outcomes for the adult dataset (b) QII of inputs on Outcomes for the arrests dataset
0.28 0.09 0.18
0.08
0.23

QII on Group Disparity

QII on Group Disparity

0.13 0.05
0.18 Original Discrimination
-0.00 -0.00 -0.00 -0.00 -0.00
-0.02 -0.02 -0.02 -0.02 -0.02
0.13 0.08 Original Discrimination
-0.07 -0.01 -0.00 -0.00
-0.01
0.08 -0.09
0.03
0.03
ce

ar
n

y
y
lat ce
ip
pit on
Ed um
cu x

s
n
pe ge

Wo try
Co in
pit s
us

ati k

Se

gio

tor
tor
las
Se

Ca l Los
tio

Ye
uc ee

Ra
sh
Ga

Re Ra
Ca cati
tat

urs A

un
-N

His
His
Re
pa

rkc
Ed r w

ion

th
lS

al
on

a
u

Bir
us

ug
ing
rita

Oc

ns

Dr
ok
Ma

Ce

Sm
Ho

Feature Feature
(c) QII of Inputs on Group Disparity by Sex in the adult dataset (d) Influence on Group Disparity by Race in the arrests dataset
Fig. 2: QII measures for the adult and arrests datasets

logistic kernel-svm decision-tree decision-forest

QII on Group Disparity 0.56 234.93 0.57 0.73
Average QII 0.85 322.82 0.77 1.12
QII on Individual Outcomes (Shapley) 6.85 2522.3 7.78 9.30
QII on Individual Outcomes (Banzhaf) 6.77 2413.3 7.64 10.34

TABLE III: Runtimes in seconds for transparency report computation

VIII. D ISCUSSION 1
For the Banzhaf index, we have Pr[S] = 2n−1 , the Shapley
k!(n−k−1)!
A. Probabilistic Interpretation of Power Indices value has Pr[S] = n! (here, |S| = k), and the Deegan-
In order to quantitatively measure the influence of data Packel Index selects minimal winning coalitions uniformly at
inputs on classification outcomes, we propose causal inter- random. These choices of values for Pr[S] are based on some
ventions on sets of features; as we argue in Section III, natural assumptions on the way that players (features) interact,
the aggregate marginal influence of i for different subsets but they are by no means exhaustive. One can define other
of features is a natural quantity representing its influence. In sampling methods that are more appropriate for the model
order to aggregate the various influences i has on the outcome, at hand; for example, it is entirely possible that the only
it is natural to define some probability distribution over (or interventions that are possible in a certain setting are of size
equivalently, a weighted sum of) subsets of N \ {i}, where ≤ k + 1, it is reasonable to aggregate the marginal influence
Pr[S] represents the probability of measuring
the marginal
contribution of i to S; Pr[S] yields a value S⊆N \{i} mi (S).

610
 Age 23
0.40
Workclass Private
Sex
0.35
Education 11th
Marital Status Education-Num 7
0.30 Marital Status Never-married
Occupation Craft-repair
QII on Outcomes

0.25 Relationship Own-child

Race Asian-Pac-Islander
0.20 Gender Male
Capital Gain 14344
0.15 Capital Loss 0
Hours per week 40
0.10 Country Vietnam
0.05 (a) Mr. X’s profile
0.5
0.00
0.0 0.2 0.4 0.6 0.8 1.0
0.4

QII on Outcomes (Shapley)

Fraction of Discriminatory Zip Codes
0.3
(a) Change in QII of inputs as discrimination by Zip Code increases
in the adult dataset 0.2
0.35 0.1
Race
Drug History 0.0
0.30
0.1
QII on Outcomes

0.25 0.2
0.3
0.20

pe ce

rita hip
la on
Ed um
s

Co n
try
ati e
Ge n
Wo er

Oc Loss

us
pit k
las

tio

uc Ag
i

Ca wee
Ga

urs Ra
nd

Re ati

tat
un

Ma tions
-N
pa
rkc

uc
al

lS
al

on
r

cu
pit
Ca

0.15

Ed
Ho

0.10 (b) Transparency report for Mr. X’s negative classification

Fig. 4: Mr. X
0.05
0.0 0.2 0.4 0.6 0.8 1.0
Fraction of Discriminatory Zip Codes
If features are not binary, then there is no canonical way to
(b) Change in QII of inputs as discrimination by Zip Code increases transition from the data prior to a prior over subsets of features.
in the arrests dataset
B. Fairness
Fig. 3: The effect of discrimination on QII.
Due to the widespread and black box use of machine
learning in aiding decision making, there is a legitimate
concern of algorithms introducing and perpetuating social
of i over sets of size ≤ k, i.e. harms such as racial discrimination [28], [6]. As a result,
 1 the algorithmic foundations of fairness in personal informa-
if |S| ≤ k
Pr[S] = ( |S| )
n−1 tion processing systems have received significant attention
0 otherwise. recently [29], [30], [31], [12], [32]. While many of of the
algorithmic approaches [29], [31], [32] have focused on group
The key point here is that one must define some aggregation parity as a metric for achieving fairness in classification,
method, and that choice reflects some normative approach on Dwork et al. [12] argue that group parity is insufficient as
how (and which) marginal contributions are considered. The a basis for fairness, and propose a similarity-based approach
Shapley and Banzhaf indices do have some highly desirable which prescribes that similar individuals should receive similar
properties, but they are, first and foremost, a-priori measures classification outcomes. However, this approach requires a
of influence. That is, they do not factor in any assumptions on similarity metric for individuals which is often subjective and
what interventions are possible or desirable. difficult to construct.
One natural candidate for a probability distribution over S QII does not suggest any normative definition of fairness.
is some natural extension of the prior distribution over the Instead, we view QII as a diagnostic tool to aid fine-grained
dataset; for example, if all features are binary, one can identify fairness determinations. In fact, QII can be used in the spirit
a set with a feature vector (namely by identifying each S ⊆ N of the similarity based definition of [12]. By comparing the
with its indicator vector), and set Pr[S] = π(S) for all S ⊆ N . personalized privacy reports of individuals who are perceived

611
 Age 27 Birth Year 1984
Workclass Private Drug History None
Education Preschool Smoking History None
Education-Num 1 Census Region West
Marital Status Married-civ-spouse Race Black
Occupation Farming-fishing Gender Male
Relationship Other-relative (a) Mr. Z’s profile
Race White
Gender Male 0.4
Capital Gain 41310
0.3

QII on Outcome (Shapley)

Capital Loss 0
Hours per week 24
Country Mexico 0.2
(a) Mr. Y’s profile
0.4 0.1
0.3 0.0
QII on Outcome (Shapley)

0.2
0.1 0.1
0.0 0.2
0.1

ce

ar

n
er

y
gio

tor

tor
Ye
Ra

nd

His

His
Re
Ge
th
Bir
0.2

us

ug

ing
ns

Dr

ok
Ce

Sm
0.3
0.4 (b) Transparency report for Mr. Z’s positive classification
0.5 Fig. 6: Mr. Z.
C ce
ip
Wo tion

Oc -Num
Ed ex

lat ss

n
pit y

pe ge
in

rita oss
us

ati k
Ca ountr

tio
uc ee
sh
Ga

Re cla

Ra
S

tat
urs A
Ma al L
a

pa
Ed r w
ion
k
uc

lS
al

on
r

cu
pit
Ca

Ho

outcome. Our view is that this is a policy question and different

legal frameworks might take different viewpoints on it. At a
(b) Transparency report for Mr. Y’s negative classification
technical level, from the standpoint of information use, the
Fig. 5: Mr. Y. two situations are identical: the sensitive input is not really
used although it is supplied. However, the very fact that it
was supplied might be indicative of an intent to discriminate
to be similar but received different classification outcomes, even if that intended goal was not achieved. No matter what
and identifying the inputs which were used by the classifier to the policy decision is on this question, QII remains a useful
provide different outcomes. Additionally, when group parity diagnostic tool for discrimination because of the presence of
is used as a criteria for fairness, QII can identify the features proxy variables as described earlier.
that lead to group disparity, thereby identifying features being
IX. R ELATED W ORK
used by a classifier as a proxy for sensitive attributes.
The determination of whether using certain proxies for A. Quantitative Causal Measures
sensitive attributes is discriminatory is often a task-specific Causal models and probabilistic interventions have been
normative judgment. For example, using standardized test used in a few other settings. While the form of the inter-
scores (e.g., SAT scores) for admissions decisions is by and ventions in some of these settings may be very similar, our
large accepted, although SAT scores may be a proxy for generalization to account for different quantities of interests
several protected attributes. In fact, several universities have enables us to reason about a large class of transparency
recently announced that they will not use SAT scores for queries for data analytics systems ranging from classification
admissions citing this reason [33], [34]. Our goal is not outcomes of individuals to disparity among groups. Further,
to provide such normative judgments. Rather we seek to the notion of marginal contribution which we use to compute
provide fine-grained transparency into input usage (e.g., what’s responsibility does not appear in this line of prior work.
the extent to which SAT scores influence decisions), which Janzing et al. [35] use interventions to assess the causal
is useful to make determinations of discrimination from a importance of relations between variables in causal graphs;
specific normative position. in order to assess the causal effect of a relation between two
Finally, we note that an interesting question is whether variables, X → Y (assuming that both take on specific values
providing a sensitive attribute as an input to a classifier is X = x and Y = y), a new causal model is constructed, where
fundamentally discriminatory behavior, even if QII can show the value of X is replaced with a prior over the possible values
that the sensitive input has no significant impact on the of X. The influence of the causal relation is defined as the KL-

612
Divergence of the joint distribution of all the variables in the of models in this approach is restricted, a loss in predictive
two causal models with and without the value of X replaced. accuracy is a concern, and therefore, the central focus in
The approach of the intervening with a random value from the this line of work is the minimization of the loss in accuracy
prior is similar to our approach of constructing X−S . while maintaining interpretability. On the other hand, our
Independently, there has been considerable work in the approach to interpretability is forensic. We add interpretability
machine learning community to define importance metrics for to machine learning models after they have been learnt. As a
variables, mainly for the purpose of feature selection (see [36] result, our approach does not constrain the choice of models
for a comprehensive overview). One important metric is called that can be used.
Permutation Importance [37], which measures the importance
of a feature towards classification by randomly permuting D. Experimentation on Web Services
the values of the feature and then computing the difference There is an emerging body of work on systematic experi-
of classification accuracies before and after the permutation. mentation to enhance transparency into Web services such as
Replacing a feature with a random permutation can be viewed targeted advertising [50], [51], [52], [53], [54]. The setting in
as a sampling the feature independently from the prior. this line of work is different since they have restricted access
There exists extensive literature on establishing causal re- to the analytics systems through publicly available interfaces.
lations, as opposed to quantifying them. Prominently, Pearl’s As a result they only have partial control of inputs, partial
work [38] provides a mathematical foundation for causal rea- observability of outputs, and little or no knowledge of input
soning and inference. In [39], Tian and Pearl discuss measures distributions. The intended use of these experiments is to
of causal strength for individual binary inputs and outputs in a enable external oversight into Web services without any coop-
probabilistic setting. Another thread of work by Halpern and eration. Our framework is more appropriate for a transparency
Pearl discusses actual causation [40], which is extended in [41] mechanism where an entity proactively publishes transparency
to derive a measure of responsibility as degree of causality. reports for individuals and groups. Our framework is also
In [41], Chockler and Halpern define the responsibility of a appropriate for use as an internal or external oversight tool
variable X to an outcome as the amount of change required with access to mechanisms with control and knowledge of
in order to make X the counterfactual cause. As we discuss input distributions, thereby forming a basis for testing.
in Appendix A-B, the Deegan-Packel index is strongly related
to causal responsibility. E. Game-Theoretic Influence Measures
Recent years have seen game-theoretic influence measures
B. Quantitative Information Flow used in various settings. Datta et al. [55] also define a measure
One can think of our results as a causal alternative to for quantifying feature influence in classification tasks. Their
quantitative information flow. Quantitative information flow is measure does not account for the prior on the data, nor does
a broad class of metrics that quantify the information leaked it use interventions that break correlations between sets of
by a process by comparing the information contained before features. In the terminology of this paper, the quantity of
and after observing the outcome of the process. Quantitative interest used by [55] is the ability of changing the outcome by
Information Flow traces its information-theoretic roots to the changing the state of a feature. This work greatly extends and
work of Shannon [42] and Rényi [43]. Recent works have generalizes the concepts presented in [55], by both accounting
proposed measures for quantifying the security of information for interventions on sets, and by generalizing the notion of
by measuring the amount of information leaked from inputs to influence to include a wide range of system behaviors, such
outputs by certain variables; we point the reader to [44] for an as group disparity, group outcomes and individual outcomes.
overview, and to [45] for an exposition on information theory. Game theoretic measures have been used by various re-
Quantitative Information Flow is concerned with information search disciplines to measure influence. Indeed, such measures
leaks and therefore needs to account for correlations between are relevant whenever one is interested in measuring the
inputs that may lead to leakage. The dual problem of trans- marginal contribution of variables, and when sets of variables
parency, on the other hand, requires us to destroy correlations are able to cause some measurable effect. Lindelauf et al. [56]
while analyzing the outcomes of a system to identify the causal and Michalak et al. [57] use game theoretic influence measures
paths for information leakage. on graph-based games in order to identify key members of
terrorist networks. Del Pozo et al. [58] and Michalak et al. [59]
C. Interpretable Machine Learning use similar ideas for identifying important members of large
An orthogonal approach to adding interpretability to ma- social networks, providing scalable algorithms for influence
chine learning is to constrain the choice of models to those that computation. Bork et al. [60] use the Shapley value to assign
are interpretable by design. This can either proceed through importance to protein interactions in large, complex biological
regularization techniques such as Lasso [46] that attempt interaction networks; Keinan et al. [61] employ the Shapley
to pick a small subset of the most important features, or value in order to measure causal effects in neurophysical
by using models that structurally match human reasoning models. The novelty in our use of the game theoretic power
such as Bayesian Rule Lists [47], Supersparse Linear Integer indices lies in the conception of a cooperative game via a
Models [48], or Probabilistic Scaling [49]. Since the choice valuation function ι(S), defined by an randomized intervention

613
on inputs S. Such an intervention breaks correlations and [10] M. Lichman, “UCI machine learning repository,” 2013. [Online].
Available: http://archive.ics.uci.edu/ml
allows us to compute marginal causal influences on a wide [11] “National longitudinal surveys,” http://www.bls.gov/nls/.
range of system behaviors. [12] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, “Fairness
through awareness,” in Proceedings of the 3rd Innovations in Theoretical
X. C ONCLUSION & F UTURE W ORK Computer Science Conference (ITCS 2012), 2012, pp. 214–226.
[13] M. Lichman, “UCI machine learning repository,” 2013. [Online].
In this paper, we present QII, a general family of metrics Available: http://archive.ics.uci.edu/ml
for quantifying the influence of inputs in systems that process [14] L. Shapley, “A value for n-person games,” in Contributions to the Theory
personal information. In particular, QII lends insights into the of Games, vol. 2, ser. Annals of Mathematics Studies, no. 28. Princeton
University Press, 1953, pp. 307–317.
behavior of opaque machine learning algorithms by allowing
[15] M. Maschler, E. Solan, and S. Zamir, Game Theory. Cambridge
us to answer a wide class of transparency queries ranging University Press, 2013.
from influence on individual causal outcomes to influence [16] L. S. Shapley and M. Shubik, “A method for evaluating the distribution
on disparate impact. To achieve this, QII breaks correlations of power in a committee system,” The American Political Science
Review, vol. 48, no. 3, pp. 787–792, 1954.
between inputs to allow causal reasoning, and computes the [17] J. Banzhaf, “Weighted voting doesn’t work: a mathematical analysis,”
marginal influence of inputs in situations where inputs cannot Rutgers Law Review, vol. 19, pp. 317–343, 1965.
affect outcomes alone. Also, we demonstrate that QII can [18] J. Deegan and E. Packel, “A new index of power for simple n-person
games,” International Journal of Game Theory, vol. 7, pp. 113–123,
be efficiently approximated, and can be made differentially 1978.
private with negligible noise addition in many cases. [19] H. Young, “Monotonic solutions of cooperative games,” International
An immediate next step in this line of work is to explore Journal of Game Theory, vol. 14, no. 2, pp. 65–72, 1985.
[20] S. Kullback and R. A. Leibler, “On information and sufficiency,” Annals
adoption strategies in the many areas that use personal infor- of Mathematical Statistics, vol. 22, no. 1, pp. 79–86, 1951.
mation to aid decision making. Areas such as healthcare [3], [21] G. Chalkiadakis, E. Elkind, and M. Wooldridge, Computational Aspects
predictive policing [1], education [4], and defense [5] all have of Cooperative Game Theory. Morgan and Claypool, 2011.
a particularly acute need for transparency in their decision [22] Y. Bachrach, E. Markakis, E. Resnick, A. Procaccia, J. Rosenschein,
and A. Saberi, “Approximating power indices: theoretical and empirical
making. It is likely that specific applications will guide us in analysis,” Autonomous Agents and Multi-Agent Systems, vol. 20, no. 2,
our choice of a QII metric that is appropriate for that scenario, pp. 105–122, 2010.
which includes a choice for our game-theoretic power index. [23] S. Maleki, L. Tran-Thanh, G. Hines, T. Rahwan, and A. Rogers, “Bound-
ing the estimation error of sampling-based shapley value approximation
We have not considered situations where inputs do not with/without stratifying,” CoRR, vol. abs/1306.4265, 2013.
have well understood semantics. Such situations arise often in [24] W. Hoeffding, “Probability inequalities for sums of bounded random
settings such as image or speech recognition, and automated variables,” Journal of the American Statistical Association, vol. 58, no.
301, pp. 13–30, March 1963. [Online]. Available: http://www.jstor.org/
video surveillance. With the proliferation of immense process- stable/2282952?
ing power, complex machine learning models such as deep [25] N. Li, W. H. Qardaji, and D. Su, “Provably private data anonymization:
neural networks have become ubiquitous in these domains. Or, k-anonymity meets differential privacy,” CoRR, vol. abs/1101.2604,
2011. [Online]. Available: http://arxiv.org/abs/1101.2604
Defining transparency and developing analysis techniques in [26] Z. Jelveh and M. Luca, “Towards diagnosing accuracy loss in
such settings is important future work. discrimination-aware classification: An application to predictive polic-
ing,” Fairness, Accountability and Transparency in Machine Learning,
R EFERENCES vol. 26, no. 1, pp. 137–141, 2014.
[27] C. M. Bishop, Pattern Recognition and Machine Learning (Information
[1] W. L. Perry, B. McInnis, C. C. Price, S. C. Smith, and J. S. Hollywood,
Science and Statistics). Secaucus, NJ, USA: Springer-Verlag New York,
Predictive Policing: The Role of Crime Forecasting in Law Enforcement
Inc., 2006.
Operations. RAND Corporation, 2013.
[28] S. Barocas and H. Nissenbaum, “Big data’s end run around procedural
[2] T. Alloway, “Big data: Credit where credits due,” http://www.ft.com/
privacy protections,” Communications of the ACM, vol. 57, no. 11, pp.
cms/s/0/7933792e-a2e6-11e4-9c06-00144feab7de.html.
31–33, Oct. 2014.
[3] T. B. Murdoch and A. S. Detsky, “The inevitable application of big
data to health care,” http://jama.jamanetwork.com/article.aspx?articleid= [29] T. Calders and S. Verwer, “Three naive bayes approaches for
1674245. discrimination-free classification,” Data Mining and Knowledge
[4] “Big data in education,” https://www.edx.org/course/ Discovery, vol. 21, no. 2, pp. 277–292, 2010. [Online]. Available:
big-data-education-teacherscollegex-bde1x. http://dx.doi.org/10.1007/s10618-010-0190-x
[5] “Big data in government, defense and homeland security [30] A. Datta, M. Tschantz, and A. Datta, “Automated experiments on
2015 - 2020,” http://www.prnewswire.com/news-releases/ ad privacy settings: A tale of opacity, choice, and discrimination,” in
big-data-in-government-defense-and-homeland-security-2015---2020. Proceedings on Privacy Enhancing Technologies (PoPETs 2015), 2015,
html. pp. 92–112.
[6] J. Podesta, P. Pritzker, E. Moniz, J. Holdern, and J. Zients, “Big [31] T. Kamishima, S. Akaho, and J. Sakuma, “Fairness-aware learning
data: Seizing opportunities, preserving values,” Executive Office of the through regularization approach,” in Proceedings of the 2011 IEEE 11th
President - the White House, Tech. Rep., May 2014. International Conference on Data Mining Workshops (ICDMW 2011),
[7] “E.G. Griggs v. Duke Power Co., 401 U.S. 424, 91 S. Ct. 849, 28 L. 2011, pp. 643–650.
Ed. 2d 158 (1977).” [32] R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, “Learning fair
[8] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating representations,” in Proceedings of the 30th International Conference on
noise to sensitivity in private data analysis,” in Proceedings of the Machine Learning (ICML 2013), 2013, pp. 325–333.
Third Conference on Theory of Cryptography, ser. TCC’06. Berlin, [33] G. W. University, “Standardized test scores will be optional for
Heidelberg: Springer-Verlag, 2006, pp. 265–284. [Online]. Available: gw applicants,” 2015. [Online]. Available: https://gwtoday.gwu.edu/
http://dx.doi.org/10.1007/11681878 14 standardized-test-scores-will-be-optional-gw-applicants
[9] S. Kasiviswanathan, H. Lee, K. Nissim, S. Raskhodnikova, and [34] The National Center for Fair and Open Testing, “850+ colleges and
A. Smith, “What can we learn privately?” in Proceedings of the 49th universities that do not use sat/act scores to admit substantial numbers
IEEE Symposion on Foundations of Computer Science (FOCS 2008), of students into bachelor degree programs,” 2015. [Online]. Available:
Oct 2008, pp. 531–540. http://www.fairtest.org/university/optional

614
[35] D. Janzing, D. Balduzzi, M. Grosse-Wentrup, and B. Schölkopf, “Quan- [51] P. Barford, I. Canadi, D. Krushevskaja, Q. Ma, and S. Muthukrishnan,
tifying causal influences,” Ann. Statist., vol. 41, no. 5, pp. 2324–2358, “Adscape: Harvesting and analyzing online display ads,” in Proceedings
10 2013. of the 23rd International Conference on World Wide Web, ser. WWW
[36] I. Guyon and A. Elisseeff, “An introduction to variable and feature ’14. New York, NY, USA: ACM, 2014, pp. 597–608.
selection,” J. Mach. Learn. Res., vol. 3, pp. 1157–1182, Mar. 2003. [52] M. Lécuyer, G. Ducoffe, F. Lan, A. Papancea, T. Petsios, R. Spahn,
[Online]. Available: http://dl.acm.org/citation.cfm?id=944919.944968 A. Chaintreau, and R. Geambasu, “Xray: Enhancing the web’s trans-
[37] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. parency with differential correlation,” in Proceedings of the 23rd
5–32, Oct. 2001. [Online]. Available: http://dx.doi.org/10.1023/A: USENIX Conference on Security Symposium, ser. SEC’14. Berkeley,
1010933404324 CA, USA: USENIX Association, 2014, pp. 49–64.
[38] J. Pearl, Causality: Models, Reasoning and Inference, 2nd ed. New [53] A. Datta, M. C. Tschantz, and A. Datta, “Automated experiments on ad
York, NY, USA: Cambridge University Press, 2009. privacy settings.” PoPETs, vol. 2015, no. 1, pp. 92–112, 2015.
[39] J. Tian and J. Pearl, “Probabilities of causation: Bounds and identifi- [54] M. Lecuyer, R. Spahn, Y. Spiliopolous, A. Chaintreau, R. Geambasu,
cation,” Annals of Mathematics and Artificial Intelligence, vol. 28, no. and D. Hsu, “Sunlight: Fine-grained targeting detection at scale with
1-4, pp. 287–313, 2000. statistical confidence,” in Proceedings of the 22Nd ACM SIGSAC Con-
[40] J. Halpern and J. Pearl, “Causes and explanations: A structural-model ference on Computer and Communications Security, ser. CCS ’15. New
approach. part i: Causes,” The British journal for the philosophy of York, NY, USA: ACM, 2015, pp. 554–566.
science, vol. 56, no. 4, pp. 843–887, 2005. [55] A. Datta, A. Datta, A. Procaccia, and Y. Zick, “Influence in classification
[41] H. Chockler and J. Halpern, “Responsibility and blame: A structural- via cooperative game theory,” in Proceedings of the 24th International
model approach,” Journal of Artificial Intelligence Research, vol. 22, Joint Conference on Artificial Intelligence (IJCAI 2015), 2015, pp. 511–
pp. 93–115, 2004. 517.
[42] C. E. Shannon, “A mathematical theory of communication,” Bell [56] R. Lindelauf, H. Hamers, and B. Husslage, “Cooperative game theoretic
System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. [Online]. centrality analysis of terrorist networks: The cases of jemaah islamiyah
Available: http://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x and al qaeda.” European Journal of Operational Research, vol. 229,
[43] A. Rényi, “On measures of entropy and information,” in Proceedings no. 1, pp. 230–238, 2013.
of the Fourth Berkeley Symposium on Mathematical Statistics and [57] T. Michalak, T. Rahwan, P. Szczepanski, O. Skibski, R. Narayanam,
Probability, Volume 1: Contributions to the Theory of Statistics. M. Wooldridge, and N. Jennings, “Computational analysis of connec-
Berkeley, Calif.: University of California Press, 1961, pp. 547–561. tivity games with applications to the investigation of terrorist networks,”
[Online]. Available: http://projecteuclid.org/euclid.bsmsp/1200512181 in Proceedings of the 23rd International Joint Conference on Artificial
[44] G. Smith, “Quantifying information flow using min-entropy,” in Pro- Intelligence (IJCAI 2013), 2013, pp. 293–301.
ceedings of the 8th International Conference on Quantitative Evaluation [58] M. del Pozo, C. Manuel, E. González-Arangüena, and G. Owen,
of Systems (QEST 2011), 2011, pp. 159–167. “Centrality in directed social networks. a game theoretic approach,”
[45] T. M. Cover and J. A. Thomas, Elements of information theory. John Social Networks, vol. 33, no. 3, pp. 191–200, 2011.
Wiley & Sons, 2012. [59] T. Michalak, K. Aaditha, P. Szczepanski, B. Ravindran, and N. Jennings,
[46] R. Tibshirani, “Regression shrinkage and selection via the lasso: “Efficient computation of the shapley value for game-theoretic network
a retrospective,” Journal of the Royal Statistical Society Series centrality,” Journal of Artificial Intelligence Research, vol. 46, pp. 607–
B, vol. 73, no. 3, pp. 273–282, 2011. [Online]. Available: http: 650, 2013.
//EconPapers.repec.org/RePEc:bla:jorssb:v:73:y:2011:i:3:p:273-282 [60] P. Bork, L. Jensen, C. von Mering, A. Ramani, I. Lee, and E. Marcott,
[47] B. Letham, C. Rudin, T. H. McCormick, and D. Madigan, “Interpretable “Protein interaction networks from yeast to human,” Current Opinions
classifiers using rules and bayesian analysis: Building a better stroke in Structural Biology, vol. 14, no. 3, pp. 292–299, 2004.
prediction model,” Ann. Appl. Stat., vol. 9, no. 3, pp. 1350–1371, 09 [61] A. Keinan, B. Sandbank, C. Hilgetag, I. Meilijson, and E. Ruppin, “Fair
2015. [Online]. Available: http://dx.doi.org/10.1214/15-AOAS848 attribution of functional contribution in artificial and biological net-
[48] B. Ustun, S. Trac, and C. Rudin, “Supersparse linear integer models for works,” Neural Computation, vol. 16, no. 9, pp. 1887–1915, September
interpretable classification,” ArXiv e-prints, 2013. [Online]. Available: 2004.
http://arxiv.org/pdf/1306.5860v1 [62] M. Malawski, “Equal treatment, symmetry and banzhaf value axiom-
[49] S. Rping, “Learning interpretable models.” Ph.D. dissertation, Dortmund atizations,” International Journal of Game Theory, vol. 31, no. 1, pp.
University of Technology, 2006, http://d-nb.info/997491736. 47–67, 2002.
[50] S. Guha, B. Cheng, and P. Francis, “Challenges in measuring online
advertising systems,” in Proceedings of the 10th ACM SIGCOMM
Conference on Internet Measurement, ser. IMC ’10. New York, NY,
USA: ACM, 2010, pp. 81–87.

615
 A PPENDIX A characterizes the Banzhaf index; the axiom, called 2-efficiency,
A LTERNATIVE G AME -T HEORETIC I NFLUENCE M EASURES prescribes the behavior of an influence measure when two
In what follows, we describe two alternatives to the Shapley players merge. First, let us define a merged game; given a
value used in this work. The Shapley value makes intuitive game N, v
, and two players i, j ∈ N , we write T = {i, j}.
sense in our setting, as we argue in Section III-B. However, We define the game v̄ on N \ T ∪ {t̄} as follows: for every set
other measures may be appropriate for certain input data S ⊆ N \ {i, j}, v̄(S) = v(S), and v̄(S ∪ {t̄}) = v(S ∪ {i, j}),
generation processes. In what follows we revisit the Banzhaf note that the added player t̄ represents the two players i and
index, briefly discussed in Section III-A, and introduce the j who are now acting as one. The 2-Efficiency axiom states
readers to the Deegan-Packel index, a game-theoretic influ- that influence should be invariant under merges.
ence measure with deep connections to a formal theory of Definition 14 (2-Efficiency (2-EFF)). Given two players i, j ∈
responsibilty and blame [41]. N , let v̄ be the game resulting from the merge of i and j into
A. The Banzhaf Index a single player t̄; an influence measure φ satisfies 2-Efficiency
if φi (N, v) + φj (N, v) = φt̄ (N \ {i, j} ∪ {t̄}, v̄).
Recall that the Banzhaf index, denoted βi (N, v) is defined
as follows: Theorem 15 ([62]). The Banzhaf index is the only function
1  to satisfy (Sym), (D), (Mono) and (2-EFF).
βi (N, v) = n−1 mi (S).
2 In our context, 2-Efficiency can be interpreted as follows:
S⊆N \{i}
suppose that we artificially treat two features i and j as
The Banzhaf index can be thought of as follows: each
one, keeping all other parameters fixed; in this setting, 2-
j ∈ N \ {i} will join a work effort with probability 12
efficiency means that the influence of merged features equals
(or, equivalently, each S ⊆ N \ {i} has an equal chance
the influence they had as separate entities.
of forming); if i joins as well, then its expected marginal
contribution to the set formed is exactly the Banzhaf index. B. The Deegan-Packel Index
Note the marked difference between the probabilistic models:
Finally, we discuss the Deegan-Packel index [18]. While
under the Shapley value, we sample permutations uniformly
the Shapley value and Banzhaf index are well-defined for any
at random, whereas under the regime of the Banzhaf index,
coalitional game, the Deegan-Packel index is only defined for
we sample sets uniformly at random. The different sampling
simple games. A cooperative game is said to be simple if
protocols reflect different normative assumptions. For one,
v(S) ∈ {0, 1} for all S ⊆ N . In our setting, an influence
the

Banzhaf index is not guaranteed to be efficient; that is, measure would correspond to a simple game if it is binary
i∈N βi (N, v) is not
necessarily equal to v(N ), whereas it
n (e.g. it measures some threshold behavior, or corresponds to
is always the case that i=1 ϕi (N, v) = v(N ). Moreover, the
a binary classifier). The binary requirement is rather strong;
Banzhaf index is more biased towards √ measuring the marginal
however, we wish to draw the reader’s attention to the Deegan-
contribution of i to sets of size n2 ±O( n); this is because the
Packel index, as it has an interesting connection to causal
expected size of a randomly selected set follows a binomial
responsibility [41], a variant of the classic Pearl-Halpern
distribution B(n, 12 ). On the other hand, the Shapley value is
causality model [40], which aims to measure the degree to
equally likely to measure the marginal contribution of i to sets
which a single variable causes an outcome.
of any size k ∈ {0, . . . , k}, as i is equally likely to be in any
one position in a randomly selected permutation σ (and, in Given a simple game v : 2N → {0, 1}, let M(v) be the set
particular, the the set of i’s predecessors in σ is equally likely of minimal winning coalitions; that is, for every S ∈ M(v),
to have any size k ∈ {0, . . . , n − 1}). v(S) = 1, and v(T ) = 0 for every strict subset of S. The
Going back to the QII setting, the difference in sampling Deegan-Packel index assigns a value of
procedure is not merely an interesting anecdote: it is a sig- 1  1
δi (N, v) = .
nificant modeling choice. Intuitively, the Banzhaf index is |M(v)| |S|
S∈M(v):i∈S
more appropriate if we assume that large sets of features
would have a significant influence on outcomes, whereas the The intuition behind the Deegan-Packel index is as follows:
Shapley value is more appropriate if we assume that even players will not form coalitions any larger than what they
small sets of features might cause significant effects on the absolutely have to in order to win, so it does not make sense
outcome. Indeed, as we mention in Section VIII, aggregating to measure their effect on non-minimal winning coalitions.
the marginal influence of i over sets is a significant modeling Furthermore, when a minimal winning coalition is formed,
choice; while using the measures proposed here is perfectly the benefits from its formation are divided equally among its
reasonable in many settings, other aggregation methods may members; in particular, small coalitions confer a greater benefit
be applicable in others. for those forming them than large ones. The Deegan-Packel
Unlike the Shapley value, the Banzhaf index is not guar- index measures the expected payment one receives, assuming
anteed to be efficient (although it does satisfy the symmetry that every minimal winning coalition is equally likely to form.
and dummy properties). Indeed, [62] shows that replacing Interestingly, the Deegan-Packel index corresponds nicely to
the efficiency axiom with an alternative axiom, uniquely the notion of responsibility and blame described in [41].

616
Suppose that we have a set of variables X1 , . . . , Xn set to
x1 , . . . , xn , and some binary effect f (x1 , . . . , xn ) (written
as f (x)) occurs (say, f (x) = 1). To establish a causal
relation between the setting of Xi to xi and f (x) = 1,
[40] require that there is some set S ⊆ N \ {i} and some
values (yj )j∈S∪{i} such that f (x−S∪{i} , (yj )j∈S∪{i} ) = 0,
but f (x−S , (yj )j∈S ) = 1. In words, an intervention on the
values of both S and i may cause a change in the value of
f , but performing the same intervention just on the variables
in S would not cause such a change. This definition is at the
heart of the marginal contribution approach to interventions
that we describe in Section III-A. [41] define the responsibility
1
of i for an outcome as k+1 , where k is the size of the
smallest set S for which the causality definition holds with
respect to i. The Deegan-Packel index can thus be thought of
as measuring a similar notion: instead of taking the overall
minimal number of changes necessary in order to make i a
direct, counterfactual cause, we observe all minimal sets that
do so. Taking the average responsibility of i (referred to as
blame in [41]) according to this variant, we obtain the Deegan-
Packel index.
Example 16. Let us examine the following setup, based on
Example 3.3 in [41]. There are n = 2k + 1 voters (n is an
odd number) who must choose between two candidates, Mr.
B and Mr. G ([41] describe the setting with n = 11). All
voters elected Mr. B, resulting in an n-0 win. It is natural to
ask: how responsible was voter i for the victory of Mr. B?
According to [41], the degree of responsibility of each voter
1
is k+1 . It will require that i and k additional voters change
their vote in order for the outcome to change. Modeling this
setup as a cooperative game is quite natural: the voters are the
players N = {1, . . . , n}; for every subset S ⊆ N we have

1 if |S| ≥ k + 1
v(S) =
0 otherwise.
That is, v(S) = 1 if and only if the set S can change the
outcome of the election. The minimal winning coalitions here
are the subsets of N of size k + 1, thus the Deegan-Packel
index of player i is
1  1
δi (N, v) =
|M(v)| |S|
S∈M(v):i∈S

1 n 1 1 1
= n  = =
k+1
k k+1 n−k k+1
We note that if one assumes that all voters are equally likely
to prefer Mr. B over Mr. G, then the blame of voter i would
be computed in the exact manner as the Deegan-Packel index.

617