IEEE International Conference on Data Engineering

A Model for Data Leakage Detection

Panagiotis Papadimitriou 1 , Hector Garcia-Molina 2
Stanford University
353 Serra Street, Stanford, CA 94305, USA
1
[email protected]
2
[email protected]

Abstract— We study the following problem: A data distributor the distributor discovers some of those same objects in an
has given sensitive data to a set of supposedly trusted agents unauthorized place. (For example, the data may be found on
(third parties). Some of the data is leaked and found in an a web site, or may be obtained through a legal discovery
unauthorized place (e.g., on the web or somebody’s laptop). The
distributor must assess the likelihood that the leaked data came process.) At this point the distributor can assess the likelihood
from one or more agents, as opposed to having been indepen- that the leaked data came from one or more agents, as opposed
dently gathered by other means. We propose data allocation to having been independently gathered by other means.
strategies (across the agents) that improve the probability of In this paper we develop a model for assessing the “guilt”
identifying leakages. These methods do not rely on alterations of agents. We also study algorithms for distributing objects to
of the released data (e.g., watermarks). In some cases we can
also inject “realistic but fake” data records to further improve agents, in a way that improves our chances of identifying a
our chances of detecting leakage and identifying the guilty party. leaker. In these algorithms we consider the option of adding
“fake” objects to the distributed set. Such objects do not
correspond to real entities but appear realistic to the agents.
I. I NTRODUCTION In a sense, the fake objects act as a type of watermark for the
In the course of doing business, sometimes sensitive data entire set, without modifying any individual members. Our
must be handed over to supposedly trusted third parties. For algorithms are presented and evaluated in [2]; here we only
example, a hospital may give patient records to researchers briefly describe two of them to illustrate the choices that must
who will devise new treatments. Similarly, a company may be made.
have partnerships with other companies that require sharing
II. P ROBLEM S ETUP AND N OTATION
customer data. Another enterprise may outsource its data pro-
cessing, so data must be given to various other companies. We A. Entities and Agents
call the owner of the data the distributor and the supposedly A distributor owns a set T = {t1 , t2 , t3 , . . . } of valuable
trusted third parties the agents. Our goal is to detect when the data objects. The distributor wants to share some of the objects
distributor’s sensitive data has been leaked by agents, and if with a set of agents U1 , U2 , ..., Un , but does not wish the
possible to identify the agent that leaked the data. objects be leaked to other third parties. The objects in T could
We consider applications where the original sensitive data be of any type and size, e.g., they could be tuples in a relation,
cannot be perturbed. Perturbation is a very useful technique or relations in a database.
where the data is modified and made “less sensitive” before An agent Ui receives a subset of objects Ri ⊆ T , deter-
being handed to agents. For example, one can add random mined either by a sample request or an explicit request:
noise to certain attributes, or one can replace exact values by • Sample request Ri = S AMPLE (T, mi ): Any subset of mi
ranges [1]. However, in some cases it is important not to alter records from T can be given to Ui .
the original distributor’s data. For example, if an outsourcer is • Explicit request Ri = E XPLICIT (T, condi ): Agent Ui
doing our payroll, he must have the exact salary and customer receives all the T objects that satisfy condi .
identification numbers. If medical researchers will be treating Example. Say T contains customer records for a given com-
patients (as opposed to simply computing statistics), they may pany A. Company A hires a marketing agency U1 to do an
need accurate data for the patients. on-line survey of customers. Since any customers will do for
Traditionally, leakage detection is handled by watermarking, the survey, U1 requests a sample of 1000 customer records.
e.g., a unique code is embedded in each distributed copy. If At the same time, company A subcontracts with agent U2 to
that copy is later discovered in the hands of an unauthorized handle billing for all California customers. Thus, U2 receives
party, the leaker can be identified. Watermarks can be very all T records that satisfy the condition “state is California.”
useful in some cases, but again, involve some modification of
the original data. Furthermore, watermarks can sometimes be B. Guilty Agents
destroyed if the data recipient is malicious. Suppose that after giving objects to agents, the distributor
In this paper we study unobtrusive techniques for detecting discovers that a set S ⊆ T has leaked. This means that some
leakage of a set of objects or records. Specifically, we study third party called the target, has been caught in possession of
the following scenario: After giving a set of objects to agents, S. For example, this target may be displaying S on its web

1084-4627/09 $25.00 © 2009 IEEE 1307

DOI 10.1109/ICDE.2009.227
site, or perhaps as part of a legal discovery process, the target estimates for the guilt of agents, which is consistent with our
turned over S to the distributor. goals.
Since the agents U1 , . . . , Un have some of the data, it
Assumption 2. An object t ∈ S can only be obtained by the
is reasonable to suspect them leaking the data. However,
target in one of two ways:
the agents can argue that they are innocent, and that the S
• A single agent Ui leaked t from his own Ri set; or
data was obtained by the target through other means. For
• The target guessed (or obtained through other means) t
example, say one of the objects in S represents a customer
without the help of any of the n agents.
X. Perhaps X is also a customer of some other company, and
In other words, for all t ∈ S, the event that the target guesses
that company provided the data to the target. Or perhaps X
t and the events that agent Ui (i = 1, . . . , n) leaks object t
can be reconstructed from various publicly available sources
are disjoint.
on the web.
Our goal is to estimate the likelihood that the leaked data Before we present the general formula for computing
came from the agents as opposed to other sources. Intuitively, P r{Gi |S}, we provide a simple example. Assume that sets
the more data in S, the harder it is for the agents to argue T , R’s and S are as follows:
they did not leak anything. Similarly, the “rarer” the objects,
T = {t1 , t2 , t3 }, R1 = {t1 , t2 }, R2 = {t1 , t3 }, S = {t1 , t2 , t3 }.
the harder it is to argue that the target obtained them through
other means. Not only do we want to estimate the likelihood In this case, all three of the distributor’s objects have been
the agents leaked data, but we would also like to find out if leaked and appear in S.
one of them in particular was more likely to be the leaker. Let us first consider how the target may have obtained object
For instance, if one of the S objects was only given to agent t1 , which was given to both agents. From Assumption 2, the
U1 , while the other objects were given to all agents, we may target either guessed t1 or one of U1 or U2 leaked it. We know
suspect U1 more. The model we present next captures this that the probability of the former event is p, so assuming that
intuition. the probability that each of the two agents leaked t1 is the
We say an agent Ui is guilty if it contributes one or more same we have the following cases:
objects to the target. We denote the event that agent Ui is • the target guessed t1 with probability p;
guilty for a given leaked set S by Gi |S. Our next step is to • agent U1 leaked t1 to S with probability (1 − p)/2
estimate P r{Gi |S}, i.e., the probability that agent Ui is guilty • agent U2 leaked t1 to S with probability (1 − p)/2
given evidence S. Similarly, we find that agent U1 leaked t2 to S with probability
1 − p since it is the only agent that has this data object.
III. AGENT G UILT M ODEL
Given these values, the probability that agent U1 is not
To compute this P r{Gi |S}, we need an estimate for the guilty, namely that U1 did not leak either object is:
probability that values in S can be “guessed” by the target.
For instance, say some of the objects in T are emails of P r{Ḡ1 |S} = (1 − (1 − p)/2) × (1 − (1 − p)) (1)
individuals. We can conduct an experiment and ask a person Hence, the probability that U1 is guilty is:
with approximately the expertise and resources of the target to
find the email of say 100 individuals. If this person can find say P r{G1 |S} = 1 − P r{Ḡ1 |S} (2)
90 emails, then we can reasonably guess that the probability
In the general case (with our assumptions), to find the
of finding one email is 0.9. On the other hand, if the objects
probability that an agent Ui is guilty given a set S, first we
in question are bank account numbers, the person may only
compute the probability that he leaks a single object t to S.
discover say 20, leading to an estimate of 0.2. We call this
To compute this we define the set of agents Vt = {Ui |t ∈ Ri }
estimate pt , the probability that object t can be guessed by
that have t in their data sets. Then using Assumption 2 and
the target.
known probability p, we have:
To simplify the formulas that we present in the rest of the
paper, we assume that all T objects have the same pt , which P r{some agent leaked t to S} = 1 − p. (3)
we call p. Our equations can be easily generalized to diverse
Assuming that all agents that belong to Vt can leak t to S
pt ’s though they become cumbersome to display.
with equal probability and using Assumption 2 we obtain:
Next, we make two assumptions regarding the relationship (
among the various leakage events. The first assumption simply 1−p
, if Ui ∈ Vt
states that an agent’s decision to leak an object is not related P r{Ui leaked t to S} = |Vt | (4)
0, otherwise
to other objects.
Given that agent Ui is guilty if he leaks at least one value
Assumption 1. For all t, t0 ∈ S such that t 6= t0 the to S, with Assumption 1 and Equation 4 we can compute the
provenance of t is independent of the provenance of t0 . probability P r{Gi |S} that agent Ui is guilty:
To simplify our formulas, the following assumption states Y  1−p

that joint events have a negligible probability. As we argue in P r{Gi |S} = 1 − 1− (5)
|Vt |
the example below, this assumption gives us more conservative t∈S∩Ri

1308
 IV. DATA A LLOCATION P ROBLEM value is, the easier it is to identify Ui as the leaking agent.
The main focus of our work is the data allocation problem: Thus, we want to distribute data so that ∆ values are large:
how can the distributor “intelligently” give data to agents to Problem Definition. Let the distributor have data requests
improve the chances of detecting a guilty agent? As illustrated from n agents. The distributor wants to give tables R1 , . . . , Rn
in Figure 1, there are four instances of this problem we to agents U1 , . . . , Un respectively, so that:
address, depending on the type of data requests made by agents • he satisfies agents’ requests; and
(E for Explicit and S for Sample requests) and whether “fake • he maximizes the guilt probability differences ∆(i, j) for
objects” are allowed (F for the use of fake objects, and F for all i, j = 1, . . . , n and i 6= j.
the case where fake objects are not allowed). Assuming that the Ri sets satisfy the agents’ requests, we can
Fake objects are objects generated by the distributor that are express the problem as a multi-criterion optimization problem:
not in set T . The objects are designed to look like real objects,
and are distributed to agents together with the T objects, in maximize (. . . , ∆(i, j), . . . ) i 6= j (7)
(over R1 , . . . , Rn )
order to increase the chances of detecting agents that leak data.
We discuss fake objects in more detail in [2]. If the optimization problem has an optimal solution, that
means that there exists an allocation D∗ = {R1∗ , . . . , Rn∗ } such
that any other feasible allocation D = {R1 , . . . , Rn } yields
∆∗ (i, j) ≥ ∆(i, j) for all i, j. If there is no optimal allocation
D∗ , a multi-criterion problem has Pareto optimal allocations.
If Dpo = {R1po , . . . , Rnpo } is a Pareto optimal allocation, that
means that there is no other allocation that yields ∆(i, j) ≥
∆po (i, j) for all i, j.
B. Objective Approximation
Fig. 1. Leakage Problem Instances
We can approximate the objective of Equation 7 with
Equation 8 that does not depend on agents’ guilt probabilities
A. Optimization Problem and therefore on p.
 
|Ri ∩Rj |
The distributor’s data allocation to agents has one constraint minimize . . . , |R i|
, . . . i 6= j (8)
(over R1 , . . . , Rn )
and one objective. The distributor’s constraint is to satisfy
agents’ requests, by providing them with the number of objects This approximation is valid if minimizing the relative over-
|Ri ∩Rj |
they request or with all available objects that satisfy their lap |R i|
maximizes ∆(i, j). The intuitive argument for this
conditions. His objective is to be able to detect an agent who approximation is that the fewer leaked objects set Rj contains,
leaks any of his data objects. the less guilty agent Uj will appear compared to Ui (since
We consider the constraint as strict. The distributor may not S = Ri ). In [2] we prove that problems 7 and 8 are equivalent
deny serving an agent request as in [3] and may not provide if each T object is allocated to the same number of agents,
agents with different perturbed versions of the same objects as regardless of who these agents are.
in [4]. We consider fake object allocation as the only possible We present two different scalar versions of our problem in
constraint relaxation. Equations 9a and 9b. We will refer to objective 9a as the sum-
Our detection objective is ideal and intractable. Detection objective and to objective 9b as the max-objective.
would be assured only if the distributor gave no data object to n n
X 1 X
any agent. We use instead the following objective: maximize minimize |Ri ∩ Rj | (9a)
the chances of detecting a guilty agent that leaks all his objects. (over R1 , . . . , Rn )
i=1
|Ri | j=1
j6=i
We now introduce some notation to state formally the
distributor’s objective. Recall that P r{Gj |S = Ri } or simply |Ri ∩ Rj |
minimize max (9b)
P r{Gj |Ri } is the probability that agent Uj is guilty if the (over R1 , . . . , Rn ) i,j=1,...,n |Ri |
j6=i
distributor discovers a leaked table S that contains all Ri
objects. We define the difference functions ∆(i, j) as: Both scalar optimization problems yield the optimal solution
of the problem of Equation 8, if such solution exists. If there
∆(i, j) = P r{Gi |Ri } − P r{Gj |Ri } i, j = 1, . . . , n(6)
is no global optimal solution, the sum-objective yields the
Note that differences ∆ have non-negative values: given that Pareto optimal solution that allows the distributor to detect the
set Ri contains all the leaked objects, agent Ui is at least as guilty agent on average (over all different agents) with higher
likely to be guilty as any other agent. Difference ∆(i, j) is confidence than any other distribution. The max-objective
positive for any agent Uj , whose set Rj does not contain all yields the solution that guarantees that the distributor will
data of S. It is zero, if Ri ⊆ Rj . In this case the distributor detect the guilty agent with a certain confidence in the worst
will consider both agents Ui and Uj equally guilty since they case. Such a guarantee may adversely impact the average
have both received all the leaked objects. The larger a ∆(i, j) performance of the distribution.

1309
Input: R1 , . . . , Rn , cond1 , . . . , condn , b1 , . . . ,bn , B 1: function S ELECTA“GENT(R, R1 , . . . ”
, Rn )
Output: R1 , . . . , Rn , F1 ,. . . ,Fn i ← argmax |R1 0 | − |R 01|+1
P
2: j |Ri0 ∩ Rj |
1: R ← ∅ . Agents that can receive fake objects i0 :Ri0 ∈R i i

2: for i = 1, . . . , n do 3: return i
3: if bi > 0 then
4: R ← R ∪ {i}
Algorithm 2: Optimal Agent Selection
5: Fi ← ∅ . Set of fake objects given to agent Ui
6: while B > 0 do
7: i ← S ELECTAGENTAT R ANDOM(R, R1 , . . . , Rn ) provide with fake objects. We return back to our example and
8: f ← C REATE FAKE O BJECT(Ri , Fi , condi ) see how the objective would change if the distributor adds fake
9: Ri ← Ri ∪ {f }
10: Fi ← Fi ∪ {f } object f to R2 instead of R1 . In this case the sum-objective
11: bi ← bi − 1 would be 12 + 12 = 1 < 1.33.
12: if bi = 0 then The reason why we got a greater improvement is that the
13: R ← R\{Ri }
addition of a fake object to R2 has greater impact on the
14: B ←B−1
corresponding summation terms, since
Algorithm 1: Random Fake Object Allocation
1 1 1 1 1 1
− = < − = .
|R1 | |R1 | + 1 6 |R2 | |R2 | + 1 2
V. A LLOCATION S TRATEGIES FOR E XPLICIT R EQUESTS The left side of the inequality corresponds to the objective
In this paper we discuss only allocation strategies that are improvement after the addition of a fake object to R1 and right
applicable to problem instances with explicit data requests. part to R2 . We can use this observation to improve Algorithm 1
Strategies for the rest of the instances and an extensive with the use of procedure S ELECTAGENT() of Algorithm 2 in
experimental evaluation are available in [2]. place of procedure S ELECTAGENTAT R ANDOM() in line 7.
In problems of class EF the distributor is not allowed to Algorithm 2 makes a greedy choice by selecting the agent
add fake objects to the distributed data. So, the data allocation that will yield the greatest improvement in the sum-objective.
is fully defined by the agents’ data requests. Therefore, there The cost of this greedy choice in O(n2 ) in every iteration.
is nothing to optimize. The overall running time of the resulting algorithm is O(n +
In EF problems, objective values are initialized by agents’ n2 B) = O(n2 B). This greedy approach finds an optimal dis-
data requests. Say, for example, that T = {t1 , t2 } and there are tribution with respect to both optimization objectives defined
two agents with explicit data requests such that R1 = {t1 , t2 } in Equation 9 (proof in [2]).
and R2 = {t1 }. The value of the sum-objective is in this case:
VI. C ONCLUSIONS
2 n
X 1 X 1 1 In a perfect world there would be no need to hand over
|Ri ∩ Rj | = + = 1.5.
|R i | 2 1 sensitive data to agents that may unknowingly or maliciously
i=1 j=1
j6=i leak it. And even if we had to hand over sensitive data, in
The distributor cannot remove or alter the R1 or R2 data to a perfect world we could watermark each object so that we
decrease the overlap R1 ∩R2 . However, say the distributor can could trace its origins with absolute certainty. However, in
create one fake object (B = 1) and both agents can receive many cases we must indeed work with agents that may not be
one fake object (b1 = b2 = 1). In this case, the distributor 100% trusted, and we may not be certain if a leaked object
can add one fake object to either R1 or R2 to increase the came from an agent or from some other source. In spite of
corresponding denominator of the summation term. Assume these difficulties, we have shown it is possible to assess the
that the distributor creates a fake object f and he gives it to R1 . likelihood that an agent is responsible for a leak, based on
Agent U1 has now R1 = {t1 , t2 , f } with F1 = {f } and the the overlap of its data with the leaked data and the data of
value of the sum-objective decreases to 13 + 11 = 1.33 < 1.5. other agents, and based on the probability that objects can be
If the distributor is able to create more fake objects, he could “guessed” by other means. Our model is relatively simple, but
further improve the objective. We present in Algorithm 1 a we believe it captures the essential trade-offs.
strategy for randomly allocating fake objects. R EFERENCES
In lines 1-5, Algorithm 1 finds agents that are eligible to [1] L. Sweeney, “Achieving k-anonymity privacy protection using general-
receiving fake objects in O(n) time. Then, in the main loop, ization and suppression,” River Edge, NJ, USA, pp. 571–588, 2002.
the algorithm creates one fake object in every iteration and [2] P. Papadimitriou and H. Garcia-Molina, “Data leakage detection,”
Stanford University, Tech. Rep., 2008. [Online]. Available:
allocates it to a randomly selected agent from set R in line 7. http://dbpubs.stanford.edu/pub/2008-23
The main loop takes O(B) time. Hence, the running time of [3] S. U. Nabar, B. Marthi, K. Kenthapadi, N. Mishra, and R. Motwani,
the algorithm Pnis O(n + B).
“Towards robustness in query auditing,” in VLDB ’06: Proceedings of
the 32nd international conference on Very large data bases. VLDB
If B ≥ i=1 bi , the algorithm minimizes every term of Endowment, 2006, pp. 151–162.
the objective summation by adding the maximum number bi [4] R. Agrawal and J. Kiernan, “Watermarking relational databases,” in VLDB
Pn set Ri , yielding the optimal solution.
of fake objects to every ’02: Proceedings of the 28th international conference on Very Large Data
Bases. VLDB Endowment, 2002, pp. 155–166.
Otherwise, if B < i=1 bi (as in our example where B = 1 <
b1 + b2 = 2) the algorithm just selects at random the agents to

1310