DATA PRIVACY

Fatih Turkmen, PhD

E-mail : [email protected]

Some slides are based on Dwork&Roth’s book (The Algorithmic Foundations of Differential Privacy),
Machanavajjhala et al.’s SIGMOD’17 tutorial and Takahashi’s Slides (Data Science with Privacy at Scale).
 TODAY

• Introduction to Differential Privacy (DP)

• DP Formalization
• Algorithms for DP
• Laplace
• Randomized Response
• Exponential Mechanism
• Compositions
• Shuffle Model
• Privacy-preserving ML with DP

2
 DIFFERENTIAL PRIVACY 𝜺

• Lots of publicly available data, many statistical studies…

• SEE THIS: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-
226.ipd.pdf

• Concrete Examples:
Useful Info
• Medical records of a (e.g. statistics)
Governor Statistical
Inference
• IMDB + Netflix à user
identification DB with sensitive Individual
data Identification
• Individual Identification (i.e. privacy violation)
from AOL search queries

• Paradoxical situation: Learning nothing about an individual while

learning useful information about a population
3
 DIFFERENTIAL PRIVACY: DEFINITION 𝜺

• Differential privacy aims to solve the problem:

“The risk associated with privacy violation of an individual should not
substantially increase as a result of participating in a statistical database”

• Here we need an algorithm/mechanism K (a differentially private

mechanism) that for all pairs of very similar data sets D and D’, it will behave
approximately the same on both data sets: If K(D) = X and K(D’) = Y then X
and Y should be indistinguishable.

• Most common method: Add controlled noise to data (e.g. Laplace,

Gaussian..) but there are many others...

4
 DIFFERENTIAL PRIVACY FORMALIZATION 𝜺

• A randomized function K gives ε-differential privacy if for all data sets D

and Dʹ differing on one entry and all S ⊆ Range(K), s.t.

Pr [K(D) ∈ S] ≤ 𝑒 ! . Pr(K(D’) ∈ S)

in other words, let O Î S as above :

Pr[𝐾 𝐷𝑘 = 𝑂]
≤ 𝑒!
Pr[𝐾 𝐷𝑘 ∓ 1 = 𝑂]
What is the relation
between ε and privacy?
The larger ε means more or
less privacy?
5
 VALUES OF 𝜀 𝜺

Figure borrowed from https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-

226.ipd.pdf

6
 DIFFERENTIAL PRIVACY FORMALIZATION (CONT.) 𝜺

Pr [K(D) = O] ≤ exp(𝜀) . Pr[K(D’) = O]

Intuition: By looking at the output O, the adversary

should not be able to make a distinction between
different (even if very close!) inputs D and D’.

Intuition: ε gives a means to control the distinction

between D and D’.

7
 EXAMPLE

• Let’s start with an example:

Suppose you have access to a database that allows you to

compute the total income of all residents in a certain area. If
you knew that Mr.White was going to move to that area
from his current location, simply querying this database before
and after his move would allow you to deduce his income.

Example by the courtesy of http://research.neustar.biz/2014/09/08/differential-privacy-the-basics/

8
PUBLIC DATAB ASES

9
PUBLIC DATAB ASES

10
 B ACK TO EXAMPLE

• Assume table represents the residents Id Name Income

of the selected region (where Mr White is 1 John Malkovich 80K

going to move).
2 Jamal Malik 90K
à 100 residents (+1 with Mr White)
3 Amelie Steiner 100K
4 Mirko Stanavic 75K
• Adversary has a query (Q(i)) mechanism 5 Mike Stanley 140K
to get the sum of income up to the given
6 Mehmet Uzun 90K
row i. Run Q before and after Mr White
7 Stijn Neuer 60K
moves…
.. .. ..
.. .. ..
• Q(101) – Q(100) = Mr White’s income

11
 B ACK TO EXAMPLE

• If K behaves as expected then we have a Id Name Income

guarantee that whether an individual is in a 1 John Malkovich 80K
given data set or not will not effect the
outcome of a query significantly. 2 Jamal Malik 90K

3 Amelie Steiner 100K

• Q(5) = 485K, Q(6) = 575K 4 Mirko Stanavic 75K

5 Mike Stanley 140K
6 Mehmet Uzun 90K
• K(Q(5)) = X, K(Q(6)) = Y
7 Stijn Neuer 60K
.. .. ..
• How do we define X and Y? .. .. ..

12
 ALGORITHMS FOR K

• Deterministic algorithms do not guarantee differential privacy

Inputs Outputs
Pr = 0
D
O1

O2
D’

Pr [K(D) = O1]
≤ 𝑒 $ where D,D’∈ {Inputs} and
Pr[K(D’) = O2]
O1,O2 ∈ {Outputs}
Ref: https://courses.cs.duke.edu/fall12/compsci590.3/slides/lec7.pdf 13
 ALGORITHMS FOR K

• Random sampling (i.e., selecting a subset of individuals from a population ) does

not guarantee differential privacy.

K : Performing the
“aggregate function” over a
random sample from D1 or
D2.
à This may have 0
probability:
Pr[K(D2) = O] = 0
Pr [D1à O] if the samples contain the
Pr[D2à O] = 0 means =∞
Pr[D2à O] elements from the
difference (e.g., D1 \ D2)
Ref: https://courses.cs.duke.edu/fall12/compsci590.3/slides/lec7.pdf 14
 DIFFERENTIAL PRIVACY FORMALIZATION 𝜺, 𝜹

• What we have seen (i.e., ε-differential) “pure” privacy, 𝛿 = 0.

• Approximate Differential Privacy: A randomized function K gives (ε, 𝛿)-
differential privacy (i.e., epsilon/delta privacy) if for all data sets D and
Dʹ differing on on entry and all S ⊆ Range(K), s.t.
Pr [K(D) = O] ≤ 𝑒 ! . Pr[K(D’) = O] + 𝜹

• Pure DP is a bit rigid: the probabilities of “unlikely events” with much

rare appearances (i.e., smaller Pr [K(D) ∈ S]) are preserved.

• Approximate Differential Privacy: Events with probabilities much

smaller than 𝜹 (𝜹 ≫ Pr[K(D’) = O]) are pruned.

15
 OUTPUT RANDOMIZATION

Query Query
K
O O’ (=O+ 𝜂)
Database

Adding noise to the query result:

• Results do not leak info about the database.
• O’ is very close to O.

Ref: https://courses.cs.duke.edu/fall12/compsci590.3/slides/lec7.pdf 16
 NOTE ON “ 𝜂 ” (NOISE) 𝜺

• Probability Mass Function (pmf) where X and Y are discrete random variables,
i.e., X, Y Î {2.6, 2.8, 3.0, 3.3 ….}

• Probability Density Function (pdf) where X and Y are continuous random

variables, i.e., X and Y are in a range such as 2.8 ≤ X ≤ 3.0…

• So K is really about the distribution of values in its range (K(D)) for the data sets
it is applied, i.e., the addition of noise.

17
 FUNCTION SENSITIVITY 𝜺

Pr [K(D) = O] ≤ 𝑒 ! . Pr[K(D’) = O]

• Implements a (aggregate) query Q.

• Function/Query Sensitivity: The largest (possible) distance between
the query results.
S(q) =

• In other words, the smallest number s.t. for any neighboring tables D
and D’.
|QD - QD’| ≤ S(q) What is the sensitivity of
COUNT?

18
 FUNCTION SENSITIVITY (CONT.)

Id Name Income

Say Income has the range [50K , 200K] 1 John Malkovich 80K

2 Jamal Malik 90K

3 Amelie Steiner 100K

What is the sensitivity of SUM for the income?
4 Mirko Stanavic 75K
5 Mike Stanley 140K
6 Mehmet Uzun 90K
7 Stijn Neuer 60K
That said though, summation queries
have unbounded sensitivity when no lower .. .. ..
and upper bounds exist on the value of the .. .. ..
attribute being summed!!

19
 FUNCTION SENSITIVITY (CONT.)

We’re going to run our differentially private mechanisms on

Any two neighbouring datasets an actual dataset - shouldn’t we consider neighbors
of that dataset?

Fix one of the two datasets to be the actual dataset being queried, and consider
all of its neighbours. Pay attention to parameter x from the “”fixed dataset

20
 DIFFERENTIAL PRIVACY WITH LAPLACE

Let S(q) denote the sensitivity of a query q.

(the maximum difference in the values QD and QD’,
for D and D’, a pair of databases that differ in only one row)

• Laplace Mechanism: Add controlled noise

with Laplace(μ,b):

|"#$|
C F( )
Lap(x | 𝜇, b) = 𝑒 %
DE
where
• b is the scale parameter and is set to
Courtesy of
S(q) / ε (calibrating the noise to the function’s https://en.wikipedia.org/wiki/Laplace_distribution

sensitivity)
• μ is the location parameter and it refers to distance to function’s true value (often
set to 0)
21
 DIFFERENTIAL PRIVACY WITH LAPLACE (CONT.)

• Now, given the noise 𝜂 drawn from

the Laplace distribution, the result of
the function is:
K(D) = QD + 𝜂
Why?

Because b = S(COUNT) / ε
Disease
(Y/N)
Example: Count the number of
Y
people with the disease. Courtesy of
Y
https://en.wikipedia.org/wiki/Laplace_distribution
N
Solution: 3 + 𝜂 where 𝜂 is drawn
Y from Lap(1 / ε).
N - b = 1 / ε , thus the variance is 2 / ε2.
N - No shift, so μ (mean) is 0.
22
 RANDOMIZED RESPONSE

Originally intended for improving bias in survey responses. Mostly used over Have you
“Yes/No” (i.e., binary) type of data aggregation but can be generalized. commited
a crime?
1. Flip a coin
2. If the coin is heads, answer the question truthfully
3. If the coin is tails, flip another coin
4. If the second coin is heads, answer “yes”; if it is tails, answer “no”
heads tails

If “yes” (property) answer is incriminating, Respond

randomized response provides that with Truthfully
probability at least 1/4 whether or not the
respondent actually has property P. heads tails

à Provides plausible deniability!! Respond Respond

Yes No

27
 RANDOMIZED RESPONSE

Disease Disease

Y
With probaility Y
p, report true
Y value N

N N

Y N
With probaility
N Y
1-p, report
N flipped value N

Ref: https://sigmod2017.org/wp-content/uploads/2017/03/04-Differential-Privacy-in-the-wild-1.pdf
28
 RANDOMIZED RESPONSE

Related to randomized process (not about output randomization!).

Randomized response satisfies ε-differential privacy for ε = ln(3) = 1.09

The Chrome Web browser has implemented and deployed RAPPOR to collect data
about Chrome clients à Based on Randomized Response [RAPPOR14].

Example Implementation: https://blog.openmined.org/randomized-response-in-privacy/

[RAPPOR14] Erlingsson et al.: RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response. CCS 2014: 1054-1067
https://arxiv.org/pdf/1407.6981
29
 EXPONENTIAL MECHANISM

Laplace/Gaussian à The utility of the response is directly related to the noise

values generated; that is, the popularity of the name or condition is appropriately
measured on the same scale and in the same units as the magnitude of the noise.
Ø Focus on numerical answers
Ø Add noise directly to the answer itself.

Exponential Mechanism:
Ø For aggregates that do not return a (real)
number!
ØWhen perturbation leads to invalid outputs.

30
 EXPONENTIAL MECHANISM

The analyst defines which element is the “best” by specifying a scoring

function that outputs a score for each element in the set, and also defines
the set of things to pick from.

Sensitivity of the scoring

function

Note: the output of the exponential mechanism is always a member of the set ℛ.

31
 EXPONENTIAL MECHANISM

• The mechanism provides differential privacy by approximately maximizing the

score of the element it returns
• To satisfy differential privacy, the exponential mechanism sometimes returns an
element from the set which does not have the highest score.

32
 COMPOSABILITY

1. Aggregate functions are often combined with other aggregate functions!

2. Repeatedly computing the same statistic using a DP mechanism will

degrade the protection provided by 𝜺, 𝜹.

Dinur/Nissim Result: A vast majority of records in a database of size n can

be reconstructed when n log(n)2 queries are answered by a statistical
database …

3. A statistical database must leak some information about each individual for
providing utility after all..

33
 COMPOSABILITY (CONT.)

Compositions: It is important to be able to reason about privacy

guarantees when complex functions are built from simple building
blocks!
• If building blocks are proven to be private, it would be
easy to reason about privacy of a complex algorithm built
entirely using these building blocks.

If K1, K2, ..., Kk are algorithms that access a

private database D such that each Ki satisfies
εi -differential privacy, then running all k
algorithms sequentially satisfies ε-differential This is like going to shopping…
privacy with ε=ε1+...+εk What limits you in your
shopping?

34
 COMPOSABILITY (CONT.)

ε=ε1+...+εk

ε= max{ε1,...,εk}

Courtesy of: https://programming-dp.com/ch6.html

35
LOC AL DP (LDP)

36
CENTRALIZED DP VS LDP

37
 CENTRALIZED DP VS LDP VS SHUFFLE

Courtesy of https://blog.openmined.org/differential-privacy-by-shuffling/

38
SHUFFLE MODEL

39
SHUFFLE MODEL (CONT.)

Courtesy of https://speakerdeck.com/line_developers/differential-
privacy-data-science-with-privacy-at-scale?slide=56

40
 PRIVACY-PRESERVING ML WITH DP [5]

• Not a collaborative setting, centralized training with DP:

• Suitable for applications of machine learning on mobile phones, tablets,
and other devices.
• Storing models on-device enables power-efficient, low-latency inference,
and may contribute to privacy since inference does not require
communicating user data to a central server

• Recall Differential Privacy

41
 PRIVACY-PRESERVING DL WITH DP: DP-SGD

• Uses Gaussian noise

• Defined over loss function
(not cost)

1. Compute the gradients

2. Clip each gradient in l2

norm, i.e., replace g by
g / max (1, ||g||2 / C).

• The differential privacy

guarantee of Algorithm 1
requires bounding the influence
of each individual example on gt

Abadi et al., Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on
Computer and Communications Security, pages 308–318, 2016

42
 PRIVACY-PRESERVING DL WITH DP: DP-SGD

3. Compute the average,

while adding noise

4. Take a step in the

opposite direction of
this average noisy
gradient.

43
 WHAT DID WE LEARN?

• Motivation for Differential Privacy

• Formal Definition Bad Privacy Protection
• Algorithms
• Output Randomization (Laplace),

Privacy

Utility
Randomized Response, Exponential Mechanism
• Composability
• Shuffle Model
• Privacy-preserving ML with DP Bad Service Experience

44