Media Summary

Differential privacy is a mathematical framework for ensuring the privacy of individuals in datasets. It mitigates the privacy risk of disclosing sensitive information about individuals within the dataset during data analysis. Under such a framework, we are interested in finding the relationship between two variables (via statistical regression) after they are linked from two data sources with uncertainties. A pre-processing procedure of linking datasets is called record linkage, and the uncertainties should be taken into account in the downstream analysis. In the article, we propose two algorithms that satisfy differential privacy for regression estimation problems with linked data. The theoretical results regarding privacy guarantees and statistical accuracy are provided. We demonstrate the performance of the proposed algorithms through simulations and an application.

1. Introduction

Data for the same group of entities are often scattered across different resources, lacking unique identifiers for perfect linkage. To conduct statistical modeling or inference on the integrated information, it is necessary to probabilistically link multiple datasets by comparing the common quasi-identifiers (e.g., names, gender, address) as a pre-processing step. Such a procedure is called record linkage (RL), also known as entity resolution, or data matching (Christen2012data), which is an essential component of data integration in big data analytics (Dong2015). Thanks to its wide application in many disciplines such as public health and official statistics, record linkage has been studied for decades. Earlier pioneering works include Newcombe1959; fellegi1969; Jaro1989AdvancesIR. In addition, record linkage is frequently used in current practice. The U.S. Census Bureau has a long tradition using record linkage methodology for multiple endeavors. A current prominent example is the Decennial Census (census2022rl). In this context, record linkage involves using administrative records and other data sources to improve data quality, with efforts underway to construct a comprehensive “reference database” including individuals from multiple administrative records. A recent review paper, Olivier2022RL, provided a comprehensive summary of record linkage. Broadly speaking, there are two perspectives regarding record linkage (Chambers2019SmallAE): (1) the primary viewpoint concerns how to link the records; (2) the secondary perspective is focused on how to propagate the uncertainty to the downstream statistical learning tasks after the linkage has been determined. Our focus in this paper will adopt the second of these two perspectives.

Closely related to record linkage is data privacy. In the area of privacy-preserving record linkage (PPRL), two or more private datasets owned by different organizations are linked without revealing the data to one another (Hall2010pprl; Christen2020). The outcome of PPRL is the information regarding which pairs or sets are matched. PPRL, in turn, is associated with secure multiparty computation (SMPC) in that SMPC techniques are commonly used to solve PPRL problems (Kuzu2013; He2017pprl; Rao2019). PPRL to date only engages in the private linkage process from a primary perspective, without concerning how the linkage uncertainties would impact the downstream analysis. On the contrary, the secondary perspective is to modify the statistical tools to account for the linkage uncertainty. Our goal is to incorporate privacy into the secondary perspective of record linkage, which is different from yet complementary to PPRL or SMPC.

Privacy concerns have, if anything, become significantly more exacerbated with the emergence of individual-level big data. Releasing information about a sensitive dataset is subject to a variety of privacy attacks (Dwork2017Exposed). Therefore, there has been a growing demand for establishing robust privacy-preserving methodologies for modern statistics and machine learning. A mathematical framework proposed by Dwork2006, differential privacy (DP), is now considered the gold standard for rigorous privacy protection and has made its way to broad application in industry (apple2017dp; ms2020dp; google2021dp) and the public sector (census2021dp). The literature on differential privacy has been flourishing in recent years and the interface of differential privacy and statistics has started to draw increasing attention from the statistics community.

Recent work on differential privacy focuses primarily on individual statistical and machine learning tasks, with nontrivial upstream pre-processing, such as record linkage, typically not incorporated. In this paper, we consider the linear regression problem, i.e.,

(1) $$\bm{y}=X\bm{\beta}+\bm{e},\quad\bm{e}\sim\mathcal{N}(0,\sigma^{2}I_{n})$$

but where $X$ and $\bm{y}$ are observed in two separate datasets. As a result, rather than having $X$ and $\bm{y}$ in hand, we are instead provided with a pair $X$ and $\bm{z}$. Here $\bm{z}$ is a permutation of $\bm{y}$ resulting from record linkage performed by an external entity, who also supplies a minimum amount of information about the linkage accuracy. In the regression procedure, we take into account the linkage uncertainty as well as offer differential privacy guarantees. As shown in Figure 1 which depicts the pipeline of the problem we consider, we assume that an external analyst conducts record linkage a priori. From there, we aim to devise a private estimator for the regression coefficients of ultimate interest with the help of differential privacy.

Figure 1. Pipeline of private regression with linked data.

Specifically, we propose two algorithms for linear regression after record linkage to meet differential privacy: (1) post-RL noisy gradient descent (NGD), and (2) post-RL sufficient statistics perturbation (SSP). Our work builds on the seminal work by lahiri2005 where an estimator is proposed for linear regression with linked data in a non-privacy-aware setting. We construct a private estimator, ${\bm{\hat{\beta}}^{\text{priv}}}$, by deploying differential privacy tools to achieve privacy protections. To the best of our knowledge, our work is the first one in the literature to consider a statistical model after record linkage in a privacy-aware setting.

The two proposed algorithms also extend the noisy gradient method (Bassily2014PrivateER) and the “Analyze Gauss” algorithm (DworkTT014), which are applied to linear regression, to additionally handle the presence of linkage errors. Prior works (Sheffet17; wang2018; Bernstein2019; Cai2019TheCO; Alabi2022) on differentially private linear regression do not consider possible record linkage pre-processing. If the data are linked beforehand, directly applying their algorithms to the imperfectly linked data is not ideal. It is well known that overlooking the linkage errors leads to substantial bias even with a high linkage accuracy (Neter1965TheEO; Scheuren1993). Figure 2 showcases a toy example of record linkage, where mismatches, if treated as true, change the sign of the slope estimate. Our illustrative application later in the paper confirms this, where around 90% of the records are correctly linked, and the estimators ignoring linkage errors end up with large biases.

Figure 2. A toy example of record linkage with mismatches (dashed links).
The true dataset $(X,\bm{y})$ is $\{(1,2),(2,4),(3,6),(4,8)\}$, yielding a slop estimate $\hat{\beta}_{1}=2$, while the linked set $(X,\bm{z})$ is given by $\{(1,8),(2,4),(3,6),(4,2)\}$, yielding $\hat{\beta}_{1}=-1.6$.

Accompanying the estimators resulting from our algorithms, we provide mean-squared error bounds under typical regularity assumptions and record linkage schemes. When no linkage errors are present (i.e., a special case in our scenario), our result in Theorem 4.4 improves upon the noisy gradient method proposed in Cai2019TheCO by using zero-concentrated differential privacy (zCDP, Bun2016ConcentratedDP) to enable tighter bounds on privacy cost (see Lemma 10). Additionally, we have presented (approximate) theoretical variances for ${\bm{\hat{\beta}}^{\text{priv}}}$ resulting from both proposed algorithms. There appear to be very few other works that have addressed the issue of uncertainty. Two that we are aware of are Alabi, who provided confidence bounds for the univariate case, and Sheffet17, who provided confidence intervals dependent on differential privacy noise. Our work focuses on the multivariate case and appears to be the first to directly work on exact variances rather than relying on bounds.

The remainder of this paper is organized as follows. Section 2 provides preliminaries on linear regression with linked data and differential privacy. We propose our two algorithms in Section 3 and present the relevant theoretical results in Section 4. In Section 5, we conduct a series of simulation studies and an application to synthetic data. Section 6 concludes and discusses future work. Complete proofs of all theorems can be found in the supplementary materials.

2. Preliminaries

In this section, we review the background results of linear regression after record linkage upon which we build our work, and fundamental concepts from differential privacy. Related work on linear regression with linked data and record linkage with privacy awareness are discussed.

2.1. Linear Regression with Record Linkage

Let $(X,\Phi_{X})$ and $(\bm{y},\Phi_{\bm{y}})$ be two datasets that refer to the same group of $n$ entities, with unknown one-to-one correspondence. The quasi-identifiers $\Phi_{X}$ and $\Phi_{\bm{y}}$ are used to perform the linkage procedure. Let $(X,\bm{z})$ be the linked data where $\bm{z}$ is a permutation of $\bm{y}$. Consider the following model for $\bm{z}$:

(2) $$\mathbb{P}(z_{i}=y_{j})=q_{ij},\quad i,j=1,\dots,n,$$

then $\sum_{j=1}^{n}q_{ij}=1$ for all $i$ and $\sum_{i=1}^{n}q_{ij}=1$ for all $j$. Thus, $q_{ii}$ is the probability of the $i$th record being linked correctly. Let $Q=(q_{ij})$, which we call the matching probability matrix (MPM), a doubly stochastic matrix. The matrix $Q$ can be estimated, for example, through bootstrapping (Chipperfield2015; Chipperfield2020). In some cases, estimating $Q$ can require inference on only a single parameter (e.g., in the exchangeable linkage error (ELE) model described in Section 2.1.1).

For the fixed-design homoskedastic linear model (1), when inference is done after record linkage based on $(X,{\bm{z}})$, lahiri2005 proposed an unbiased estimator

(3) $${\hat{\bm{\beta}}^{\text{RL}}}=(W^{\top}W)^{-1}W^{\top}\bm{z},$$

where $W=QX$. Let $\bm{w}_{i}$ be the $i$-th row vector of $W$, then $\bm{w}_{i}=\sum_{j=1}^{n}q_{ij}\bm{x}_{j}$. Note that $\mathbb{E}(z_{i})=\bm{w}_{i}^{\top}\bm{\beta}$, where the expectation is taken over both linkage uncertainties and $\bm{y}$. Transforming $X$ into $W$ offers bias correction for regression estimation after record linkage.

In addition, the variance of ${\hat{\bm{\beta}}^{\text{RL}}}$ is given by

(4) $$\Sigma^{\text{RL}}\stackrel{{\scriptstyle def}}{{=}}\operatorname{Var}({\hat{\bm{\beta}}^{\text{RL}}})=(W^{\top}W)^{-1}W^{\top}\Sigma_{\bm{z}}W(W^{\top}W)^{-1},$$

where $\Sigma_{\bm{z}}\stackrel{{\scriptstyle def}}{{=}}\operatorname{Var}(\bm{z})$. lahiri2005 provide the following characterization of the first two moments of $\bm{z}$.

Lemma 2.1 (Theorem A.1, lahiri2005).

Under the model described by (1) and (2), we have for $i,j=1,\dots,n$

• $\mathbb{E}(z_{i})=\bm{w}_{i}^{\top}\bm{\beta}$;

• $\operatorname{Var}(z_{i})=\sigma^{2}+\bm{\beta}^{\top}A_{i}\bm{\beta}\mbox{ with }A_{i}=\sum_{j=1}^{n}q_{ij}(\bm{x}_{j}-\bm{w}_{i})(\bm{x}_{j}-\bm{w}_{i})^{\top}$;

• $\operatorname{Cov}(z_{i},z_{j})=\bm{\beta}^{\top}A_{ij}\bm{\beta}\mbox{ with }A_{ij}=\sum_{u=1}^{n}\sum_{v\neq u}^{n}q_{iu}q_{jv}(\bm{x}_{i}-\bm{w}_{u})(\bm{x}_{j}-\bm{w}_{v})^{\top}$.

Note that $\Sigma_{\bm{z}}$ involves the true coefficients $\bm{\beta}$ and $\Sigma_{\bm{z}}=\sigma^{2}I_{d}+h(\bm{\beta},Q,X)$ where $h(\bm{\beta},Q,X)$ is a function of $\bm{\beta},Q,X$ as elaborated in Lemma 2.1. Compared to the covariance of $\bm{y}$, $\Sigma_{\bm{z}}$ has an additional component $h(\bm{\beta},Q,X)$ due to the uncertainty of record linkage.

2.1.1. Structural Schemes of MPM

The matching probability matrix (MPM) $Q$ is generally assumed to have a simple structure. Two schemes used commonly in the literature are as follows.

Blocking Scheme

It is assumed that the MPM is a block diagonal matrix, which means the true matches only happen within blocks. Blocking significantly reduces the number of pairs for comparison and allows scalable record linkage. This scheme is used in almost all real-world applications, and different methods for blocking have been developed (Christen2012data; Steorts2014a; Christophides2020an).

Exchangeable Linkage Errors (ELE) Model

The ELE model (Chambers2009RegressionAO) assumes homogeneous linkage accuracy and errors:

(5) $$\mathbb{P}(\text{correct linkage})=q_{ii}=\gamma,$$

$$\mathbb{P}(\text{incorrect linkage})=q_{ij}=\frac{1-\gamma}{n-1}\text{ for }i\neq j.$$

The ELE model has been adopted in recent works, such as Chambers2019SmallAE; Chambers2022, for various estimation problems. Even though (5) may oversimplify the reality, it is a representative model for a secondary analyst who has minimum information about the linkage quality. When blocking is used, the homogeneous linkage accuracy assumption is imposed within individual blocks. In other words, it still allows heterogeneous linkage accuracy between blocks.

2.2. Differential Privacy

Let $\mathcal{X}$ be some data space, and $D,D^{\prime}\in\mathcal{X}^{n}$ be two neighboring datasets of size $n$ which only differ in one record. Such a relation is denoted by $D\sim D^{\prime}$.

Definition 1 ($(\epsilon,\delta)$-DP, DworkR14).

For $\epsilon>0,\delta\geq 0$, a randomized algorithm $A$: $\mathcal{X}^{n}\rightarrow\mathcal{R}$ is $(\epsilon,\delta)$-differentially private if, for all $D\sim D^{\prime}\in\mathcal{X}^{n}$ and any $\mathcal{O}\subseteq\mathcal{R}$,

(6) $$\mathbb{P}(A(D)\in\mathcal{O})\leq e^{\epsilon}\cdot\mathbb{P}(A(D^{\prime})\in\mathcal{O})+\delta.$$

The expression (6) controls the distance between the output distributions on two neighboring datasets through the privacy budget $\epsilon$ and $\delta$. Intuitively, differential privacy ensures that $D$ is not distinguishable from $D^{\prime}$ based on the outputs. Thus, $\epsilon$ should be small enough for the privacy level to be meaningful. Typically, $\epsilon\in(10^{-3},10)$ and $\delta=o(1/n)$.

Differential privacy enjoys the following properties that facilitate the construction of differentially private algorithms.

Proposition 2.1 (Basic composition, DworkR14).

If $f_{1}$ is $(\epsilon_{1},\delta_{1})$-DP and $f_{2}$ is $(\epsilon_{2},\delta_{2})$-DP, then $f:=(f_{1},f_{2})$ is $(\epsilon_{1}+\epsilon_{2},\delta_{1}+\delta_{2})$-DP.

Proposition 2.2 (Post-processing, DworkR14).

If $f$ is $(\epsilon,\delta)$-DP, for any deterministic mapping $g$ that takes $f(D)$ as an input, then $g(f(D))$ is $(\epsilon,\delta)$-DP.

Generally, a differentially private algorithm is constructed by adding random noise from a certain structured distribution, such as the Laplace or Gaussian distributions. A notion central to the amount of noise we add is the sensitivity of the estimation function we desire to release privately.

Definition 2 ($\ell_{2}$-sensitivity).

Let $f:\mathcal{X}^{n}\rightarrow\mathbb{R}^{d}$ be an algorithm. The $\ell_{2}$-sensitivity of $f$ is defined as

(7) $$\Delta_{f}=\max_{D\sim D^{\prime}\in\mathcal{X}^{n}}\|f(D)-f(D^{\prime})\|_{2}.$$

The sensitivity of a function characterizes how much the output would change if one record in the dataset changes. To achieve $(\epsilon,\delta)$-DP, the amount of noise we need depends on both the budget and the sensitivity. The Gaussian Mechanism is a canonical example that will be employed herein, which does just that.

Lemma 2.2 (Gaussian mechanism, DworkR14).

Let $0<\epsilon<1$ and $\delta>0$. For an algorithm $f$ on the dataset $D$, the Gaussian Mechanism $A(\cdot)$ defined as

(8) $$A(D):=f(D)+u,$$

where $u\sim\mathcal{N}(0,2\ln(1.25/\delta)(\Delta_{f}/\epsilon)^{2})$, is $(\epsilon,\delta)$-DP.

Combining the basic composition rule and the Gaussian mechanism, for a sequence of functions $(f_{1},f_{2},\dots,f_{T})$, let

$$u_{t}\sim\mathcal{N}\left(0,\frac{2T^{2}\Delta_{t}^{2}\ln(1.25T/\delta)}{\epsilon^{2}}\right),$$

where $\Delta_{t}$ is the $\ell_{2}$-sensitivity of $f_{t}$. Then, $A:=(f_{1}+u_{1},f_{2}+u_{2},\dots,f_{t}+u_{T})$ satisfies $(\epsilon,\delta)$-DP. However, as $T$ increases, this construction tends to add more noise than necessary due to the loose composition. Instead, we could utilize zero-concentrated differential privacy (zCDP, Bun2016ConcentratedDP), another variant of DP, to achieve tighter composition for $(\epsilon,\delta)$-DP. The following Lemma essentially captures the results from Bun2016ConcentratedDP, formulated for our purposes.

Lemma 2.3 (Better composition for $(\epsilon,\delta)$-DP via zCDP).

Let $\epsilon>0,\delta>0$. For a sequence of functions $(f_{1},f_{2},\dots,f_{T})$, let

(9) $$u_{t}\sim\mathcal{N}\left(0,\frac{T\Delta_{t}^{2}}{2\rho}\right),$$

with $\rho:=\epsilon+2\ln(1/\delta)-2\sqrt{(\epsilon+\ln(1/\delta))\ln(1/\delta)}$. Then, the randomized algorithm $A:=(f_{1}+u_{1},f_{2}+u_{2},\dots,f_{T}+u_{T})$ satisfies $(\epsilon,\delta)$-DP. If $\epsilon\leq\frac{8\ln(1/\delta)}{2+\sqrt{2}}$, it suffices to have

(10) $$u_{t}\sim\mathcal{N}\left(0,\frac{4T\Delta_{t}^{2}\ln(1/\delta)}{\epsilon^{2}}\right).$$

Please refer to the supplementary materials for details. Since, in most practical budget settings, we have $\epsilon\leq\frac{8\ln(1/\delta)}{2+\sqrt{2}}$, we will apply (10) for composition and analysis in the rest of the paper, acknowledging that (9) is valid for all parameter ranges.

In Section 3, we shall employ Lemmas 2.2 and 10 in devising two distinct algorithms for linear regression after record linkage.

2.3. Related Work

Linear regression with linked data is a fundamental statistical task that has been explored in various articles. Scheuren1993 initially considered the linkage model (2) for linear regression and proposed an estimator that is not generally unbiased. Later, lahiri2005 introduced an exactly unbiased OLS-like estimator given in (3) with an expression for the variance, which outperformed the approach by Scheuren1993. Besides, Chambers2009RegressionAO; Zhang2021linkage offered a few other estimators. According to their simulation studies, some of the estimators provided performance that was at most similar, but not noticeably better, compared to the one proposed by lahiri2005. Yet, Zhang2021linkage relaxed the condition by not assuming that the probability of correct linkage, $q_{ii}$ in the model (2), can be obtained or estimated. For more extensive reviews of this literature, Wang2022reg gave an account of the recent development of various methods on regression analysis with linked datasets. Chambers2022 reviewed current research on robust regression of linked data.

On the other hand, there is ongoing research on privacy-preserving record linkage (PPRL) in the field of computer science. PPRL aims to privately link multiple sensitive datasets held by different organizations when they are unwilling or not permitted to share their data with external parties due to privacy and confidentiality concerns. To achieve privacy protection, techniques such as SMPC and DP are combined with machine learning and deep learning methods for conducting PPRL (Christen2020; Divanis2021; Ranbaduge2022PPRL). PPRL primarily concerns data leakage during the linkage process and produces a linked dataset that can be used for further analysis, yet most applications treat the linked data as if there were no linkage errors. Neither the uncertainty propagation nor private release of the downstream analysis is considered within the scope of PPRL.

Note that there are several articles on privacy-preserving analysis on vertically partitioned databases. In these databases, the attributes are distributed among multiple parties, but common unique identifiers exist to facilitate data linkage across the different parties. Unlike probabilistic record linkage, vertically partitioned databases do not involve linkage errors. Du2004; Sanil2004; Hall2011SecureML; Gasc2017 discussed the implementations of privacy-preserving linear regression protocols that prevent data disclosure across organizations, whereas Dwork2004PPVRD considered data mining from the perspective of the private release of statistical querying in a spirit similar to our work.

3. Differentially Private Algorithms

The unbiased and simply structured estimator provided in (3) with a known closed-form variance makes it a suitable prototype to construct our private estimators. We introduce two differentially private algorithms in the following, based on (1) noisy gradient descent, and (2) sufficient statistics perturbation. As the names suggest, we mitigate privacy risk by perturbing either the gradient or sufficient statistics during the computation of the linear model. Hereafter, if not specified otherwise, $\|\cdot\|$ denotes the 2-norm.

3.1. Post-RL Noisy Gradient Descent

Gradient descent methods are ubiquitous in scientific computing for numerous optimization problems. Within the framework of differential privacy, Bassily2014PrivateER provided a noisy variant of the classic gradient descent algorithm. It was later adapted by Cai2019TheCO to solve the classic linear regression problem with faster convergence. Leveraging the work by Bassily2014PrivateER; Cai2019TheCO, we tailor the noisy gradient method for the post-RL linear regression model for $(X,\bm{z})$ based on (1) and (2).

Let $\mathcal{L}_{n}(\bm{\beta})\stackrel{{\scriptstyle def}}{{=}}\frac{1}{2n}(\bm{z}-W\bm{\beta})^{\top}(\bm{z}-W\bm{\beta})$ be the loss function, where recall $W=QX$. The minimizer of $\mathcal{L}_{n}(\bm{\beta})$ is the non-private RL estimator proposed by lahiri2005. Let $\Pi_{R}(\bm{r})$ denote the projection of $\bm{r}\in\mathbb{R}^{s}$ onto the $\ell_{2}$ ball $\{\bm{r}\in\mathbb{R}^{s}:\|\bm{r}\|\leq R\}$. The post-RL noisy gradient descent (NGD) algorithm is defined as follows.

Algorithm 1 Post-RL Noisy Gradient Descent

1:Linked dataset $(X,\bm{z})$ and matching probability matrix $Q$, privacy budget ($\epsilon,\delta$), noise scale factor $B$, step size $\eta$, number of iterations $T$, truncation level $R$, feasibility $C$, initial value $\bm{\beta}^{0}$.

2:Let $W=QX$.

3:for $t=0$ to $T-1$ do

4:    Generate $\bm{u}_{t}\,{\sim}\,\mathcal{N}\left(0,\omega^{2}I_{d}\right)$ where $\displaystyle\omega=\frac{2\eta B\sqrt{T\ln(1/\delta)}}{n\epsilon}$.

5:    Compute

(11) $$\bm{\beta}^{t+1}=\Pi_{C}(\bm{\beta}^{t}-\frac{\eta}{n}\sum_{i=1}^{n}(\bm{w}_{i}^{\top}\bm{\beta}^{t}-\Pi_{R}(z_{i}))\bm{w}_{i}+\bm{u}_{t}).$$

6:end for

7:$\bm{\hat{\beta}}^{priv}=\bm{\beta}^{T}$.

Algorithm 1 is a modified version of the projected gradient descent that incorporates (1) post-RL transformation of the design matrix, (2) addition of noise $\bm{u}_{t}$ at each gradient step, and (3) use of projection $\Pi_{R}(\cdot)$ on the response variable. The regular parameters, including $\eta$, $T$ and $C$ for the projected gradient method, are specified in Theorem 4.4 for the discussion of the accuracy of ${\bm{\hat{\beta}}^{\text{priv}}}$. The injection of noise follows Lemma 10. The scale of the Gaussian noise $\bm{u}_{t}$ at step $t$ depends on the privacy budget ($\epsilon,\delta$), and the noise scale factor $B$ associated with the sensitivity in the update function (11). The purpose of the projection on $\bm{z}$ is to bound the sensitivity of the gradient. With a proper choice of $R$ that scales up with $\sqrt{\ln n}$ (specified in Section 4), the projection does not affect the accuracy of the final estimator with high probability.

The major challenge lies in calculating the sensitivity. In the non-RL least square regression, two neighboring datasets $D=(X,\bm{y})$ and $D=(X^{\prime},\bm{y}^{\prime})$ differ in a single row, making it straightforward to derive the sensitivity of the gradient of $\mathcal{L}_{n}(\bm{\beta})=\frac{1}{2n}(\bm{y}-X\bm{\beta})^{\top}(\bm{y}-X\bm{\beta})$. Here, in the context of post-RL analysis, we consider two neighboring datasets containing both linking variables and regression variables, denoted as $D=(X,\Phi_{X},\bm{y},\Phi_{\bm{y}})$ and $D^{\prime}=(X^{\prime},\Phi_{X^{\prime}},\bm{y}^{\prime},\Phi_{\bm{y}^{\prime}})$, which differ in the record of one individual. The change in one row of the quasi-identifiers $\Phi_{X}$ and $\Phi_{\bm{y}}$ may affect more than one row of the matching probability matrix $Q$. As a result, the entries of the transformed design matrix $W=QX$ subject to change are not limited to one row as in the non-RL case. Consequently, determining the sensitivity of the gradient of $\mathcal{L}_{n}(\bm{\beta})=\frac{1}{2n}(\bm{z}-W\bm{\beta})^{\top}(\bm{z}-W\bm{\beta})$ becomes non-trivial. This challenge distinguishes our work from Cai2019TheCO. However, we will demonstrate in Section 4 that, under a condition on the structure of $Q$, the sensitivity can be tracked.

3.2. Post-RL Sufficient Statistics Perturbation

Noise can be injected into the process besides the gradient computation. Since the estimator interacts with the data through its (joint) sufficient statistics, an efficient way is to perturb the sufficient statistics to protect the data. Such a technique, sufficient statistics perturbation (SSP), has been used in previous works such as Slavkovic2009; Foulds2016; wang2018. For the non-private OLS estimator ${\hat{\bm{\beta}}^{\text{OLS}}}=(X^{\top}X)^{-1}X\bm{y}$, to perturb the joint sufficient statistics $(X^{\top}X,X\bm{y})$, it suffices to add noise to $A^{\top}A$ where $A=(X\mid\bm{y})$ is the augmented matrix. DworkTT014 offered an algorithm, “Analyze Gauss”, to privately release $A^{\top}A$. It was later utilized by Sheffet17 for private linear regression, primarily perturbing the sufficient statistics.

In our work, we adapt the “Analyze Gauss” algorithm to linear regression after record linkage, as shown in Algorithm 2. The noise scale factor $B$ is the sensitivity of $A^{\top}A\stackrel{{\scriptstyle def}}{{=}}\begin{pmatrix}W^{\top}W&W^{\top}\bm{z}\\ \bm{z}^{\top}W&\bm{z}^{\top}\bm{z}\end{pmatrix}$ which is specified in Section 4. The gram matrix $A^{\top}A$ exhibits properties that facilitate the computation of its sensitivity. Algorithm 2 illustrates how incorporating the joint sufficient statistics in a comprehensive form facilitates the deployment of differential privacy.

Algorithm 2 Post-RL Sufficient Statistics Perturbation

1:Linked dataset $(X,\bm{z})$ and matching probability matrix $Q$, privacy budget ($\epsilon,\delta$), noise scale factor $B$, truncation level $R$.

2:Let $W=QX$.

3:Generate a $d\times d$ symmetric Gaussian random matrix $U$ whose upper triangle entries (including the diagonal) are sampled i.i.d. from $\mathcal{N}(0,\omega^{2})$ where $\displaystyle\omega=\frac{B\sqrt{2\ln(1.25/\delta)}}{\epsilon}$.

4:Generate a $d$-dimensional Gaussian random vector $\bm{u}$ whose entries are sampled i.i.d. from $\mathcal{N}(0,\omega^{2})$.

5:if $W^{\top}W+U$ is computationally singular then

6:    Repeat steps $2\sim 3$.

7:end if

8:$\displaystyle{\bm{\hat{\beta}}^{\text{priv}}}=(W^{\top}W+U)^{-1}(W^{\top}\bm{z}^{*}+\bm{u})$ where $z^{*}=(\Pi_{R}(z_{1}),\dots,\Pi_{R}(z_{n}))^{\top}$.

Remark 3.1.

In step 4, by post-processing, checking for singularity of $W^{\top}W+U$ consumes no extra privacy budget. In fact, the probability of $W^{\top}W+U$ being singular decreases exponentially as the sample size increases.

An alternative approach to implementing the SSP method is to add random noise separately to each sufficient statistic. In this approach, the total privacy budget should be divided between $X^{\top}X$ and $X\bm{y}$ for the estimation of linear regression, as proposed by wang2018. However, treating the joint statistics as a whole is more economical in terms of budgeting in general. Lin2023 showed through comparison that splitting the total budget among the components results in introducing larger noise on average. Although adding noise individually to the components of interest allows for the private release of each quantity, it is not part of the goal of the estimation.

4. Theoretical Results

In this section, we provide the theoretical results of the two algorithms introduced in Section 3. The results are threefold: (1) differential privacy guarantees, (2) finite-sample error bounds, and (3) variances of the private estimators. We present each of these along with a discussion of the corresponding conditions as they relate to the main variables in our record linkage model. All proofs for these results can be found in the supplementary materials.

4.1. Privacy Guarantees

The algorithms are designed to achieve certain privacy guarantees, given the corresponding sensitivity, for the post-RL case:

Theorem 4.1 (Privacy Guarantees).

Assume the following boundedness conditions hold:

(A1) There is a constant $c_{x}<\infty$ such that $\|\bm{x}\|_{2}\leq c_{x}$.

(A2) Let $Q$ and $Q^{\prime}$ be the matching probability matrices (MPMs) resulting from the neighboring datasets $D$ and $D^{\prime}$ and let $Q\sim Q^{\prime}$ denote such a relation. We assume that $\sup_{Q\sim Q^{\prime}}\|Q-Q^{\prime}\|_{1}\leq M$ for some constant $M<\infty$, where $\|\cdot\|_{1}$ is the entry-wise 1-norm.

Given the linked data $(X,\bm{z})$ and the matching probability matrix $Q$ for the regression problem in (1), under Assumptions (A1) and (A2), it follows that

1. Algorithm 1 satisfies $(\epsilon,\delta)$-differential privacy with

   (12) $$B=Rc_{x}(M+4)+2Cc^{2}_{x}(M+2),$$

2. Algorithm 2 satisfies $(\epsilon,\delta)$-differential privacy with

   (13) $$B=Rc_{x}(M+4)+\max\{2c_{x}^{2}(M+2),2R^{2}\}.$$

Essentially, we assume that the data domain is bounded, which is critical for deriving a finite sensitivity of the target function on the data. (A1) is a standard assumption for a bounded design $X$. For the linking variables that are generally categorical, there are no analogous definitions of “norm” for numerical vectors. Instead, (A2) is imposed on the MPM since it summarizes all the information of the linking variables in the linkage model we consider. Specifically, we assume that two MPMs produced by two neighboring datasets do not differ much in terms of the entry-wise 1 norm. This assumption characterizes a bounded linkage model.

The rationale of (A2) is supported by typical schemes imposed on the structures of MPM in practice, as reviewed in Section 2.1.1. For example, with the blocking scheme, the size of each block is manageably small (O(1)). When one record is altered, the fluctuation of the MPM is limited to at most two blocks. Additionally, with the ELE model (5), as long as the changes to a single record only affect a finite number of records, the linkage accuracy $\gamma$ changes at most $O(1/n)$. Therefore, we have $\sup_{Q\sim Q^{\prime}}\|Q-Q^{\prime}\|_{1}=O(1)$. In general, a robust record linkage approach should not produce two considerably different MPMs from two neighboring datasets. Therefore, it is realistic to assume a bounded linkage model.

The proofs of Theorem 4.1 revolve around calculating the sensitivity of the target function in each algorithm. Besides the upper bounds $c_{x}$ and $M$ discussed above, the sensitivity also depends on the truncation level $R$ on the response. Truncation is commonly used in DP algorithm designs when there are no priori bounds on the relevant quantities (e.g., Abadi2016). In Section 4.2, we provide a specific choice of $R$ and present an accuracy statement with high probability.

4.2. Finite-Sample Error Bounds

We study the accuracy of the proposed estimators by deriving the finite-sample error bounds. In the following, we introduce two more assumptions in addition to (A1) and (A2):

(A3) The true parameter $\bm{\beta}$ satisfies $\|\bm{\beta}\|_{2}\leq c_{0}$ for some constant $0<c_{0}<\infty$.

(A4) The minimum and maximum eigenvalues of $W^{\top}W/n$ satisfy

(14) $$0<\frac{1}{L}<d\lambda_{\min}\left(\frac{W^{\top}W}{n}\right)\leq d\lambda_{\max}\left(\frac{W^{\top}W}{n}\right)<L$$

for some constant $1<L<\infty$.

Assumption (A4) implies the smoothness and strong convexity of the loss function $\mathcal{L}_{n}(\bm{\beta})=\frac{1}{2n}(\bm{z}-W\bm{\beta})^{\top}(\bm{z}-W\bm{\beta})$, which allows for a fast convergence rate for the gradient descent method in Algorithm 1. On the other hand, for Algorithm 2, note that the term $(W^{\top}W)^{-1}$ is a component of sufficient statistics. Assumption (A4) offers a bound on the norm of $(W^{\top}W)^{-1}$, which helps derive the error bound of ${\bm{\hat{\beta}}^{\text{priv}}}$. Let Assumption (A4’) be (A4) with $W$ replaced by $X$ and the constant $L$ replaced by $L^{\prime}$. The larger of $L$ and $L^{\prime}$ can be chosen as the constant to satisfy both (A4) and (A4’). Therefore, for convenience, we consider (A4) and (A4’) to be the same assumption. We first obtain the accuracy of the non-private estimators, for comparison purposes.

Lemma 4.2.

Let ${\hat{\bm{\beta}}^{\text{OLS}}}=\underset{\bm{\beta}}{\arg\min}(\bm{y}-X\bm{\beta})^{\top}(\bm{y}-X\bm{\beta})$ be the OLS estimator. Then, under (A4), it follows that $\displaystyle\mathbb{E}\|{{\hat{\bm{\beta}}^{\text{OLS}}}-\bm{\beta}}\|^{2}=\sigma^{2}\operatorname{tr}(X^{\top}X)^{-1}=\Theta\left(\frac{\sigma^{2}d^{2}}{n}\right)$.

Lemma 4.3.

Let ${\hat{\bm{\beta}}^{\text{RL}}}=\underset{\bm{\beta}}{\arg\min}(\bm{z}-W\bm{\beta})^{\top}(\bm{z}-W\bm{\beta})$ be the non-private record linkage estimator, and $\Sigma^{\text{RL}}$ be the covariance matrix of ${\hat{\bm{\beta}}^{\text{RL}}}$. Then,

(15) $$\mathbb{E}\|{{\hat{\bm{\beta}}^{\text{RL}}}-\bm{\beta}}\|^{2}=\operatorname{tr}(\Sigma^{\text{RL}}),$$

where $\Sigma^{\text{RL}}=(W^{\top}W)^{-1}W^{\top}\Sigma_{\bm{z}}W(W^{\top}W)^{-1}$.

As a special case, when the linkage is perfect (i.e., $Q$ is an identity matrix), the expected error of ${\hat{\bm{\beta}}^{\text{RL}}}$ in (15) takes the reduced form $\sigma^{2}\operatorname{tr}(X^{\top}X)^{-1}$ which is exactly the lower bound obtained by ${\hat{\bm{\beta}}^{\text{OLS}}}$. Then, by Lemma 4.2, we know that $\mathbb{E}\|{{\hat{\bm{\beta}}^{\text{RL}}}-\bm{\beta}}\|^{2}$ is of order at least $\displaystyle\frac{\sigma^{2}d^{2}}{n}$ under (A4). From a secondary perspective regarding record linkage, it is beyond our scope to study how $\operatorname{tr}(\Sigma^{\text{RL}})$ behaves in general.

For the two proposed algorithms, we present upper bounds of the excess squared error of the private estimators, namely, $\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}$.

Theorem 4.4 (Post-RL NGD).

Given the linked data $(X,\bm{z})$ and the matching probability matrix $Q$ for the regression problem in (1), set the parameters of Algorithm 1 as follows:

• step size $\eta=d/L$, number of iterations $T=\lceil L^{2}\ln(c_{0}^{2}n)\rceil$, feasibility $C=c_{0}$, initialization $\bm{\beta}^{0}=\bm{0}$;

• truncation level $R=\sigma\sqrt{2\ln n}$;

• noise scale factor $B=Rc_{x}(M+4)+2c_{0}c^{2}_{x}(M+2)$;

Under Assumptions (A1)-(A4), given $\delta=o(1/n)$, with probability at least $1-c_{1}e^{-c_{2}\ln n}-e^{-c_{3}d}$ where $c_{1},c_{2},c_{3}$ are constants (see the proof), it follows that

(16) $$\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}=\frac{1}{n}+O\left(\frac{\sigma^{2}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right).$$

Theorem 4.5 (Post-RL SSP).

Given the linked data $(X,\bm{z})$ and the matching probability matrix $Q$ for the regression problem in (1), in Algorithm 2, set

• truncation level $R=\sigma\sqrt{2\ln n}$;

• noise scale factor $B=Rc_{x}(M+4)+2\max\{c_{x}^{2}(M+2),R^{2}\}$.

Under Assumptions (A1)-(A4), given $\delta=o(1/n)$, with probability at least $1-c_{1}e^{-c_{2}\ln n}-e^{-c_{3}d}$ where $c_{1},c_{2},c_{3}$ are constants (see the proof),

(17) $$\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}=O\left(\frac{\sigma^{4}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right).$$

In both algorithms, the response is projected with a level $R=\sigma\sqrt{2\ln n}$ where $\sigma^{2}$ is the homoskedastic variance of the random error in linear model (1). Let $\mathcal{E}=\{\Pi_{R}(z_{i})=z_{i},\forall i\in[n]\}$, then $\mathcal{E}$ is a high-probability event. The error bound is analyzed under $\mathcal{E}$, thus we obtain a statement with high probability.

In the NGD method, the bound consists of two parts on the RHS in (16). The first error term $1/n$ results from the convergence rate of gradient descent after $T$ iterations. The second error term is due to the addition of Gaussian noise for privacy and thus involves $\epsilon,\delta$. It is worth noting that the choice in theory $T=\lceil L^{2}\ln(c_{0}^{2}n)\rceil$ is, to some extent, conservative to ensure the first error term is $O\left(1/n\right)$, which is the same order as $\mathbb{E}\|{{\hat{\bm{\beta}}^{\text{OLS}}}-\bm{\beta}}\|^{2}$. However, more iterations give rise to larger random noise being added to gradient updates due to a smaller privacy budget per iteration. In practice, a smaller number of iterations may be favored for the tradeoff (see the experiment in Section 5.2), especially when $n$ is not sufficiently large.

For the SSP algorithm, the convergence rate in (17) depends on similar variables as in the NGD algorithm. The major difference is that it is controlled by $\sigma^{4}$ instead of $\sigma^{2}$ due to the sensitivity of the gram matrix $A^{\top}A$ defined in Section 3.2. However, the SSP method has a faster convergence rate when $n$ is sufficiently large. As a result, the SSP estimator is more susceptible to a large variance of the random error in the response variable whereas the NGD method is more robust. As we shall see in Section 5, the performance of the two algorithms is different under various scenarios.

Putting together Lemma 4.3 and Theorems 4.4 and 4.5, we obtain a high probability error bound for each algorithm as follows.

Corollary 4.1.

Under the regularity conditions (A1)-(A4),

(i) (Post-RL NGD)

(18) $$\mathbb{E}\|{\bm{\hat{\beta}}^{\text{priv}}}-{\bm{\beta}}\|^{2}=O\left(\operatorname{tr}(\Sigma^{\text{RL}})+\frac{\sigma^{2}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right)$$

with probability at least $1-c_{1}e^{-c_{2}\ln n}-e^{-c_{3}d}$.

(ii) (Post-RL SSP)

(19) $$\mathbb{E}\|{\bm{\hat{\beta}}^{\text{priv}}}-{\bm{\beta}}\|^{2}=O\left(\operatorname{tr}(\Sigma^{\text{RL}})+\frac{\sigma^{4}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right)$$

with probability at least $1-c_{1}e^{-c_{2}\ln n}-e^{-c_{3}d}$.

4.3. Variances

As discussed in the Introduction, although a few works (Alabi; Sheffet17) have addressed uncertainty of DP estimators through confidence bounds and intervals, the exact variance of DP estimators is rarely determined in most cases. Recent work, such as Lin2023, has explored the variance of the private estimators for population proportions that have fairly simple structures. The main barrier to the inspection of variance is that if the noise is injected into the intermediate steps of the estimation process other than the output, then it is difficult to track the variability that noise introduces to the output estimator due to the intricate nature of the algorithm.

The NGD and SSP algorithms are two examples where noise is added in the middle of the estimation process. The operations like function composition and taking the inverse complicate the inspection of the variance of the output estimator ${\bm{\hat{\beta}}^{\text{priv}}}$. To address this issue, we investigate the variance of ${\bm{\hat{\beta}}^{\text{priv}}}$ for the two algorithms by studying the variances of two proxy estimators. The theoretical variances of the proxy estimators can be used to approximate those of ${\bm{\hat{\beta}}^{\text{priv}}}$.

Theorem 4.6 (Variance for Post-RL NGD).

In Algorithm 1 , if we consider the estimator without projections

(20) $$\bm{\beta}^{t+1}=\bm{\beta}^{t}-\frac{\eta}{n}W^{\top}(W\bm{\beta}^{t}-\bm{z})+\bm{u}_{t},$$

then the variance of the $T$th iterate is given by

(21) $$\Sigma=\sum_{t=1}^{T}(I_{d}-A)^{t-1}\cdot B^{\top}\Sigma_{\bm{z}}B\cdot\sum_{t=1}^{T}(I_{d}-A)^{t-1}+\omega^{2}\sum_{t=1}^{T}(I_{d}-A)^{2t-2},$$

where $I_{d}$ is the identity matrix of size $d$, $\displaystyle A\stackrel{{\scriptstyle def}}{{=}}\frac{\eta}{n}W^{\top}W$, $\displaystyle B\stackrel{{\scriptstyle def}}{{=}}\frac{\eta}{n}W$, and $\omega^{2}$ is the variance of $\bm{u}_{t}$.

Remark 4.1.

In the non-private case where $\omega^{2}=0$, let $T\to\infty$, in which case

$$\Sigma\to A^{-1}B^{\top}\Sigma_{\bm{z}}BA^{-1}=(W^{\top}W)^{-1}W^{\top}\Sigma_{\bm{z}}W(W^{\top}W)^{-1}=\Sigma^{\text{RL}},$$

which is exactly the variance of ${\hat{\bm{\beta}}^{\text{RL}}}$ given in (4).

The estimator in Algorithm 1 is a projected variant of (20). The use of projection with level $C$ on $\bm{\beta}^{t+1}$ in (11) impedes the exact analysis of variance for ${\bm{\hat{\beta}}^{\text{priv}}}$. Instead, we provide the variance in (21) for the non-projected estimator as a conservative variance for ${\bm{\hat{\beta}}^{\text{priv}}}$. The level of projection, the scale of noise, and the number of iterations together determine how conservative it is. From Remark 4.1, we know that as $T$ increases, the first term in the RHS of (21) is getting close to $\Sigma^{\text{RL}}$. The second term, $\displaystyle\omega^{2}\sum_{t=1}^{T}(I_{d}-A)^{2t-2}$, then summarizes the cumulative variability resulting from adding random noise at each iteration. Note that this term does not converge by simply increasing $T$, due to the fact that a smaller budget leads to larger noise at each iteration.

Theorem 4.7 (Variance for Post-RL SSP).

For Algorithm 2, let $\hat{\bm{\beta}}^{\prime}={\hat{\bm{\beta}}^{\text{RL}}}+(W^{\top}W)^{-1}\bm{u}-(W^{\top}W)^{-1}\cdot U(\bm{\hat{\bm{\beta}}^{\text{RL}}}+(W^{\top}W)^{-1}\bm{u})$, then ${\bm{\hat{\beta}}^{\text{priv}}}-\hat{\bm{\beta}}^{\prime}\stackrel{{\scriptstyle p}}{{\to}}0$ as $n\to\infty$. The variance of $\hat{\bm{\beta}}^{\prime}$ is given by

(22) $$\Sigma=\Sigma^{\text{RL}}+\omega^{2}(W^{\top}W)^{-1}(I_{d}+\Sigma_{0}+\Sigma_{1}+\Sigma_{2})(W^{\top}W)^{-1},$$

where $\Sigma^{\text{RL}}=\operatorname{Cov}({\hat{\bm{\beta}}^{\text{RL}}})$ and the entries of $\Sigma_{0}$, $\Sigma_{1}$ and $\Sigma_{2}$ are given by

• $(\Sigma_{0})_{kk}=\sum_{i=1}^{d}\beta_{i}^{2}$ for $k=1,...,d$; $(\Sigma_{0})_{kl}=\beta_{k}\beta_{l}$ for $k\neq l$.

• $(\Sigma_{1})_{kk}=\sum_{i=1}^{d}\Sigma^{\text{RL}}_{ii}$ for $k=1,...,d$; $(\Sigma_{1})_{kl}=\Sigma^{\text{RL}}_{kl}$ for $k\neq l$.

• $(\Sigma_{2})_{kk}=\sum_{i=1}^{d}\Sigma^{\prime}_{ii}$ for $k=1,...,d$; $(\Sigma_{2})_{kl}=\Sigma^{\prime}_{kl}$ for $k\neq l,$ where $\Sigma^{\prime}\stackrel{{\scriptstyle def}}{{=}}\omega^{2}(W^{\top}W)^{-2}$.

Remark 4.2.

As shown in the proof of 4.7 (see the supplemental), the proxy estimator $\hat{\bm{\beta}}^{\prime}$ is a first-order approximation for ${\bm{\hat{\beta}}^{\text{priv}}}$ using Taylor series for the term $(I+U(W^{\top}W)^{-1})^{-1}$ which appears in the decomposition of ${\bm{\hat{\beta}}^{\text{priv}}}$.

The variance of $\hat{\beta}^{\prime}$ also consists of two parts: the variance of the non-private estimator ${\hat{\bm{\beta}}^{\text{RL}}}$ and the additional variation due to the noise injected for privacy purposes. Given Assumption (A4), we have $\|(W^{\top}W)^{-1}\|=O(d/n)$ that appears in $\Sigma_{1}$ and $\Sigma_{2}$. As $n$ increases, the dominant component of the second term would be $\omega^{2}(W^{\top}W)^{-1}(I_{d}+\Sigma_{0})(W^{\top}W)^{-1}$.

5. Numerical Results

To evaluate the finite-sample performance of the proposed algorithms, we conduct a series of simulation studies and an application to a synthetic dataset that contains real data.

5.1. Simulation Studies

In this section, we conduct simulation studies to assess the performance of the two proposed algorithms for simple linear regression with linked data. The non-private OLS estimator and RL estimator ${\hat{\bm{\beta}}^{\text{RL}}}$ by lahiri2005 are included as benchmarks. The private, non-RL counterpart methods are also performed in the absence of linkage errors for comparison.

For each simulation, a fixed design matrix $X$ and an matching probability matrix $Q$ are produced and a total of 1000 repetitions are run over the randomness of both the intrinsic error $\bm{e}\sim\mathcal{N}(0,\sigma^{2}I_{n})$ of the regression model and the noise injected for privacy. Figure 3 displays the $\ell_{2}$ relative error and both empirical and theoretical variances for the two settings.

Two sets of simulations are conducted to explore the performance with varying sample size $n$ and $\sigma$, the homoskedastic variance of the random error in linear model (1). The parameters are set as follows:

• ELE linkage model: blockwise linkage accuracy $\gamma_{i}$ characterizing $Q$, block size $n_{i}=25$.

  • Settings 1 and 2: $\gamma_{i}\in\text{uniform}(0.6,0.9)$, $M=1$ in Assumption (A2).

  • Setting 3: the linkage accuracy $\gamma_{i}\equiv\gamma$ which varies from 0.6 to 1, while $M$ scales from 1 to 0.

• regression model: $x_{1},\dots,x_{n}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}\text{uniform}(-1,1)$, true regression coefficient $\beta=1$.

  • Setting 1: $n$ varies from 3,000 to 10,000, $\sigma$ is fixed at 1.

  • Setting 2: $n$ is fixed at $10,000$, $\sigma$ varies from 0.5 to 1.8.

  • Setting 3: $n$ is fixed at $10,000$, $\sigma$ is fixed at 1.

• privacy budget: $\epsilon=1$, $\delta=1/n^{1.1}$.

(a) Setting 1: $\sigma=1$

(b) Setting 1: $\sigma=1$

(c) Setting 2: $n=10,000$

(d) Setting 2: $n=10,000$

(e) Setting 3: $\sigma=1$, $n=10,000$

(f) Setting 3: $\sigma=1$, $n=10,000$

Figure 3. Average $\ell_{2}$-error and variance (theoretical versus empirical), with $(\epsilon,\delta)=(1,8.5\times 10^{-5})$, against $n$ and $\sigma$, respectively.
The “RL-NGD” and “RL-SSP” algorithms are our proposed post-RL approaches applied to the linked data, compared with the non-RL “NGD” and “SSP” methods applied to $(X,\bm{y})$ (i.e., with no linkage errors). The non-private “OLS” and “RL-OLS” (lahiri2005) results are also plotted for benchmarking. The number of iterations for “RL-NGD” results fall within the range of $(210,260)$.

In setting 1, where $\sigma$ is fixed at 1, Figure 3(a) shows the errors of all methods decrease with a growing sample size. Due to the linkage errors, the post-RL methods, including ${\hat{\bm{\beta}}^{\text{RL}}}$ and our two algorithms (denoted as “RL-OLS”, “RL-NGD”, and “RL-SSP” in the figures) run on the linked data $(X,\bm{z})$, naturally always yield larger errors than their counterparts run on $(X,\bm{y})$ when no linkage has to be done beforehand. In this case, with $\sigma=1$, post-RL SSP outperforms post-RL NGD in terms of both accuracy and variance. However, as $\sigma$ increases, post-RL NGD algorithm starts perform better, as depicted in Figure 3(c) with varying $\sigma$. The error grows linearly for post-RL NGD and quadratically for post-RL SSP, which aligns with the theoretical results on the error bounds presented in Section 4.2. Similar trends are observed for comparison of the non-RL NGD and SSP algorithms. In Figure 3(e), where linkage error tends to zero, the post-RL versions of the three estimators approach the corresponding non-RL versions. NGD and SSP methods have strictly larger error than OLS due to the cost of privacy.

Figures 3(b), 3(d) and 3(f) illustrate the empirical variances (EMP) against the theoretical variances (THR) of the proxy estimators given in Section 4.3. The theoretical variance of post-RL NGD closely aligns with the empirical variance at the chosen level of projection $C$. Recall that the theoretical variance would be exact when no projection is applied. Thus, with a lower level of projection on the gradient update, we anticipate it to be conservative. On the other hand, the theoretical variance of post-RL SSP approximates well with moderately large $n$ and small $\sigma$. However, in scenarios with small $n$ and/or large $\sigma$, our theoretical variance may underestimate the reality due to the approximation’s reliance on a first-order Taylor expansion. Therefore, one can include higher-order terms for better approximation. In setting 3, where $n$ and $\sigma$ are fixed, as the linkage error vanishes, the variance reduces as a result of the smaller DP noise needed.

5.2. Application to Synthetic Data

Due to privacy concerns, pairs of datasets containing personal information, which serve as quasi-identifiers, are typically not made public. We instead synthesize from a pair of generated quasi-identifiers datasets and real data for regression, as in Chambers2019SmallAE. For quasi-identifiers, we take advantage of the datasets generated by the Freely Extensible Biomedical Record Linkage (Febrl), which are available in the module RecordLinkage by rlpython in Python. The pair of datasets for linkage we use correspond to 5000 individuals. The domain indicator (state) is used for blocking. The record linkage is performed based on the Jaro-Winkler score (Jaro1989AdvancesIR) or exact comparison on 6 quasi-identifiers (names, date of birth, address, etc.). The maximum score is 6 for pairs that have exact alignment. A threshold of 4 is chosen to link the records. For those left unlinked, we assign random links to ensure one-to-one linkage. A unique identifier is available in the dataset for verification purposes. The resulting linkage accuracy for the 9 blocks are $\bm{\gamma}=(0.880,0.903,0.918,0.938,0.905,0.875,0.898,0.917,0.898)$, making the overall accuracy 92.5%. We adopt the ELE model for $Q$ and estimate it using $\bm{\gamma}$.

On the other hand, an anonymous dataset for regression comes from the Survey on Household Income and Wealth (SHIW) from the italydataset. The net disposable income and consumption are the explanatory variable $X$ and the response $\bm{y}$, respectively. Since the SHIW dataset is larger, consisting of 8151 data points, we drop the outliers and randomly draw 5000 records and synthesize them with the Febrl dataset. Figure 4 depicts the setup of the synthesization process. Using the unique identifier from the Ferlb dataset, the regression variables $(X,\bm{y})$ are appended to the quasi-identifiers $(\Phi_{A},\Phi_{B})$, resulting in two separate datasets: $(\Phi_{A},X)$ and $(\Phi_{B},\bm{y})$. Then, record linkage is performed by comparing $\Phi_{A}$ and $\Phi_{B}$ to output the linked data $(X,\bm{z})$ and the matrix $Q$.

Figure 4. Synthesization. The Ferlb dataset provides quasi-identifiers $(\Phi_{A},\Phi_{B})$, and the SHIW dataset provides regression variables $(X,\bm{y})$.

To apply the proposed DP algorithms to the synthesized dataset, we set the (hyper)parameters as follows. The privacy budget is given by $(\epsilon,\delta)=(1,8.5\times 10^{-5})$. The variance of the random error, $\sigma^{2}$, is estimated by the MSE. The upper bounds in Assumptions (A1)-(A3) are set as: $M=1$, $c_{0}=1$, $c_{x}=\max(X)=2.78$. In the NDG method, the projection level $C$ is set to 1.2.

To illustrate the importance of propagating linkage uncertainty when conducting downstream regression, we also apply the non-RL version of NGD and SSP algorithms. We obtain the non-RL regression results by running post-RL NGD and post-RL SSP methods with $M$ set to 0 and without converting $X$ into $W$. This is equivalent to applying the non-RL methods discussed in Cai2019TheCO; Sheffet17 to the linked set $(X,\bm{z})$ as if it were perfectly linked.

Figure 5. Boxplots of DP estimates based on 1000 repetitions with $(\epsilon,\delta)=(1,8.5\times 10^{-5})$.
The red dashed line indicates the OLS estimate. The proposed post-RL algorithms are compared with the non-RL “NGD” and “SSP” methods applied to $(X,\bm{z})$ (i.e., without accounting present linkage errors). The third and fourth columns represent the two NGD methods running for $T=\lceil L^{2}\ln(c_{0}^{2}n)/3\rceil$ iterations.

Figure 5 displays the boxplots of the estimates of each algorithm. For each algorithm, a total of 1000 repetitions are done in order to reflect the randomness of the injected noise for privacy purposes. The variables $X$ and $y$ are standardized before conducting simple linear regression. The OLS estimator on $(X,\bm{y})$ (dashed line) is plotted for comparison. As can be seen, the DP estimators by running (non-RL) NGD and SSP on $(X,\bm{z})$ directly are excessively biased as a consequence of ignoring linkage errors, even when the overall linkage accuracy is as high as 92.5%. Conversely, the results of post-RL NGD and post-RL SSP yields estimates centered around the OLS estimator but with higher variances, attributed to the cost of bias correction. Post-RL NGD is more flexible due to hyperparameter tuning. Additionally, we run the NGD methods for fewer iterations with $T=\lceil L^{2}\ln(c_{0}^{2}n)/3\rceil$, which is one-third of the value recommended by theory. We have found that this approach yields smaller variance while still producing accurate results in finite samples. Therefore, the theoretical number of iterations $T=\lceil L^{2}\ln(c_{0}^{2}n)\rceil$ may be conservative in some circumstances. Moderately reducing $T$ may lead to better results.

6. Discussion

In this paper, we propose two differentially private algorithms for linear regression on a linked dataset that contains linkage errors, by leveraging the existing work on (1) linear regression after record linkage, and (2) differentially private linear regression. Figure 6 displays the connections among the related areas at a high level, including PPRL and SMPC mentioned in Sections 1 and 2.3. Our work is the first one to simultaneously consider the linkage uncertainty propagation and the privatization of the output. It also complements the area of PPRL where the main concern is the data leakage among different parties. However, we do not discuss how to link the records in the first place and thus the security issues of the linkage process are beyond our scope. Instead, we treat record linkage from a secondary perspective: we begin with linked data prepared by an external entity and we have limited information about the linkage quality.

Figure 6. Diagram of related research areas. A secure protocol ensures no data is revealed to external parties during the linkage process.

Specifically, we propose two post-RL algorithms based on the noisy gradient descent and sufficient statistics perturbation methods from the DP literature. We provide privacy guarantees and finite-sample error bounds for these algorithms and discuss the variances of the private estimators. Our simulation studies and the application demonstrate the following: (1) the proposed estimators converge as the sample size increases; (2) post-RL linear regression incurs a higher cost than the non-RL counterpart in terms of the privacy-accuracy tradeoff; (3) The NGD method is flexible with hyperparameter tuning and can be applied to more general optimization problems; (4) SSP is specific to the least-squares problem, offering greater budget efficiency and more accurate results provided that the random error of the regression model is not too large.

There are different directions to extend our work. Note that there may be different scenarios of linking between the two datasets of the same set of entities. Assuming one-to-one linkage, as in our paper, is a canonical scenario. Although we do not explore it, we expect that our methods can be extended to other scenarios (e.g., one-to-many linkage) where $Q$ still makes sense. Extra assumptions may be required when determining the relevant sensitivities for privacy purposes.

One can also consider record linkage from a primary perspective. In addition to the traditional Fellegi–Sunter model, Bayesian approaches and machine learning-based methods have gained popularity. The record linkage may take forms other than the matching probability matrix adopted here. Furthermore, when privacy concerns arise during the linkage process involving different parties, PPRL and SMPC protocols become essential. Tackling all the challenges depicted in Figure 6 simultaneously with a single efficient tool is of great practical use and significance. This interdisciplinary challenge requires expertise in both statistics and computer science.

Another important direction is exploring related statistical problems in the post-RL context, with or without privacy constraints. For example, confidence intervals and hypothesis testing are fundamental statistical inference tools. Other potential problems that interest statisticians include high-dimensional linear regression and ridge regression.

Appendix A Proofs

A.1. Lemmas

The lemmas here support the proofs in Section A.2.

Lemma A.1.

If the minimum and maximum eigenvalues of $W^{\top}W/n$ satisfy $0<a<\lambda_{\min}(W^{\top}W/n)\leq\lambda_{\max}(W^{\top}W/n)<b$ for some constant $1<L<\infty$, then the loss function $\mathcal{L}_{n}(\bm{\beta})=\frac{1}{2n}(\bm{z}-W\bm{\beta})^{\top}(\bm{z}-W\bm{\beta})$ is $b$-smooth and $a$-strongly convex.

Proof.

Since $\nabla\mathcal{L}_{n}(\bm{\beta})=\frac{1}{n}(W^{\top}W\bm{\beta}-W^{\top}\bm{z})$, then for any $\bm{\beta}_{1},\bm{\beta}_{2}$,

$$(\nabla\mathcal{L}_{n}(\bm{\beta}_{1})-\nabla\mathcal{L}_{n}(\bm{\beta}_{2}))^{\top}(\bm{\beta}_{1}-\bm{\beta}_{2}) =(\bm{\beta}_{1}-\bm{\beta}_{2})^{\top}\frac{W^{\top}W}{n}(\bm{\beta}_{1}-\bm{\beta}_{2})$$

$$\geq\lambda_{\min}\left(\frac{W^{\top}W}{n}\right)\|\bm{\beta}_{1}-\bm{\beta}_{2}\|^{2}$$

$$\geq a\|\bm{\beta}_{1}-\bm{\beta}_{2}\|^{2}.$$

By definition, $\mathcal{L}_{n}(\bm{\beta})$ is $a$-strongly convex.

For smoothness, we have

$$\|\nabla\mathcal{L}_{n}(\bm{\beta}_{1})-\nabla\mathcal{L}_{n}(\bm{\beta}_{2})\| =\left\|\frac{W^{\top}W}{n}(\bm{\beta}_{1}-\bm{\beta}_{2})\right\|$$

$$\leq\left\|\frac{W^{\top}W}{n}\right\|\|\bm{\beta}_{1}-\bm{\beta}_{2}\|$$

$$=\lambda_{\max}\left(\frac{W^{\top}W}{n}\right)\|\bm{\beta}_{1}-\bm{\beta}_{2}\|$$

$$\leq b\|\bm{\beta}_{1}-\bm{\beta}_{2}\|.$$

The second equality is due to the fact that $\|A\|=|\lambda_{\max}(A)|$ for symmetric matrix $A$. By definition, $\mathcal{L}_{n}(\bm{\beta})$ is $b$-smooth.

One can have a neater proof using alternative definitions (See Eq. (4) and (10) in lecture_smoothness2 for a twice differentiable function:

(1)$f$ is $\mu$–strongly convex if $\lambda_{\min}(\nabla^{2}f)\geq\mu$;

(1)$f$ is $L$-smooth if $\lambda_{\max}(\nabla^{2}f)\leq L$.

∎

Lemma A.2 (Bubeck2015, Proof of Theorem 3.10.).

Let $f$ be $\alpha$-strongly convex and $\beta$-smooth on $\mathcal{X}$, and $x^{*}$ be the minimizer of $f$ on $\mathcal{X}$. Then projected gradient descent with step size $\eta=\frac{1}{\beta}$ satisfies for $t\geq 0$,

$$\|x_{t+1}-x^{*}\|^{2}\leq\left(1-\frac{\alpha}{\beta}\right)\|x_{t}-x^{*}\|^{2}.$$

Lemma A.3.

$\|\bm{a}+\bm{b}\|^{2}\leq(1+c^{2})\|\bm{a}\|^{2}+(1+1/c^{2})\|\bm{b}\|^{2}$ for any scalar $c\neq 0$.

Proof.

Since

$$c^{2}\|\bm{a}\|^{2}+\frac{1}{c^{2}}\|\bm{b}\|^{2}-2\|\bm{a}\|\|\bm{b}\|=\left(\|c\bm{a}\|-\left\|\frac{1}{c}\bm{b}\right\|\right)^{2}\geq 0,$$

it follows that

$$\|\bm{a}+\bm{b}\|^{2}\leq(\|\bm{a}\|+\|\bm{b}\|)^{2}\leq\|\bm{a}\|^{2}+\|\bm{b}\|^{2}+2\|\bm{a}\|\|\bm{b}\|\leq(1+c^{2})\|\bm{a}\|^{2}+(1+\frac{1}{c^{2}})\|\bm{b}\|^{2}.$$

∎

Lemma A.4 (Cai2019TheCO, Lemma A.2).

For $X_{1},...,X_{k}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}\chi^{2}_{d}$, $\zeta>0$, $0<\rho<1$,

$$\mathbb{P}\left(\sum_{j=1}^{k}\zeta\rho^{j}X_{j}>\frac{\rho\zeta d}{1-\rho}+s\right)\leq\exp\left(-\min\left(\frac{(1-\rho^{2})s^{2}}{8\rho^{2}\zeta^{2}d},\frac{s}{8\rho\zeta}\right)\right).$$

Lemma A.5 (Sheffet17, Proof of Proposition D.2).

For any invertible matrix $A$ and any matrix $B$ such that $(I+BA^{-1})$ is invertible,

$$(A+B)^{-1}=A^{-1}-A^{-1}(I+BA^{-1})^{-1}BA^{-1}.$$

Lemma A.6.

Let X be a $d\times d$ symmetric random matrix with i.i.d upper triangle entries. Each entry has mean 0 and variance $\sigma^{2}$. Let $\bm{y}$ be a $d$-dimensional random vector, which has mean $\bm{\mu}$ and covariance matrix $\Sigma$. Let $\Sigma_{X\bm{y}}$ denote the covariance matrix of $X\bm{y}$. Then, the diagonal entries of $\Sigma_{X\bm{y}}$ are given by

(23) $$(\Sigma_{X\bm{y}})_{kk}=\sigma^{2}\sum_{i=1}^{d}(\mu_{i}^{2}+\Sigma_{ii});$$

the off-diagonal entries are

(24) $$(\Sigma_{X\bm{y}})_{kl}=\sigma^{2}(\mu_{k}\mu_{l}+\Sigma_{kl})\text{ for }k\neq l.$$

Proof.

Let $X=(\bm{x}_{1},\bm{x}_{2},...,\bm{x}_{d})$ where $\bm{x}_{i}=(x_{i1},x_{i2},...,x_{id})^{T}$ and $y=(y_{1},y_{2},...,y_{d})^{T}$. Then,

(25) $$X\bm{y}=\sum_{i=1}^{d}\bm{x}_{i}y_{i}.$$

Therefore,

(26) $$\operatorname{Var}(X\bm{y})=\operatorname{Var}\left(\sum_{i=1}^{d}\bm{x}_{i}y_{i}\right)=\sum_{i=1}^{d}\operatorname{Var}(\bm{x}_{i}y_{i})+\sum_{i\neq j}^{d}\operatorname{Cov}(\bm{x}_{i}y_{i},\bm{x}_{j}y_{j}).$$

For the first term,

$$\operatorname{Var}(\bm{x}_{i}y_{i}) =\mathbb{E}[\operatorname{Var}(\bm{x}_{i}y_{i}\mid y_{i})]+\operatorname{Var}[\mathbb{E}(\bm{x}_{i}y_{i}\mid y_{i})]$$

$$=\mathbb{E}[y_{i}^{2}\operatorname{Var}(\bm{x}_{i}\mid y_{i})]+\operatorname{Var}[y_{i}\mathbb{E}(\bm{x}_{i}\mid y_{i})]$$

$$=\mathbb{E}(y_{i}^{2})\operatorname{Var}(\bm{x}_{i})+\operatorname{Var}(y_{i})\mathbb{E}(\bm{x}_{i})$$

$$=\sigma^{2}(\mu_{i}^{2}+\Sigma_{ii})I_{d}.$$

For $i\neq j$,

$$\operatorname{Cov}(\bm{x}_{i}y_{i},\bm{x}_{j}y_{j}) =\mathbb{E}[\bm{x}_{i}y_{i}(\bm{x}_{j}y_{j})^{T}]-\mathbb{E}(\bm{x}_{i}y_{i})\mathbb{E}(\bm{x}_{j}y_{j})$$

$$=\mathbb{E}[(y_{i}y_{j})\bm{x}_{i}\bm{x}_{j}^{T}]-\mathbb{E}(\bm{x}_{i})\mathbb{E}(y_{i})\mathbb{E}(\bm{x}_{j})\mathbb{E}(y_{j})$$

$$=\mathbb{E}(y_{i}y_{j})\mathbb{E}(\bm{x}_{i}\bm{x}_{j}^{T}),$$

where

$$\mathbb{E}(y_{i}y_{j})=\mathbb{E}(y_{i})\mathbb{E}(y_{j})+\operatorname{Cov}(y_{i},y_{j})=\mu_{i}\mu_{j}+\Sigma_{ij},$$

and $\mathbb{E}(\bm{x}_{i}\bm{x}_{j}^{T})$ is a $d\times d$ matrix with the $(j,i)$ entry being $\sigma^{2}$ and 0 other wise. Putting $\operatorname{Var}(\bm{x}_{i}y_{i})$ and $\operatorname{Cov}(\bm{x}_{i}y_{i},\bm{x}_{j}y_{j})$ back in (26), we know that $\operatorname{Var}(X\bm{y})$ has the diagonal entries

$$\sigma^{2}\sum_{i=1}^{d}(\mu_{i}^{2}+\Sigma_{ii})$$

and the $(k,l)$ off-diagonal entry

$$\sigma^{2}(\mu_{k}\mu_{l}+\Sigma_{kl}).$$

∎

Remark A.1.

In Lemma A.6, $\operatorname{Var}(X\bm{y})$ is given by $\sigma^{2}(\bm{\mu}\bm{\mu}^{T}+\Sigma)$ with the diagonal entries replaced by its trace.

Lemma A.7.

Let $X$ be a $d\times d$ random matrix with $\mathbb{E}X=0_{d\times d}$. Let $\bm{y}$ be a $d$-dimensional random vector that is independent of $X$. Then, $\operatorname{Cov}(\bm{y},X\bm{y})=0_{d\times d}$.

Proof.

$$\operatorname{Cov}(\bm{y},X\bm{y}) =\mathbb{E}[(\bm{y}-\mathbb{E}\bm{y})(X\bm{y}-\mathbb{E}X\bm{y})^{\top}]$$

$$=\mathbb{E}[(\bm{y}-\mathbb{E}\bm{y})(X\bm{y}-\mathbb{E}X\mathbb{E}\bm{y})^{\top}]$$

$$=\mathbb{E}[(\bm{y}-\mathbb{E}\bm{y})(X\bm{y})^{\top}]$$

$$=\mathbb{E}\bm{y}\bm{y}^{\top}\mathbb{E}X^{\top}-\mathbb{E}\bm{y}\mathbb{E}\bm{y}^{\top}\mathbb{E}X^{\top}=0_{d\times d}.$$

∎

Lemma A.8.

Let $X$ be a $d\times d$ random matrix with $\mathbb{E}X=0_{d\times d}$. Let $\bm{y}$ be a $d$-dimensional random vector. Let $\bm{z}$ be another $d$-dimensional random vector that is independent of both $X$ and $\bm{y}$ and $\mathbb{E}\bm{z}=0_{d}$. Then, $\operatorname{Cov}(X\bm{y},X\bm{z})=0_{d\times d}$.

Proof.

$$\operatorname{Cov}(X\bm{y},X\bm{z}) =\mathbb{E}[(X\bm{y}-\mathbb{E}X\bm{y})(X\bm{z}-\mathbb{E}X\bm{z})^{\top}]$$

$$=\mathbb{E}[(X\bm{y}-\mathbb{E}X\mathbb{E}\bm{y})(X\bm{z}-\mathbb{E}X\mathbb{E}\bm{z})^{\top}]$$

$$=\mathbb{E}[X\bm{y}\bm{z}^{\top}X^{\top}].$$

Since $\bm{z}$ is independent of both $X$ and $\bm{y}$ and the entries of $\bm{z}$ appear linearly in every entry of $X\bm{y}\bm{z}^{\top}X^{\top}$. By the zero-expectation of $\bm{z}$, $\operatorname{Cov}(X\bm{y},X\bm{z})=0_{d\times d}$.

∎

A.2. Proofs

This section provides the proofs for theorems and lemmas presented in the paper.

To derive this tighter composition for $(\epsilon,\delta)$-DP, we utilize the notion of zero-concentrated differential privacy (zCDP, Bun2016ConcentratedDP), defined as follows.

Definition 3 ($\rho$-zCDP).

A randomized mechanism $M:\mathcal{X}^{n}\rightarrow\mathcal{Y}$ is $\rho$-zero-concentrated-differentially private ($\rho$-zCDP) if, for all $\bm{x}\sim\bm{x}^{\prime}\in\mathcal{X}^{n}$ differing on a single entry and all $\alpha\in(1,\infty)$,

$$\operatorname{D}_{\alpha}(M(\bm{x})\|M(\bm{x}^{\prime}))\leq\rho\alpha,$$

where $\operatorname{D}_{\alpha}(M(\bm{x})\|M(\bm{x}^{\prime}))$ is the $\alpha$-Rényi divergence van2014renyi between the distribution of $M(\bm{x})$ and the distribution of $M(\bm{x}^{\prime})$.

Proof of Lemma 10.

Bun2016ConcentratedDP have shown that, like the classic $(\epsilon,\delta)$-DP notion, $\rho$-zCDP enjoys properties including basic composition and post-processing. The corresponding Gaussian mechanism (Proposition 1.6, Bun2016ConcentratedDP) states that an algorithm $f$ is $\rho$-zCDP after adding Gaussian noise $\mathcal{N}\left(0,\frac{\Delta_{f}^{2}}{2\rho}\right)$ to it. In addition, they have shown that $\rho$-zCDP implies $(\rho+2\sqrt{\rho\ln(1/\delta}),\delta)$-DP (Proposition 1.3, Bun2016ConcentratedDP).

Therefore, to achieve $(\epsilon,\delta)$-DP, it suffices for the algorithm to be $\rho$-zCDP with $\rho:=\epsilon+2\ln(1/\delta)-2\sqrt{(\epsilon+\ln(1/\delta))\ln(1/\delta)}$. Using the Gaussian mechanism and basic composition rule of $\rho$-zCDP, it suffices to add noise

$$u_{t}\sim\mathcal{N}\left(0,\frac{T\Delta_{t}^{2}}{2\rho}\right)$$

to $f_{t}$ for all $t$. Note that if $\epsilon\leq\frac{8\ln(1/\delta)}{2+\sqrt{2}}$, then $\rho\geq\frac{\epsilon^{2}}{8\ln(1/\delta)}$. Therefore, it suffices to add noise

$$u_{t}\sim\mathcal{N}\left(0,\frac{4T\Delta_{t}^{2}\ln(1/\delta)}{\epsilon^{2}}\right)$$

to $f_{t}$ for all $t$.

∎

Proof of Theorem 4.1(i).

By the composition proposition of differential privacy, to establish that Algorithm 1 (Post-RL Noisy Gradient Descent) satisfies $(\epsilon,\delta)$-differential privacy it suffices to show that the computation of $\beta^{t+1}$ is $(\epsilon/T,\delta/T)$-differentially private. According to the Gaussian mechanism Theorem 2.1, showing the latter boils down to proving that the sensitivity is controlled at each gradient step. Let $g^{t+1}(\bm{\beta}^{t};X,\bm{y},Q)=\sum_{i=1}^{n}(\bm{w}_{i}^{\top}\bm{\beta}^{t}-\Pi_{R}(z_{i}))\bm{w}_{i}$. We will show that the $\ell_{2}$-sensitivity of $g^{t+1}$, denoted by $\Delta_{g}$, is bounded by $B$.

Without loss of generality, we assume the neighboring data sets $(X,\Phi_{X},\bm{y},\Phi_{\bm{y}})$ and $(X^{\prime},\Phi_{X^{\prime}},\bm{y}^{\prime},\Phi_{\bm{y}^{\prime}})$ differ in the $j$-th record. Recall $\bm{z}$ is a permutation of $\bm{y}$ satisfying $q_{ij}=P(z_{i}=y_{i})$. Let $\bm{z}^{\prime}$ be a copy of $\bm{z}$ but with the entry $y_{j}$ changed to $y^{\prime}_{j}$. Recall in the record linkage linear model elaborated in lahiri2005, $\bm{w}_{i}=\sum_{j=1}^{n}q_{ij}\bm{x}_{j}$, which are convex combinations of rows of $X$. We can write $\bm{w}_{i}=\sum_{k\neq j}q_{ik}\bm{x}_{k}+q_{ij}\bm{x}_{j}$ and $\bm{w}^{\prime}_{i}=\sum_{k\neq j}q^{\prime}_{ik}\bm{x}_{k}+q^{\prime}_{ij}\bm{x}^{\prime}_{j}$. From assumption (A1) that $\|\bm{x}\|<c_{\bm{x}}$ with probability 1, we know $\|\bm{w}_{i}\|<c_{\bm{x}}$ almost surely. Then, we have

(27) $$\|\bm{w}_{i}-\bm{w}^{\prime}_{i}\| =\left\|\left(\sum_{k\neq j}q_{ik}\bm{x}_{k}+q_{ij}\bm{x}_{j}\right)-\left(\sum_{k\neq j}q^{\prime}_{ik}\bm{x}_{k}+q^{\prime}_{ij}\bm{x}^{\prime}_{j}\right)\right\|$$

$$=\left\|\sum_{k\neq j}(q_{ik}-q^{\prime}_{ik})\bm{x}_{k}+(q_{ij}\bm{x}_{j}-q^{\prime}_{ij}\bm{x}_{j}^{\prime})\right\|$$

$$=\left\|\sum_{k\neq j}(q_{ik}-q^{\prime}_{ik})\bm{x}_{k}+(q_{ij}-q^{\prime}_{ij})\bm{x}_{j}+q^{\prime}_{ij}(\bm{x}_{j}-\bm{x}_{j}^{\prime})\right\|$$

$$\leq\left\|\sum_{k=1}^{n}(q_{ik}-q^{\prime}_{ik})\bm{x}_{k}\right\|+\left\|q^{\prime}_{ij}(\bm{x}_{j}-\bm{x}_{j}^{\prime})\right\|$$

$$\leq c_{\bm{x}}\left(\sum_{k=1}^{n}|q_{ik}-q^{\prime}_{ik}|+2q^{\prime}_{ij}\right).$$

Since $Q^{\prime}$ is a doubly stochastic matrix, $\sum_{i=1}^{n}q^{\prime}_{ij}=1$ for any $j$. By the arbitrariness of index $j$, it follows that

(28) $$\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\sum_{i=1}^{n}\|\bm{w}_{i}-\bm{w}^{\prime}_{i}\|\leq\sum_{i=1}^{n}c_{\bm{x}}\left(\sum_{k=1}^{n}|q_{ik}-q^{\prime}_{ik}|+2q^{\prime}_{ij}\right)=c_{\bm{x}}(\|Q-Q^{\prime}\|_{1}+2).$$

The sensitivity of $g^{t+1}$ is

(29) $$\Delta_{g} =\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\left\|g^{t+1}(\bm{\beta}^{t};X,\bm{y},Q)-g^{t+1}(\bm{\beta}^{t};X^{\prime},\bm{y}^{\prime},Q^{\prime})\right\|$$

$$=\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\left\|\sum_{i=1}^{n}(\bm{w}_{i}^{\top}\bm{\beta}^{t}-\Pi_{R}(z_{i}))\bm{w}_{i}-\sum_{i=1}^{n}(\bm{w}_{i}^{\prime T}\bm{\beta}^{t}-\Pi_{R}(z^{\prime}_{i}))\bm{w}^{\prime}_{i}\right\|$$

$$\leq\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\sum_{i=1}^{n}\left\|\bm{w}_{i}^{\top}\bm{\beta}^{t}\bm{w}_{i}-\bm{w}_{i}^{\prime T}\bm{\beta}^{t}\bm{w}^{\prime}_{i}\right\|+\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\sum_{i=1}^{n}\left\|\Pi_{R}(z_{i})\bm{w}_{i}-\Pi_{R}(z^{\prime}_{i})\bm{w}^{\prime}_{i}\right\|$$

$$=:\Delta_{1}+\Delta_{2}.$$

We use $\Delta_{1}$ and $\Delta_{2}$ to denote the two terms on the right of (29), respectively. To bound the first term $\Delta_{1}$, since

$$\|\bm{w}_{i}^{\top}\bm{\beta}^{t}\bm{w}_{i}-\bm{w}_{i}^{\prime T}\bm{\beta}^{t}\bm{w}_{i}^{\prime}\| =\|\bm{w}_{i}^{\top}\bm{\beta}^{t}(\bm{w}_{i}-\bm{w}_{i}^{\prime}+\bm{w}_{i}^{\prime})-\bm{w}_{i}^{\prime T}\bm{\beta}^{t}\bm{w}_{i}^{\prime}\|$$

$$=\|\bm{w}_{i}^{\top}\bm{\beta}^{t}(\bm{w}_{i}-\bm{w}_{i}^{\prime})+\bm{w}_{i}^{\top}\bm{\beta}^{t}\bm{w}_{i}^{\prime}-\bm{w}_{i}^{\prime T}\bm{\beta}^{t}\bm{w}_{i}^{\prime}\|$$

$$=\|\bm{w}_{i}^{\top}\bm{\beta}^{t}(\bm{w}_{i}-\bm{w}_{i}^{\prime})+(\bm{w}_{i}-\bm{w}_{i}^{\prime})^{T}\bm{\beta}^{t}\bm{w}_{i}^{\prime}\|$$

$$\leq\|\bm{w}_{i}^{\top}\bm{\beta}^{t}(\bm{w}_{i}-\bm{w}_{i}^{\prime})\|+\|(\bm{w}_{i}-\bm{w}_{i}^{\prime})^{T}\bm{\beta}^{t}\bm{w}_{i}^{\prime}\|$$

$$\leq\|\bm{w}_{i}\|\|\bm{\beta}^{t}\|\|\bm{w}_{i}-\bm{w}_{i}^{\prime}\|+\|\bm{w}_{i}-\bm{w}_{i}^{\prime}\|\|\bm{\beta}^{t}\|\|\bm{w}_{i}^{\prime}\|,$$

then by (28), $\Delta_{1}$ can be controlled:

(30) $$\Delta_{1} \leq\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\sum_{i=1}^{n}2Cc_{\bm{x}}\|\bm{w}_{i}-\bm{w}_{i}^{\prime}\|\leq 2Cc^{2}_{\bm{x}}(\|Q-Q^{\prime}\|_{1}+2)\leq 2Cc^{2}_{\bm{x}}(M+2).$$

For the second term $\Delta_{2}=\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\sum_{i=1}^{n}\|\Pi_{R}(z_{i})\bm{w}_{i}-\Pi_{R}(z^{\prime}_{i})\bm{w}^{\prime}_{i}\|$, since

(31) $$\|\Pi_{R}(z_{i})\bm{w}_{i}-\Pi_{R}(z^{\prime}_{i})\bm{w}^{\prime}_{i}\| =\|\Pi_{R}(z_{i})(\bm{w}_{i}-\bm{w}^{\prime}_{i})+(\Pi_{R}(z_{i})-\Pi_{R}(z^{\prime}_{i}))\bm{w}^{\prime}_{i}\|$$

$$\leq\|\Pi_{R}(z_{i})(\bm{w}_{i}-\bm{w}^{\prime}_{i})\|+\|(\Pi_{R}(z_{i})-\Pi_{R}(z^{\prime}_{i}))\bm{w}^{\prime}_{i}\|$$

$$\leq R\|(\bm{w}_{i}-\bm{w}^{\prime}_{i})\|+c_{\bm{x}}\|\Pi_{R}(z_{i})-\Pi_{R}(z^{\prime}_{i})\|.$$

Then,

(32) $$\Delta_{2} \leq\max_{(X,\bm{y},Q)\sim(X^{\prime},\bm{y}^{\prime},Q^{\prime})}\sum_{i=1}^{n}\left(R\|(\bm{w}_{i}-\bm{w}^{\prime}_{i})\|+c_{\bm{x}}\|\Pi_{R}(z_{i})-\Pi_{R}(z^{\prime}_{i})\|\right)$$

$$=(Rc_{\bm{x}}(\|Q-Q^{\prime}\|_{1}+2)+2Rc_{\bm{x}}).$$

$$\leq Rc_{\bm{x}}(M+4).$$

It follows that

$$\Delta_{g}\leq\Delta_{1}+\Delta_{2}\leq 2Cc^{2}_{\bm{x}}(M+2)+Rc_{\bm{x}}(M+4).$$

∎

Proof of Theorem 4.1(ii).

We now show that Algorithm 2 (Post-RL sufficient statistics perturbation) is $(\epsilon,\delta)$-differentially private. Let $A=(W\mid\bm{z})$ be the augmented matrix considering linkage errors. Then $A^{\top}A$ contains sufficient statistics for the $\bm{\beta}$. Thus, it suffices to show that the sensitivity of $A^{\top}A$ is controlled by $B=Rc_{x}(M+4)+\max\{2c_{x}^{2}(M+2),2R^{2}\}$.

Let $A^{\prime}=(W^{\prime}\mid\bm{z}^{\prime})$, where $W^{\prime}$ and $\bm{z}^{\prime}$ come from any neighboring data set, as in the proof of Theorem 4.1 (i). We have

$$A^{\top}A-A^{\prime\top}A^{\prime} =\begin{pmatrix}W^{\top}W-W^{\prime\top}W^{\prime}&0\\ 0&\bm{z}^{\top}\bm{z}-\bm{z}^{\prime\top}\bm{z}^{\prime}\end{pmatrix}+\begin{pmatrix}0&W^{\top}\bm{z}-W^{\prime\top}\bm{z}^{\prime}\\ \bm{z}^{\top}W-\bm{z}^{\prime\top}W^{\prime}&0\end{pmatrix}$$

$$=:A_{1}+A_{2}.$$

By the properties of the norm of block matrices,

$$\|A_{1}\|\leq\max\{\|W^{\top}W-W^{\prime\top}W^{\prime}\|,\|\bm{z}^{\top}\bm{z}-\bm{z}^{\prime\top}\bm{z}^{\prime}\|\}$$

and

$$\|A_{2}\| =\|W^{\top}\bm{z}-W^{\prime\top}\bm{z}^{\prime}\|$$

$$=\|W^{\top}\bm{z}-W^{\prime\top}\bm{z}+W^{\prime\top}\bm{z}-W^{\prime\top}\bm{z}^{\prime}\|$$

$$\leq\|(W-W^{\prime})^{\top}\bm{z}\|+\|W^{\prime\top}(\bm{z}-\bm{z}^{\prime})\|.$$

We have

$$\|W^{\top}W-W^{\prime\top}W^{\prime}\| =\|\sum_{i}^{n}\bm{w}_{i}\bm{w}_{i}^{\top}-\sum_{i}^{n}\bm{w}^{\prime}_{i}\bm{w}^{\prime\top}_{i}\|$$

$$\leq\sum_{i}^{n}\|\bm{w}_{i}(\bm{w}_{i}-\bm{w}_{i}^{\prime})^{\top}+(\bm{w}_{i}-\bm{w}_{i}^{\prime})\bm{w}_{i}^{\prime\top}\|$$

$$\leq 2c_{x}\sum_{i}^{n}\|\bm{w}_{i}-\bm{w}_{i}^{\prime}\|$$

$$\leq 2c_{x}^{2}(M+2),$$

and

$$\|\bm{z}^{\top}\bm{z}-\bm{z}^{\prime\top}\bm{z}^{\prime}\|=|y^{2}_{j}-y^{\prime 2}_{j}|\leq 2R^{2}.$$

Note that we can swap the rows of $\bm{z}^{\prime}$ and $Q^{\prime}$, such that $\bm{z}$ and $\bm{z}^{\prime}$ only differ in one record and it does not change the estimation using $Q^{\prime}$ and $\bm{z}^{\prime}$ after swapping. Then,

$$\|A_{1}\|\leq\max\{2c_{x}^{2}(M+2),2R^{2}\}.$$

Since

$$\|(W-W^{\prime})^{\top}\bm{z}\|=\|\sum_{i}^{n}(\bm{w}_{i}-\bm{w}^{\prime}_{i})z_{i}\|\leq R\sum_{i}^{n}\|\bm{w}_{i}-\bm{w}_{i}^{\prime}\|\leq Rc_{x}(M+2)$$

and

$$\|W^{\prime\top}(\bm{z}-\bm{z}^{\prime})\| =\|\sum_{i}^{n}\bm{w}^{\prime}_{i}(z_{i}-z_{i}^{\prime})\| =\sum_{i}^{n}\|\bm{w}_{i}\||z_{i}-z_{i}^{\prime}| \leq c_{x}\sum_{i}^{n}|z_{i}-z_{i}^{\prime}| \leq 2Rc_{x},$$

we have

$$\|A_{2}\|\leq Rc_{x}(M+4).$$

Putting the upper bounds together, we derive

$$\max_{A,A^{\prime}}\|A^{\top}A-A^{\prime\top}A^{\prime}\|\leq Rc_{x}(M+4)+\max\{2c_{x}^{2}(M+2),2R^{2}\}.$$

∎

Proof of Lemma 4.2.

To establish the behavior of the expected squared-error loss of ${\hat{\bm{\beta}}^{\text{OLS}}}$, first note that since

$$0<\frac{1}{L}<d\lambda_{\min}\left(\frac{X^{\top}X}{n}\right)\leq d\lambda_{\max}\left(\frac{X^{\top}X}{n}\right)<L,$$

we have

$$\frac{d}{Ln}<\lambda_{\max}(X^{\top}X)^{-1}\leq\lambda_{\max}(X^{\top}X)^{-1}<\frac{dL}{n}.$$

Therefore, $\operatorname{tr}(X^{\top}X)^{-1}=\sum_{i=1}^{d}\lambda_{i}((X^{\top}X)^{-1})=\Theta(d^{2}/n)$ where $\lambda_{i},\quad i=1,\dots,d$ are the eigenvalues of $(X^{\top}X)^{-1}$. ∎

Proof of Lemma 4.3.

Note that ${\hat{\bm{\beta}}^{\text{RL}}}$ is an unbiased estimator for $\bm{\beta}$, and hence

(33) $$\mathbb{E}\|{{\hat{\bm{\beta}}^{\text{RL}}}-\bm{\beta}}\|^{2} =\mathbb{E}\left[\sum_{i=1}^{d}(\hat{\beta}_{i}^{\text{RL}}-\beta_{i})^{2}\right]$$

$$=\sum_{i=1}^{d}\mathbb{E}(\hat{\beta}_{i}^{\text{RL}}-\beta_{i})^{2}$$

$$=\sum_{i=1}^{d}\operatorname{Var}(\hat{\beta}_{i}^{\text{RL}})$$

$$=\operatorname{tr}(\Sigma^{\text{RL}}).$$

∎

Proof of Theorem 4.4.

To establish an upper bound of the excess error of the private estimator, i.e., $\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}$, for Algorithm 1, we work under the event $\mathcal{E}=\{\Pi_{R}(z_{i})=z_{i},\forall i\in[n]\}$. By the concentration bound of the Gaussian distribution, with the choice of $R=\sigma\sqrt{2\ln n}$, $\mathbb{P}(\mathcal{E})\geq 1-c_{1}\exp(-c_{2}\ln n)$ where $c_{1}$ and $c_{2}$ are constants.

Recall that the loss function $\mathcal{L}_{n}(\bm{\beta})\stackrel{{\scriptstyle def}}{{=}}\frac{1}{2n}(\bm{z}-W\bm{\beta})^{\top}(\bm{z}-W\bm{\beta})$. The assumption about the eigenvalues of $W^{\top}W/n$ implies that $\mathcal{L}_{n}(\bm{\beta})$ is $\frac{L}{d}$-smooth and $\frac{1}{dL}$-strongly convex. See the proof in Lemma A.1. Under $\mathcal{E}$, the iterate $\bm{\beta}^{t+1}=\Pi_{C}(\bm{\beta}^{t}-\eta\nabla\mathcal{L}_{n}(\bm{\beta}^{t})+\bm{u}_{t})$. Let $\hat{\bm{\beta}}^{t+1}=\Pi_{C}(\bm{\beta}^{t}-\eta\nabla\mathcal{L}_{n}(\bm{\beta}^{t}))$ be the unperturbed iterate, then $\|\bm{\beta}^{t+1}-\hat{\bm{\beta}}^{t+1}\|\leq\|\bm{u}_{t}\|$. Since (A3) says that $\|\bm{\beta}\|\leq c_{0}$, we can assume $\|{\hat{\bm{\beta}}^{\text{RL}}}\|\leq c_{0}$ without loss of generality where ${\hat{\bm{\beta}}^{\text{RL}}}=\underset{\bm{\beta}}{\arg\min}\mathcal{L}_{n}(\bm{\beta})$. Then, ${\hat{\bm{\beta}}^{\text{RL}}}$ is the same as ${\hat{\bm{\beta}}^{\text{RL}}}\stackrel{{\scriptstyle def}}{{=}}\underset{\|\bm{\beta}\|\leq C}{\arg\min}\mathcal{L}_{n}(\bm{\beta})$ by setting $C=c_{0}$. By Lemma A.2, with $\eta=\frac{d}{L}$, it then follows that

$$\|\hat{\bm{\beta}}^{t+1}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}\leq\left(1-\frac{1}{L^{2}}\right)\|{\bm{\beta}}^{t}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}.$$

Let $c\geq 2$ be some constant, then by Lemma A.3, we have the following for the noisy iterate ${\bm{\beta}}^{t+1}$:

$$\|\bm{\beta}^{t+1}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2} =\|\hat{\bm{\beta}}^{t+1}-{\hat{\bm{\beta}}^{\text{RL}}}+\bm{\beta}^{t+1}-\hat{\bm{\beta}}^{t+1}\|^{2}$$

$$\leq\left(1+\frac{1}{cL^{2}}\right)\|\hat{\bm{\beta}}^{t+1}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}+\left(1+cL^{2}\right)\|\bm{u}_{t}\|^{2}$$

$$\leq\left(1+\frac{1}{cL^{2}}\right)\left(1-\frac{1}{L^{2}}\right)\|{\bm{\beta}}^{t}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}+\left(1+cL^{2}\right)\|\bm{u}_{t}\|^{2}$$

$$\leq\left(1-\frac{c-1}{cL^{2}}\right)\|{\bm{\beta}}^{t}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}+\left(1+cL^{2}\right)\|\bm{u}_{t}\|^{2}.$$

The above recursive formula yields

$$\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}\leq\left(1-\frac{c-1}{cL^{2}}\right)^{T}\|{\bm{\beta}}^{0}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}+\left(1+cL^{2}\right)\sum_{k=0}^{T-1}\left(1-\frac{c-1}{cL^{2}}\right)^{k}\|\bm{u}_{T-1-k}\|^{2}.$$

Setting $\displaystyle T=\left\lceil\ln(c_{0}^{2}n)/\ln\left(1-\frac{c-1}{cL^{2}}\right)\right\rceil$, the first term $\displaystyle\left(1-\frac{c-1}{cL^{2}}\right)^{T}\|{\bm{\beta}}^{0}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}\leq\frac{1}{n}$, given that $\bm{\beta}^{0}=\bm{0}$ and $\|{\hat{\bm{\beta}}^{\text{RL}}}\|\leq c_{0}$. When $c$ is sufficiently large, $T$ can be set to $\left\lceil L^{2}\ln(c_{0}^{2}n)\right\rceil$.

To control the second term, we apply Lemma A.4 with $\rho=1-(c-1)/(cL^{2})$ and $\zeta=2{\eta}^{2}TB^{2}\frac{\ln(1/\delta)}{n^{2}\epsilon^{2}}$ which is the variance of $\bm{u}_{T-1-k}$ for $k=1,...,T-1$. Provided $\delta=o(1/n)$, let $s=K_{1}\zeta d$ for some sufficiently large constant $K_{1}$ so $\frac{\rho\zeta d}{1-\rho}+s=\Theta(\zeta d)$ where

$$\zeta d =4{\eta}^{2}TB^{2}\frac{\ln(1/\delta)}{n^{2}\epsilon^{2}}\cdot d$$

$$=4\left(\frac{d}{L}\right)^{2}\left\lceil L^{2}\ln(c_{0}^{2}n)\right\rceil(\sigma\sqrt{2\ln n}c_{\bm{x}}(M+4)+2c_{0}c^{2}_{\bm{x}}(M+2))^{2}\frac{\ln(1/\delta)}{n^{2}\epsilon^{2}}\cdot d$$

$$=O\left(\frac{\sigma^{2}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right).$$

That is, the noise term $\displaystyle\sum_{k=0}^{T-1}\left(1-\frac{c-1}{cL^{2}}\right)^{k}\|\bm{u}_{T-1-k}\|^{2}$ is then controlled by a corresponding big-O statement with probability at least $1-e^{-c_{3}d}$, where $c_{3}=\min\left(\frac{(1-\rho^{2})K_{1}^{2}}{8\rho^{2}},\frac{K_{1}}{8\rho}\right)$, hence

$$\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}=\frac{1}{n}+O\left(\frac{\sigma^{2}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right).$$

∎

Proof of Theorem 4.5.

Algorithm 2 is also analyzed under $\mathcal{E}=\{\Pi_{R}(z_{i})=z_{i},\forall i=1,\dots,n\}$ as in Theorem 4.5. Then, with $R=\sigma\sqrt{2\ln n}$, we have $\mathbb{P}(\mathcal{E})\geq 1-c_{1}\exp(-c_{2}\ln n)$.

By Lemma A.5,

(34) $$\quad(W^{\top}W+U)^{-1} =(W^{\top}W)^{-1}-(W^{\top}W)^{-1}\cdot(I+U(W^{\top}W)^{-1})^{-1}\cdot U(W^{\top}W)^{-1}.$$

Then,

(35) $${\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}} =(W^{\top}W+U)^{-1}(W^{\top}\bm{z}+\bm{u})-(W^{\top}W)^{-1}W^{\top}\bm{z}$$

$$=(W^{\top}W)^{-1}\bm{u}-(W^{\top}W)^{-1}\cdot(I+U(W^{\top}W)^{-1})^{-1}\cdot U(W^{\top}W)^{-1}(W^{\top}\bm{z}+\bm{u})$$

$$=(W^{\top}W)^{-1}\bm{u}-(W^{\top}W)^{-1}\cdot(I+U(W^{\top}W)^{-1})^{-1}\cdot U(\bm{\hat{\bm{\beta}}^{\text{RL}}}+(W^{\top}W)^{-1}\bm{u})$$

To bound $\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|$, we need to bound the norms of $(W^{\top}W)^{-1}$, $U$, $\bm{u}$, and $(I+U(W^{\top}W)^{-1})^{-1}$. First, by the assumption (A4):

$$0<\frac{1}{L}<d\lambda_{\min}\left(\frac{W^{\top}W}{n}\right)\leq d\lambda_{\max}\left(\frac{W^{\top}W}{n}\right)<L$$

for some constant $1<L<\infty$, we have

$$\frac{d}{Ln}\leq\lambda_{\min}(W^{\top}W)^{-1}\leq\lambda_{\max}(W^{\top}W)^{-1}\leq\frac{Ld}{n}.$$

Then,

(36) $$\|(W^{\top}W)^{-1}\|=\lambda_{\max}(W^{\top}W)^{-1}=O\left(\frac{d}{n}\right).$$

Recall from Algorithm 2 that $U$ is a Gaussian symmetric matrix with upper triangle given by iid $N(0,\omega).$ The Gaussian concentration bounds give that w.p. $\geq 1-e^{-c_{3}d}$, we have

(37) $$\|U\|=O(\omega\sqrt{d})=O\left(\frac{\sigma^{2}\ln n\sqrt{d\ln(1/\delta)}}{\epsilon}\right).$$

The result (37) is from random matrix theory (Vershynin, Corollary 4.4.8). Since the vector norm is bounded by the norm of the matrix that contains the vector as a column, thus we also have

(38) $$\|\bm{u}\|=O\left(\frac{\sigma^{2}\ln n\sqrt{d\ln(1/\delta)}}{\epsilon}\right).$$

For $\|(I+U(W^{\top}W)^{-1})^{-1}\|$, consider its Taylor series:

(39) $$(I+U(W^{\top}W)^{-1})^{-1}=\sum_{i=0}^{\infty}(-U(W^{\top}W)^{-1})^{i}$$

W. p. $\geq 1-e^{-c_{3}d}$,

$$\|U(W^{\top}W)^{-1}\|\leq\|U\|\|(W^{\top}W)^{-1}\|=O\left(\frac{\sigma^{2}d\ln n\sqrt{d\ln(1/\delta)}}{n\epsilon}\right)$$

is going to zero as $n\to\infty$. Therefore, the Taylor Series (39) converges. Then,

(40) $$\|(I+U(W^{\top}W)^{-1})^{-1}\| =\|\sum_{i=0}^{\infty}(-U(W^{\top}W)^{-1})^{i}\|$$

$$\leq\sum_{i=0}^{\infty}\|(-U(W^{\top}W)^{-1})^{i}\|$$

$$\leq\sum_{i=0}^{\infty}\|U(W^{\top}W)^{-1})\|^{i}$$

$$=\frac{1}{1-\|U(W^{\top}W)^{-1})^{-1}\|} =O(1)$$

w.p. $\geq 1-e^{-c_{3}d}$. Plugging all the bounds into (35), we have derived

$$\|{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}\|^{2}=O\left(\frac{\sigma^{4}d^{3}\ln^{2}n\ln(1/\delta)}{n^{2}\epsilon^{2}}\right).$$

∎

Proof of Corollary 4.1.

The proof is completed by using the inequality

$$\|{{\bm{\hat{\beta}}^{\text{priv}}}-\bm{\beta}}\|^{2} \leq 2(\|{{\hat{\bm{\beta}}^{\text{RL}}}-\bm{\beta}}\|^{2}+\|{{\bm{\hat{\beta}}^{\text{priv}}}-{\hat{\bm{\beta}}^{\text{RL}}}}\|^{2}).$$

∎

Proof of Theorem 4.6.

Consider the non-projected estimator

(41) $$\bm{\beta}^{t+1}=\bm{\beta}^{t}-\frac{\eta}{n}W^{\top}(W\bm{\beta}^{t}-\bm{z})+\bm{u}_{t}.$$

Let $\displaystyle A\stackrel{{\scriptstyle def}}{{=}}\frac{\eta}{n}W^{\top}W$ and $\displaystyle B\stackrel{{\scriptstyle def}}{{=}}\frac{\eta}{n}W$. From the recursive form $\bm{\beta}^{t+1}=(I_{d}-A)\bm{\beta}^{t}+B^{\top}\bm{z}+\bm{u}_{t}$, we derive for ${\bm{\hat{\beta}}^{\text{priv}}}\stackrel{{\scriptstyle def}}{{=}}\bm{\beta}^{T}$:

(42) $${\bm{\hat{\beta}}^{\text{priv}}}=(I_{d}-A)^{T}\bm{\beta}^{0}+\sum_{t=1}^{T}(I_{d}-A)^{t-1}(B^{\top}\bm{z}+\bm{u}_{T-t}).$$

Then,

(43) $$\operatorname{Var}({\bm{\hat{\beta}}^{\text{priv}}})=\sum_{t=1}^{T}(I_{d}-A)^{t-1}\cdot B^{\top}\Sigma_{\bm{z}}B\cdot\sum_{t=1}^{T}(I_{d}-A)^{t-1}+\omega^{2}\sum_{t=1}^{T}(I_{d}-A)^{2t-2}.$$

∎

Proof of Theorem 4.7.

By rewriting ${\bm{\hat{\beta}}^{\text{priv}}}$ as in (35) and ignoring the remainder of the first-order Taylor expansion in (39), we have

(44) $${\bm{\hat{\beta}}^{\text{priv}}} ={\hat{\bm{\beta}}^{\text{RL}}}+(W^{\top}W)^{-1}\bm{u}-(W^{\top}W)^{-1}U\bm{\hat{\bm{\beta}}^{\text{RL}}}-(W^{\top}W)^{-1}U(W^{\top}W)^{-1}\bm{u}.$$

Then,

(45) $$\operatorname{Var}({\bm{\hat{\beta}}^{\text{priv}}}) =\operatorname{Var}({\hat{\bm{\beta}}^{\text{RL}}})+\operatorname{Var}[(W^{\top}W)^{-1}\bm{u}]+\operatorname{Var}[(W^{\top}W)^{-1}U\bm{\hat{\bm{\beta}}^{\text{RL}}}]$$

$$\quad+\operatorname{Var}[(W^{\top}W)^{-1}U(W^{\top}W)^{-1}\bm{u}]-2\operatorname{Cov}({\hat{\bm{\beta}}^{\text{RL}}},(W^{\top}W)^{-1}U\bm{\hat{\bm{\beta}}^{\text{RL}}})$$

$$\quad-2\operatorname{Cov}((W^{\top}W)^{-1}\bm{u},(W^{\top}W)^{-1}U(W^{\top}W)^{-1}\bm{u})$$

$$\quad+2\operatorname{Cov}((W^{\top}W)^{-1}U\bm{\hat{\bm{\beta}}^{\text{RL}}},(W^{\top}W)^{-1}U(W^{\top}W)^{-1}\bm{u}).$$

Since $\bm{u}\sim\mathcal{N}(0,\omega^{2}I_{d})$, we have

$$\operatorname{Var}((W^{\top}W)^{-1}\bm{u})=\omega^{2}(W^{\top}W)^{-2}.$$

For the third term, let $\Sigma_{1}\stackrel{{\scriptstyle def}}{{=}}\operatorname{Var}(U\bm{\hat{\bm{\beta}}^{\text{RL}}})$. By Lemma A.6,

$$(\Sigma_{1})_{kk}=\omega^{2}\sum_{i=1}^{d}(\beta_{i}^{2}+\Sigma^{\text{RL}}_{ii})$$

and

$$(\Sigma_{1})_{kl}=\omega^{2}(\beta_{k}\beta_{l}+\Sigma^{\text{RL}}_{kl}).$$

For the fourth term, let $\Sigma_{2}\stackrel{{\scriptstyle def}}{{=}}\operatorname{Var}(U(W^{\top}W)^{-1}\bm{u})$ and let $\Sigma^{\prime}=\operatorname{Var}((W^{\top}W)^{-1}\bm{u})=\omega^{2}(W^{\top}W)^{-2}$.

$$(\Sigma_{2})_{kk}=\omega^{2}\sum_{i=1}^{d}\Sigma^{\prime}_{ii}$$

and

$$(\Sigma_{2})_{kl}=\omega^{2}\Sigma^{\prime}_{kl}.$$

On the other hand, the covariances in (45) are all zeros by Lemmas A.7 and A.8 due to the independencies and zero expectations of $U$ and $\bm{u}$. Then putting them together, we have

(46) $$\Sigma=\Sigma^{\text{RL}}+(W^{\top}W)^{-1}(\omega^{2}I_{d}+\Sigma_{1}+\Sigma_{2})(W^{\top}W)^{-1}.$$

Note that $\Sigma_{1}$ and $\Sigma_{2}$ have a common factor $\omega^{2}$. By rescaling $\Sigma_{1}$ and $\Sigma_{2}$, we have the expression for $\Sigma$ as stated in the theorem.

∎