-
A Fast Algorithm for Adaptive Private Mean Estimation
Authors:
John Duchi,
Saminul Haque,
Rohith Kuditipudi
Abstract:
We design an $(\varepsilon, δ)$-differentially private algorithm to estimate the mean of a $d$-variate distribution, with unknown covariance $Σ$, that is adaptive to $Σ$. To within polylogarithmic factors, the estimator achieves optimal rates of convergence with respect to the induced Mahalanobis norm $||\cdot||_Σ$, takes time $\tilde{O}(n d^2)$ to compute, has near linear sample complexity for su…
▽ More
We design an $(\varepsilon, δ)$-differentially private algorithm to estimate the mean of a $d$-variate distribution, with unknown covariance $Σ$, that is adaptive to $Σ$. To within polylogarithmic factors, the estimator achieves optimal rates of convergence with respect to the induced Mahalanobis norm $||\cdot||_Σ$, takes time $\tilde{O}(n d^2)$ to compute, has near linear sample complexity for sub-Gaussian distributions, allows $Σ$ to be degenerate or low rank, and adaptively extends beyond sub-Gaussianity. Prior to this work, other methods required exponential computation time or the superlinear scaling $n = Ω(d^{3/2})$ to achieve non-trivial error with respect to the norm $||\cdot||_Σ$.
△ Less
Submitted 17 January, 2023;
originally announced January 2023.
-
Concentration bounds for SSP Q-learning for average cost MDPs
Authors:
Shaan Ul Haque,
Vivek Borkar
Abstract:
We derive a concentration bound for a Q-learning algorithm for average cost Markov decision processes based on an equivalent shortest path problem, and compare it numerically with the alternative scheme based on relative value iteration.
We derive a concentration bound for a Q-learning algorithm for average cost Markov decision processes based on an equivalent shortest path problem, and compare it numerically with the alternative scheme based on relative value iteration.
△ Less
Submitted 12 June, 2022; v1 submitted 7 June, 2022;
originally announced June 2022.
-
Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classification
Authors:
Niladri S. Chatterji,
Saminul Haque,
Tatsunori Hashimoto
Abstract:
While a broad range of techniques have been proposed to tackle distribution shift, the simple baseline of training on an $\textit{undersampled}$ balanced dataset often achieves close to state-of-the-art-accuracy across several popular benchmarks. This is rather surprising, since undersampling algorithms discard excess majority group data. To understand this phenomenon, we ask if learning is fundam…
▽ More
While a broad range of techniques have been proposed to tackle distribution shift, the simple baseline of training on an $\textit{undersampled}$ balanced dataset often achieves close to state-of-the-art-accuracy across several popular benchmarks. This is rather surprising, since undersampling algorithms discard excess majority group data. To understand this phenomenon, we ask if learning is fundamentally constrained by a lack of minority group samples. We prove that this is indeed the case in the setting of nonparametric binary classification. Our results show that in the worst case, an algorithm cannot outperform undersampling unless there is a high degree of overlap between the train and test distributions (which is unlikely to be the case in real-world datasets), or if the algorithm leverages additional structure about the distribution shift. In particular, in the case of label shift we show that there is always an undersampling algorithm that is minimax optimal. In the case of group-covariate shift we show that there is an undersampling algorithm that is minimax optimal when the overlap between the group distributions is small. We also perform an experimental case study on a label shift dataset and find that in line with our theory, the test accuracy of robust neural network classifiers is constrained by the number of minority samples.
△ Less
Submitted 19 June, 2023; v1 submitted 25 May, 2022;
originally announced May 2022.
-
Joint Probability Estimation Using Tensor Decomposition and Dictionaries
Authors:
Shaan ul Haque,
Ajit Rajwade,
Karthik S. Gurumoorthy
Abstract:
In this work, we study non-parametric estimation of joint probabilities of a given set of discrete and continuous random variables from their (empirically estimated) 2D marginals, under the assumption that the joint probability could be decomposed and approximated by a mixture of product densities/mass functions. The problem of estimating the joint probability density function (PDF) using semi-par…
▽ More
In this work, we study non-parametric estimation of joint probabilities of a given set of discrete and continuous random variables from their (empirically estimated) 2D marginals, under the assumption that the joint probability could be decomposed and approximated by a mixture of product densities/mass functions. The problem of estimating the joint probability density function (PDF) using semi-parametric techniques such as Gaussian Mixture Models (GMMs) is widely studied. However such techniques yield poor results when the underlying densities are mixtures of various other families of distributions such as Laplacian or generalized Gaussian, uniform, Cauchy, etc. Further, GMMs are not the best choice to estimate joint distributions which are hybrid in nature, i.e., some random variables are discrete while others are continuous. We present a novel approach for estimating the PDF using ideas from dictionary representations in signal processing coupled with low rank tensor decompositions. To the best our knowledge, this is the first work on estimating joint PDFs employing dictionaries alongside tensor decompositions. We create a dictionary of various families of distributions by inspecting the data, and use it to approximate each decomposed factor of the product in the mixture. Our approach can naturally handle hybrid $N$-dimensional distributions. We test our approach on a variety of synthetic and real datasets to demonstrate its effectiveness in terms of better classification rates and lower error rates, when compared to state of the art estimators.
△ Less
Submitted 3 March, 2022;
originally announced March 2022.
-
Is Importance Weighting Incompatible with Interpolating Classifiers?
Authors:
Ke Alexander Wang,
Niladri S. Chatterji,
Saminul Haque,
Tatsunori Hashimoto
Abstract:
Importance weighting is a classic technique to handle distribution shifts. However, prior work has presented strong empirical and theoretical evidence demonstrating that importance weights can have little to no effect on overparameterized neural networks. Is importance weighting truly incompatible with the training of overparameterized neural networks? Our paper answers this in the negative. We sh…
▽ More
Importance weighting is a classic technique to handle distribution shifts. However, prior work has presented strong empirical and theoretical evidence demonstrating that importance weights can have little to no effect on overparameterized neural networks. Is importance weighting truly incompatible with the training of overparameterized neural networks? Our paper answers this in the negative. We show that importance weighting fails not because of the overparameterization, but instead, as a result of using exponentially-tailed losses like the logistic or cross-entropy loss. As a remedy, we show that polynomially-tailed losses restore the effects of importance reweighting in correcting distribution shift in overparameterized models. We characterize the behavior of gradient descent on importance weighted polynomially-tailed losses with overparameterized linear models, and theoretically demonstrate the advantage of using polynomially-tailed losses in a label shift setting. Surprisingly, our theory shows that using weights that are obtained by exponentiating the classical unbiased importance weights can improve performance. Finally, we demonstrate the practical value of our analysis with neural network experiments on a subpopulation shift and a label shift dataset. When reweighted, our loss function can outperform reweighted cross-entropy by as much as 9% in test accuracy. Our loss function also gives test accuracies comparable to, or even exceeding, well-tuned state-of-the-art methods for correcting distribution shifts.
△ Less
Submitted 4 March, 2022; v1 submitted 24 December, 2021;
originally announced December 2021.
-
Improved assessment of the accuracy of record linkage via an extended MaCSim approach
Authors:
Shovanur Haque,
Kerrie Mengersen
Abstract:
Record linkage is the process of bringing together the same entity from overlapping data sources while removing duplicates. Huge amounts of data are now being collected by public or private organizations as well as by researchers and individuals. Linking and analysing relevant information from this massive data reservoir can provide new insights into society. However, this increase in the amount o…
▽ More
Record linkage is the process of bringing together the same entity from overlapping data sources while removing duplicates. Huge amounts of data are now being collected by public or private organizations as well as by researchers and individuals. Linking and analysing relevant information from this massive data reservoir can provide new insights into society. However, this increase in the amount of data may also increase the likelihood of incorrectly linked records among databases. It has become increasingly important to have effective and efficient methods for linking data from different sources. Therefore, it becomes necessary to assess the ability of a linking method to achieve high accuracy or to compare between methods with respect to accuracy. In this paper, we improve on a Markov Chain based Monte Carlo simulation approach (MaCSim) for assessing a linking method. MaCSim utilizes two linked files that have been previously linked on similar types of data to create an agreement matrix and then simulates the matrix using a proposed algorithm developed to generate re-sampled versions of the agreement matrix. A defined linking method is used in each simulation to link the files and the accuracy of the linking method is assessed. The improvement proposed here involves calculation of a similarity weight for every linking variable value for each record pair, which allows partial agreement of the linking variable values. A threshold is calculated for every linking variable based on adjustable parameter "tolerance" for that variable. To assess the accuracy of linking method, correctly linked proportions are investigated for each record. The extended MaCSim approach is illustrated using a synthetic dataset provided by the Australian Bureau of Statistics (ABS) based on realistic data settings. Test results show higher accuracy of the assessment of linkages.
△ Less
Submitted 12 October, 2020; v1 submitted 12 March, 2020;
originally announced March 2020.
-
Assessing the accuracy of individual link with varying block sizes and cut-off values using MaCSim approach
Authors:
Shovanur Haque,
Kerrie Mengersen
Abstract:
Record linkage is the process of matching together records from different data sources that belong to the same entity. Record linkage is increasingly being used by many organizations including statistical, health, government etc. to link administrative, survey, and other files to create a robust file for more comprehensive analysis. Therefore, it becomes necessary to assess the ability of a linkin…
▽ More
Record linkage is the process of matching together records from different data sources that belong to the same entity. Record linkage is increasingly being used by many organizations including statistical, health, government etc. to link administrative, survey, and other files to create a robust file for more comprehensive analysis. Therefore, it becomes necessary to assess the ability of a linking method to achieve high accuracy or compare between methods with respect to accuracy. In this paper, we evaluate the accuracy of individual link using varying block sizes and different cut-off values by utilizing a Markov Chain based Monte Carlo simulation approach (MaCSim). MaCSim utilizes two linked files to create an agreement matrix. The agreement matrix is simulated to generate re-sampled versions of the agreement matrix. A defined linking method is used in each simulation to link the files and the accuracy of the linking method is assessed. The aim of this paper is to facilitate optimal choice of block size and cut-off value to achieve high accuracy in terms of minimizing average False Discovery Rate and False Negative Rate. The analyses have been performed using a synthetic dataset provided by the Australian Bureau of Statistics (ABS) and indicated promising results.
△ Less
Submitted 23 November, 2020; v1 submitted 12 March, 2020;
originally announced March 2020.
-
Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
Authors:
Qiyang Li,
Saminul Haque,
Cem Anil,
James Lucas,
Roger Grosse,
Jörn-Henrik Jacobsen
Abstract:
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance while provably enforcing a Lipschitz constraint. In principle, one can design Lips…
▽ More
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance while provably enforcing a Lipschitz constraint. In principle, one can design Lipschitz constrained architectures using the composition property of Lipschitz functions, but Anil et al. recently identified a key obstacle to this approach: gradient norm attenuation. They showed how to circumvent this problem in the case of fully connected networks by designing each layer to be gradient norm preserving. We extend their approach to train scalable, expressive, provably Lipschitz convolutional networks. In particular, we present the Block Convolution Orthogonal Parameterization (BCOP), an expressive parameterization of orthogonal convolution operations. We show that even though the space of orthogonal convolutions is disconnected, the largest connected component of BCOP with 2n channels can represent arbitrary BCOP convolutions over n channels. Our BCOP parameterization allows us to train large convolutional networks with provable Lipschitz bounds. Empirically, we find that it is competitive with existing approaches to provable adversarial robustness and Wasserstein distance estimation.
△ Less
Submitted 9 November, 2019; v1 submitted 3 November, 2019;
originally announced November 2019.
-
Assessing the accuracy of record linkages with Markov chain based Monte Carlo simulation approach
Authors:
Shovanur Haque,
Kerrie Mengersen,
Steven Stern
Abstract:
Record linkage is the process of finding matches and linking records from different data sources so that the linked records belong to the same entity. There is an increasing number of applications of record linkage in statistical, health, government and business organisations to link administrative, survey, population census and other files to create a complete set of information for more complete…
▽ More
Record linkage is the process of finding matches and linking records from different data sources so that the linked records belong to the same entity. There is an increasing number of applications of record linkage in statistical, health, government and business organisations to link administrative, survey, population census and other files to create a complete set of information for more complete and comprehensive analysis. To make valid inferences using a linked file, it is increasingly becoming important to assess the linking method. It is also important to find techniques to improve the linking process to achieve higher accuracy. This motivates to develop a method for assessing linking process and help decide which linking method is likely to be more accurate for a linking task. This paper proposes a Markov Chain based Monte Carlo simulation approach, MaCSim for assessing a linking method and illustrates the utility of the approach using a realistic synthetic dataset received from the Australian Bureau of Statistics to avoid privacy issues associated with using real personal information. A linking method applied by MaCSim is also defined. To assess the defined linking method, correct re-link proportions for each record are calculated using our developed simulation approach. The accuracy is determined for a number of simulated datasets. The analyses indicated promising performance of the proposed method MaCSim of the assessment of accuracy of the linkages. The computational aspects of the methodology are also investigated to assess its feasibility for practical use.
△ Less
Submitted 29 September, 2020; v1 submitted 15 January, 2019;
originally announced January 2019.
-
Statement networks: a power structure narrative as depicted by newspapers
Authors:
Shoumik Sharar Chowdhury,
Nazmus Saquib,
Niamat Zawad,
Manash Kumar Mandal,
Syed Haque
Abstract:
We report a data mining pipeline and subsequent analysis to understand the core periphery power structure created in three national newspapers in Bangladesh, as depicted by statements made by people appearing in news. Statements made by one actor about another actor can be considered a form of public conversation. Named entity recognition techniques can be used to create a temporal actor network f…
▽ More
We report a data mining pipeline and subsequent analysis to understand the core periphery power structure created in three national newspapers in Bangladesh, as depicted by statements made by people appearing in news. Statements made by one actor about another actor can be considered a form of public conversation. Named entity recognition techniques can be used to create a temporal actor network from such conversations, which shows some unique structure, and reveals much room for improvement in news reporting and also the top actors' conversation preferences. Our results indicate there is a presence of cliquishness between powerful political leaders when it comes to their appearance in news. We also show how these cohesive cores form through the news articles, and how, over a decade, news cycles change the actors belonging in these groups.
△ Less
Submitted 10 December, 2018;
originally announced December 2018.
