-
Persistent Reductions in Regularized Loss Minimization for Variable Selection
Authors:
Amin Jalali
Abstract:
In the context of regularized loss minimization with polyhedral gauges, we show that for a broad class of loss functions (possibly non-smooth and non-convex) and under a simple geometric condition on the input data it is possible to efficiently identify a subset of features which are guaranteed to have zero coefficients in all optimal solutions in all problems with loss functions from said class,…
▽ More
In the context of regularized loss minimization with polyhedral gauges, we show that for a broad class of loss functions (possibly non-smooth and non-convex) and under a simple geometric condition on the input data it is possible to efficiently identify a subset of features which are guaranteed to have zero coefficients in all optimal solutions in all problems with loss functions from said class, before any iterative optimization has been performed for the original problem. This procedure is standalone, takes only the data as input, and does not require any calls to the loss function. Therefore, we term this procedure as a persistent reduction for the aforementioned class of regularized loss minimization problems. This reduction can be efficiently implemented via an extreme ray identification subroutine applied to a polyhedral cone formed from the datapoints. We employ an existing output-sensitive algorithm for extreme ray identification which makes our guarantee and algorithm applicable in ultra-high dimensional problems.
△ Less
Submitted 29 November, 2020;
originally announced November 2020.
-
New Computational and Statistical Aspects of Regularized Regression with Application to Rare Feature Selection and Aggregation
Authors:
Amin Jalali,
Adel Javanmard,
Maryam Fazel
Abstract:
Prior knowledge on properties of a target model often come as discrete or combinatorial descriptions. This work provides a unified computational framework for defining norms that promote such structures. More specifically, we develop associated tools for optimization involving such norms given only the orthogonal projection oracle onto the non-convex set of desired models. As an example, we study…
▽ More
Prior knowledge on properties of a target model often come as discrete or combinatorial descriptions. This work provides a unified computational framework for defining norms that promote such structures. More specifically, we develop associated tools for optimization involving such norms given only the orthogonal projection oracle onto the non-convex set of desired models. As an example, we study a norm, which we term the doubly-sparse norm, for promoting vectors with few nonzero entries taking only a few distinct values. We further discuss how the K-means algorithm can serve as the underlying projection oracle in this case and how it can be efficiently represented as a quadratically constrained quadratic program. Our motivation for the study of this norm is regularized regression in the presence of rare features which poses a challenge to various methods within high-dimensional statistics, and in machine learning in general. The proposed estimation procedure is designed to perform automatic feature selection and aggregation for which we develop statistical bounds. The bounds are general and offer a statistical framework for norm-based regularization. The bounds rely on novel geometric quantities on which we attempt to elaborate as well.
△ Less
Submitted 10 April, 2019;
originally announced April 2019.
-
2-Wasserstein Approximation via Restricted Convex Potentials with Application to Improved Training for GANs
Authors:
Amirhossein Taghvaei,
Amin Jalali
Abstract:
We provide a framework to approximate the 2-Wasserstein distance and the optimal transport map, amenable to efficient training as well as statistical and geometric analysis. With the quadratic cost and considering the Kantorovich dual form of the optimal transportation problem, the Brenier theorem states that the optimal potential function is convex and the optimal transport map is the gradient of…
▽ More
We provide a framework to approximate the 2-Wasserstein distance and the optimal transport map, amenable to efficient training as well as statistical and geometric analysis. With the quadratic cost and considering the Kantorovich dual form of the optimal transportation problem, the Brenier theorem states that the optimal potential function is convex and the optimal transport map is the gradient of the optimal potential function. Using this geometric structure, we restrict the optimization problem to different parametrized classes of convex functions and pay special attention to the class of input-convex neural networks. We analyze the statistical generalization and the discriminative power of the resulting approximate metric, and we prove a restricted moment-matching property for the approximate optimal map. Finally, we discuss a numerical algorithm to solve the restricted optimization problem and provide numerical experiments to illustrate and compare the proposed approach with the established regularization-based approaches. We further discuss practical implications of our proposal in a modular and interpretable design for GANs which connects the generator training with discriminator computations to allow for learning an overall composite generator.
△ Less
Submitted 19 February, 2019;
originally announced February 2019.
-
Variational Gram Functions: Convex Analysis and Optimization
Authors:
Amin Jalali,
Maryam Fazel,
Lin Xiao
Abstract:
We propose a new class of convex penalty functions, called \emph{variational Gram functions} (VGFs), that can promote pairwise relations, such as orthogonality, among a set of vectors in a vector space. These functions can serve as regularizers in convex optimization problems arising from hierarchical classification, multitask learning, and estimating vectors with disjoint supports, among other ap…
▽ More
We propose a new class of convex penalty functions, called \emph{variational Gram functions} (VGFs), that can promote pairwise relations, such as orthogonality, among a set of vectors in a vector space. These functions can serve as regularizers in convex optimization problems arising from hierarchical classification, multitask learning, and estimating vectors with disjoint supports, among other applications. We study convexity for VGFs, and give efficient characterizations for their convex conjugates, subdifferentials, and proximal operators. We discuss efficient optimization algorithms for regularized loss minimization problems where the loss admits a common, yet simple, variational representation and the regularizer is a VGF. These algorithms enjoy a simple kernel trick, an efficient line search, as well as computational advantages over first order methods based on the subdifferential or proximal maps. We also establish a general representer theorem for such learning problems. Lastly, numerical experiments on a hierarchical classification problem are presented to demonstrate the effectiveness of VGFs and the associated optimization algorithms.
△ Less
Submitted 11 April, 2017; v1 submitted 16 July, 2015;
originally announced July 2015.
-
Terminal retrograde turn of rolling rings
Authors:
Mir Abbas Jalali,
Milad S. Sarebangholi,
Reza Alam
Abstract:
We report an unexpected reverse spiral turn in the final stage of the motion of rolling rings. It is well known that spinning disks rotate in the same direction of their initial spin until they stop. While a spinning ring starts its motion with a kinematics similar to disks, i.e. moving along a cycloidal path prograde with the direction of its rigid body rotation, the mean trajectory of its center…
▽ More
We report an unexpected reverse spiral turn in the final stage of the motion of rolling rings. It is well known that spinning disks rotate in the same direction of their initial spin until they stop. While a spinning ring starts its motion with a kinematics similar to disks, i.e. moving along a cycloidal path prograde with the direction of its rigid body rotation, the mean trajectory of its center of mass later develops an inflection point so that the ring makes a spiral turn and revolves in a retrograde direction around a new center. Using high speed imaging and numerical simulations of models featuring a rolling rigid body, we show that the hollow geometry of a ring tunes the rotational air drag resistance so that the frictional force at the contact point with the ground changes its direction at the inflection point and puts the ring on a retrograde spiral trajectory. Our findings have potential applications in designing topologically new surface-effect flying objects capable of performing complex reorientation and translational maneuvers.
△ Less
Submitted 24 August, 2015; v1 submitted 4 December, 2014;
originally announced December 2014.
-
Collective Dynamics of Interacting Particles in Unsteady Flows
Authors:
Maryam Abedi,
Mir Abbas Jalali
Abstract:
We use the Fokker-Planck equation and its moment equations to study the collective behavior of interacting particles in unsteady one-dimensional flows. Particles interact according to a long-range attractive and a short-range repulsive potential field known as Morse potential. We assume Stokesian drag force between particles and their carrier fluid, and find analytic single-peaked traveling soluti…
▽ More
We use the Fokker-Planck equation and its moment equations to study the collective behavior of interacting particles in unsteady one-dimensional flows. Particles interact according to a long-range attractive and a short-range repulsive potential field known as Morse potential. We assume Stokesian drag force between particles and their carrier fluid, and find analytic single-peaked traveling solutions for the spatial density of particles in the catastrophic phase. In steady flow conditions the streaming velocity of particles is identical to their carrier fluid, but we show that particle streaming is asynchronous with an unsteady carrier fluid. Using linear perturbation analysis, the stability of traveling solutions is investigated in unsteady conditions. It is shown that the resulting dispersion relation is an integral equation of the Fredholm type, and yields two general families of stable modes: singular modes whose eigenvalues form a continuous spectrum, and a finite number of discrete global modes. Depending on the value of drag coefficient, stable modes can be over-damped, critically damped, or decaying oscillatory waves. The results of linear perturbation analysis are confirmed through the numerical solution of the fully nonlinear Fokker-Planck equation.
△ Less
Submitted 3 August, 2014;
originally announced August 2014.
-
Simultaneously Structured Models with Application to Sparse and Low-rank Matrices
Authors:
Samet Oymak,
Amin Jalali,
Maryam Fazel,
Yonina C. Eldar,
Babak Hassibi
Abstract:
The topic of recovery of a structured model given a small number of linear observations has been well-studied in recent years. Examples include recovering sparse or group-sparse vectors, low-rank matrices, and the sum of sparse and low-rank matrices, among others. In various applications in signal processing and machine learning, the model of interest is known to be structured in several ways at t…
▽ More
The topic of recovery of a structured model given a small number of linear observations has been well-studied in recent years. Examples include recovering sparse or group-sparse vectors, low-rank matrices, and the sum of sparse and low-rank matrices, among others. In various applications in signal processing and machine learning, the model of interest is known to be structured in several ways at the same time, for example, a matrix that is simultaneously sparse and low-rank.
Often norms that promote each individual structure are known, and allow for recovery using an order-wise optimal number of measurements (e.g., $\ell_1$ norm for sparsity, nuclear norm for matrix rank). Hence, it is reasonable to minimize a combination of such norms. We show that, surprisingly, if we use multi-objective optimization with these norms, then we can do no better, order-wise, than an algorithm that exploits only one of the present structures. This result suggests that to fully exploit the multiple structures, we need an entirely new convex relaxation, i.e. not one that is a function of the convex relaxations used for each structure. We then specialize our results to the case of sparse and low-rank matrices. We show that a nonconvex formulation of the problem can recover the model from very few measurements, which is on the order of the degrees of freedom of the matrix, whereas the convex problem obtained from a combination of the $\ell_1$ and nuclear norms requires many more measurements. This proves an order-wise gap between the performance of the convex and nonconvex recovery problems in this case. Our framework applies to arbitrary structure-inducing norms as well as to a wide range of measurement ensembles. This allows us to give performance bounds for problems such as sparse phase retrieval and low-rank tensor completion.
△ Less
Submitted 25 July, 2014; v1 submitted 16 December, 2012;
originally announced December 2012.
-
High-dimensional Sparse Inverse Covariance Estimation using Greedy Methods
Authors:
Christopher C. Johnson,
Ali Jalali,
Pradeep Ravikumar
Abstract:
In this paper we consider the task of estimating the non-zero pattern of the sparse inverse covariance matrix of a zero-mean Gaussian random vector from a set of iid samples. Note that this is also equivalent to recovering the underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We present two novel greedy approaches to solving this problem. The first estimates the non-zero…
▽ More
In this paper we consider the task of estimating the non-zero pattern of the sparse inverse covariance matrix of a zero-mean Gaussian random vector from a set of iid samples. Note that this is also equivalent to recovering the underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We present two novel greedy approaches to solving this problem. The first estimates the non-zero covariates of the overall inverse covariance matrix using a series of global forward and backward greedy steps. The second estimates the neighborhood of each node in the graph separately, again using greedy forward and backward steps, and combines the intermediate neighborhoods to form an overall estimate. The principal contribution of this paper is a rigorous analysis of the sparsistency, or consistency in recovering the sparsity pattern of the inverse covariance matrix. Surprisingly, we show that both the local and global greedy methods learn the full structure of the model with high probability given just $O(d\log(p))$ samples, which is a \emph{significant} improvement over state of the art $\ell_1$-regularized Gaussian MLE (Graphical Lasso) that requires $O(d^2\log(p))$ samples. Moreover, the restricted eigenvalue and smoothness conditions imposed by our greedy methods are much weaker than the strong irrepresentable conditions required by the $\ell_1$-regularization based methods. We corroborate our results with extensive simulations and examples, comparing our local and global greedy methods to the $\ell_1$-regularized Gaussian MLE as well as the Neighborhood Greedy method to that of nodewise $\ell_1$-regularized linear regression (Neighborhood Lasso).
△ Less
Submitted 29 December, 2011;
originally announced December 2011.
-
On Learning Discrete Graphical Models Using Greedy Methods
Authors:
Ali Jalali,
Chris Johnson,
Pradeep Ravikumar
Abstract:
In this paper, we address the problem of learning the structure of a pairwise graphical model from samples in a high-dimensional setting. Our first main result studies the sparsistency, or consistency in sparsity pattern recovery, properties of a forward-backward greedy algorithm as applied to general statistical models. As a special case, we then apply this algorithm to learn the structure of a d…
▽ More
In this paper, we address the problem of learning the structure of a pairwise graphical model from samples in a high-dimensional setting. Our first main result studies the sparsistency, or consistency in sparsity pattern recovery, properties of a forward-backward greedy algorithm as applied to general statistical models. As a special case, we then apply this algorithm to learn the structure of a discrete graphical model via neighborhood estimation. As a corollary of our general result, we derive sufficient conditions on the number of samples n, the maximum node-degree d and the problem size p, as well as other conditions on the model parameters, so that the algorithm recovers all the edges with high probability. Our result guarantees graph selection for samples scaling as n = Omega(d^2 log(p)), in contrast to existing convex-optimization based algorithms that require a sample complexity of Ω(d^3 log(p)). Further, the greedy algorithm only requires a restricted strong convexity condition which is typically milder than irrepresentability assumptions. We corroborate these results using numerical simulations at the end.
△ Less
Submitted 16 July, 2011;
originally announced July 2011.
-
A Dirty Model for Multiple Sparse Regression
Authors:
Ali Jalali,
Pradeep Ravikumar,
Sujay Sanghavi
Abstract:
Sparse linear regression -- finding an unknown vector from linear measurements -- is now known to be possible with fewer samples than variables, via methods like the LASSO. We consider the multiple sparse linear regression problem, where several related vectors -- with partially shared support sets -- have to be recovered. A natural question in this setting is whether one can use the sharing to fu…
▽ More
Sparse linear regression -- finding an unknown vector from linear measurements -- is now known to be possible with fewer samples than variables, via methods like the LASSO. We consider the multiple sparse linear regression problem, where several related vectors -- with partially shared support sets -- have to be recovered. A natural question in this setting is whether one can use the sharing to further decrease the overall number of samples required. A line of recent research has studied the use of \ell_1/\ell_q norm block-regularizations with q>1 for such problems; however these could actually perform worse in sample complexity -- vis a vis solving each problem separately ignoring sharing -- depending on the level of sharing.
We present a new method for multiple sparse linear regression that can leverage support and parameter overlap when it exists, but not pay a penalty when it does not. A very simple idea: we decompose the parameters into two components and regularize these differently. We show both theoretically and empirically, our method strictly and noticeably outperforms both \ell_1 or \ell_1/\ell_q methods, over the entire range of possible overlaps (except at boundary cases, where we match the best method). We also provide theoretical guarantees that the method performs well under high-dimensional scaling.
△ Less
Submitted 28 June, 2011;
originally announced June 2011.
