-
Mixed precision iterative refinement for least squares with linear equality constraints and generalized least squares problems
Authors:
Bowen Gao,
Yuxin Ma,
Meiyue Shao
Abstract:
Recent development on mixed precision techniques has largely enhanced the performance of various linear algebra solvers, one of which being the least squares problem $\min_{x}\lVert b-Ax\rVert_{2}$. By transforming the least squares problem into an augmented linear system, mixed precision techniques are capable of refining the lower precision solution to the working precision. In this paper, we pr…
▽ More
Recent development on mixed precision techniques has largely enhanced the performance of various linear algebra solvers, one of which being the least squares problem $\min_{x}\lVert b-Ax\rVert_{2}$. By transforming the least squares problem into an augmented linear system, mixed precision techniques are capable of refining the lower precision solution to the working precision. In this paper, we propose mixed precision iterative refinement algorithms for two variants of the least squares problem -- the least squares problem with linear equality constraints (LSE) and the generalized least squares problem (GLS). Both classical and GMRES-based iterative refinement can be applied to augmented systems of these two problems to improve the accuracy of the solution. For reasonably well-conditioned problems our algorithms reduce the execution time by a factor of 40% in average compared to the fixed precision ones from LAPACK on the x86-64 architecture.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Symplectic Stiefel manifold: tractable metrics, second-order geometry and Newton's methods
Authors:
Bin Gao,
Nguyen Thanh Son,
Tatjana Stykel
Abstract:
Optimization under the symplecticity constraint is an approach for solving various problems in quantum physics and scientific computing. Building on the results that this optimization problem can be transformed into an unconstrained problem on the symplectic Stiefel manifold, we construct geometric ingredients for Riemannian optimization with a new family of Riemannian metrics called tractable met…
▽ More
Optimization under the symplecticity constraint is an approach for solving various problems in quantum physics and scientific computing. Building on the results that this optimization problem can be transformed into an unconstrained problem on the symplectic Stiefel manifold, we construct geometric ingredients for Riemannian optimization with a new family of Riemannian metrics called tractable metrics and develop Riemannian Newton schemes. The newly obtained ingredients do not only generalize several existing results but also provide us with freedom to choose a suitable metric for each problem. To the best of our knowledge, this is the first try to develop the explicit second-order geometry and Newton's methods on the symplectic Stiefel manifold. For the Riemannian Newton method, we first consider novel operator-valued formulas for computing the Riemannian Hessian of a~cost function, which further allows the manifold to be endowed with a weighted Euclidean metric that can provide a preconditioning effect. We then solve the resulting Newton equation, as the central step of Newton's methods, directly via transforming it into a~saddle point problem followed by vectorization, or iteratively via applying any matrix-free iterative method either to the operator Newton equation or its saddle point formulation. Finally, we propose a hybrid Riemannian Newton optimization algorithm that enjoys both global convergence and quadratic/superlinear local convergence at the final stage. Various numerical experiments are presented to validate the proposed methods.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity
Authors:
Yan Yang,
Bin Gao,
Ya-xiang Yuan
Abstract:
Bilevel reinforcement learning (RL), which features intertwined two-level problems, has attracted growing interest recently. The inherent non-convexity of the lower-level RL problem is, however, to be an impediment to developing bilevel optimization methods. By employing the fixed point equation associated with the regularized RL, we characterize the hyper-gradient via fully first-order informatio…
▽ More
Bilevel reinforcement learning (RL), which features intertwined two-level problems, has attracted growing interest recently. The inherent non-convexity of the lower-level RL problem is, however, to be an impediment to developing bilevel optimization methods. By employing the fixed point equation associated with the regularized RL, we characterize the hyper-gradient via fully first-order information, thus circumventing the assumption of lower-level convexity. This, remarkably, distinguishes our development of hyper-gradient from the general AID-based bilevel frameworks since we take advantage of the specific structure of RL problems. Moreover, we propose both model-based and model-free bilevel reinforcement learning algorithms, facilitated by access to the fully first-order hyper-gradient. Both algorithms are provable to enjoy the convergence rate $\mathcal{O}(ε^{-1})$. To the best of our knowledge, this is the first time that AID-based bilevel RL gets rid of additional assumptions on the lower-level problem. In addition, numerical experiments demonstrate that the hyper-gradient indeed serves as an integration of exploitation and exploration.
△ Less
Submitted 30 May, 2024;
originally announced May 2024.
-
On ergodic optimization for unimodal maps
Authors:
Bing Gao,
Rui Gao
Abstract:
In this article, we show that for a typical non-uniformly expanding unimodal map, the unique maximizing measure of a generic Lipschitz function is supported on a periodic orbit.
In this article, we show that for a typical non-uniformly expanding unimodal map, the unique maximizing measure of a generic Lipschitz function is supported on a periodic orbit.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Optimization without Retraction on the Random Generalized Stiefel Manifold
Authors:
Simon Vary,
Pierre Ablin,
Bin Gao,
P. -A. Absil
Abstract:
Optimization over the set of matrices $X$ that satisfy $X^\top B X = I_p$, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP). Solving these problems is typically done by iterative methods that require…
▽ More
Optimization over the set of matrices $X$ that satisfy $X^\top B X = I_p$, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP). Solving these problems is typically done by iterative methods that require a fully formed $B$. We propose a cheap stochastic iterative method that solves the optimization problem while having access only to a random estimates of $B$. Our method does not enforce the constraint in every iteration; instead, it produces iterations that converge to critical points on the generalized Stiefel manifold defined in expectation. The method has lower per-iteration cost, requires only matrix multiplications, and has the same convergence rates as its Riemannian optimization counterparts that require the full matrix $B$. Experiments demonstrate its effectiveness in various machine learning applications involving generalized orthogonality constraints, including CCA, ICA, and the GEVP.
△ Less
Submitted 5 June, 2024; v1 submitted 2 May, 2024;
originally announced May 2024.
-
LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspace
Authors:
Bin Gao,
Yan Yang,
Ya-xiang Yuan
Abstract:
Bilevel optimization, with broad applications in machine learning, has an intricate hierarchical structure. Gradient-based methods have emerged as a common approach to large-scale bilevel problems. However, the computation of the hyper-gradient, which involves a Hessian inverse vector product, confines the efficiency and is regarded as a bottleneck. To circumvent the inverse, we construct a sequen…
▽ More
Bilevel optimization, with broad applications in machine learning, has an intricate hierarchical structure. Gradient-based methods have emerged as a common approach to large-scale bilevel problems. However, the computation of the hyper-gradient, which involves a Hessian inverse vector product, confines the efficiency and is regarded as a bottleneck. To circumvent the inverse, we construct a sequence of low-dimensional approximate Krylov subspaces with the aid of the Lanczos process. As a result, the constructed subspace is able to dynamically and incrementally approximate the Hessian inverse vector product with less effort and thus leads to a favorable estimate of the hyper-gradient. Moreover, we propose a~provable subspace-based framework for bilevel problems where one central step is to solve a small-size tridiagonal linear system. To the best of our knowledge, this is the first time that subspace techniques are incorporated into bilevel optimization. This successful trial not only enjoys $\mathcal{O}(ε^{-1})$ convergence rate but also demonstrates efficiency in a synthetic problem and two deep learning tasks.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
Products and powers of principal symmetric ideals
Authors:
Eric Dannetun,
Riccardo Formenti,
Bo Y. Gao,
Juliann Geraci,
Ross Kogel,
Yuelin Li,
Shreya Mandal,
Vinuge Rupasinghe,
Alexandra Seceleanu,
Duc Van Khank Tran,
Noah Walker
Abstract:
Principal symmetric ideals were recently introduced by Harada, Seceleanu, and Sega, with a focus on their homological properties. They are ideals generated by the orbit of a single polynomial under permutations of variables in a polynomial ring. In this paper we seek to determine when a product of two principal symmetric ideals is principal symmetric and when all the powers of a principal symmetri…
▽ More
Principal symmetric ideals were recently introduced by Harada, Seceleanu, and Sega, with a focus on their homological properties. They are ideals generated by the orbit of a single polynomial under permutations of variables in a polynomial ring. In this paper we seek to determine when a product of two principal symmetric ideals is principal symmetric and when all the powers of a principal symmetric ideal are again principal symmetric ideals. We characterize the ideals that have the latter property as being generated by polynomials invariant up to a scalar multiple under permutation of variables. Recognizing principal symmetric ideals is an open question for the purpose of which we produce certain obstructions. We also demonstrate that the Hilbert functions of symmetric monomial ideals are not all given by symmetric monomial ideals, in contrast to the non-symmetric case.
△ Less
Submitted 17 June, 2024; v1 submitted 25 February, 2024;
originally announced February 2024.
-
Low-rank optimization on Tucker tensor varieties
Authors:
Bin Gao,
Renfeng Peng,
Ya-xiang Yuan
Abstract:
In the realm of tensor optimization, the low-rank Tucker decomposition is crucial for reducing the number of parameters and for saving storage. We explore the geometry of Tucker tensor varieties -- the set of tensors with bounded Tucker rank -- which is notably more intricate than the well-explored matrix varieties. We give an explicit parametrization of the tangent cone of Tucker tensor varieties…
▽ More
In the realm of tensor optimization, the low-rank Tucker decomposition is crucial for reducing the number of parameters and for saving storage. We explore the geometry of Tucker tensor varieties -- the set of tensors with bounded Tucker rank -- which is notably more intricate than the well-explored matrix varieties. We give an explicit parametrization of the tangent cone of Tucker tensor varieties and leverage its geometry to develop provable gradient-related line-search methods for optimization on Tucker tensor varieties. To the best of our knowledge, this is the first work concerning geometry and optimization on Tucker tensor varieties. In practice, low-rank tensor optimization suffers from the difficulty of choosing a reliable rank parameter. To this end, we incorporate the established geometry and propose a Tucker rank-adaptive method that aims to identify an appropriate rank with guaranteed convergence. Numerical experiments on tensor completion reveal that the proposed methods are in favor of recovering performance over other state-of-the-art methods. The rank-adaptive method performs the best across various rank parameter selections and is indeed able to find an appropriate rank.
△ Less
Submitted 13 July, 2024; v1 submitted 30 November, 2023;
originally announced November 2023.
-
Optimization on product manifolds under a preconditioned metric
Authors:
Bin Gao,
Renfeng Peng,
Ya-xiang Yuan
Abstract:
Since optimization on Riemannian manifolds relies on the chosen metric, it is appealing to know that how the performance of a Riemannian optimization method varies with different metrics and how to exquisitely construct a metric such that a method can be accelerated. To this end, we propose a general framework for optimization problems on product manifolds where the search space is endowed with a…
▽ More
Since optimization on Riemannian manifolds relies on the chosen metric, it is appealing to know that how the performance of a Riemannian optimization method varies with different metrics and how to exquisitely construct a metric such that a method can be accelerated. To this end, we propose a general framework for optimization problems on product manifolds where the search space is endowed with a preconditioned metric, and we develop the Riemannian gradient descent and Riemannian conjugate gradient methods under this metric. Specifically, the metric is constructed by an operator that aims to approximate the diagonal blocks of the Riemannian Hessian of the cost function, which has a preconditioning effect. We explain the relationship between the proposed methods and the variable metric methods, and show that various existing methods, e.g., the Riemannian Gauss--Newton method, can be interpreted by the proposed framework with specific metrics. In addition, we tailor new preconditioned metrics and adapt the proposed Riemannian methods to the canonical correlation analysis and the truncated singular value decomposition problems, and we propose the Gauss--Newton method to solve the tensor ring completion problem. Numerical results among these applications verify that a delicate metric does accelerate the Riemannian optimization methods.
△ Less
Submitted 15 June, 2023;
originally announced June 2023.
-
Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints
Authors:
Pierre Ablin,
Simon Vary,
Bin Gao,
P. -A. Absil
Abstract:
Orthogonality constraints naturally appear in many machine learning problems, from Principal Components Analysis to robust neural network training. They are usually solved using Riemannian optimization algorithms, which minimize the objective function while enforcing the constraint. However, enforcing the orthogonality constraint can be the most time-consuming operation in such algorithms. Recentl…
▽ More
Orthogonality constraints naturally appear in many machine learning problems, from Principal Components Analysis to robust neural network training. They are usually solved using Riemannian optimization algorithms, which minimize the objective function while enforcing the constraint. However, enforcing the orthogonality constraint can be the most time-consuming operation in such algorithms. Recently, Ablin & Peyré (2022) proposed the Landing algorithm, a method with cheap iterations that does not enforce the orthogonality constraint but is attracted towards the manifold in a smooth manner. In this article, we provide new practical and theoretical developments for the landing algorithm. First, the method is extended to the Stiefel manifold, the set of rectangular orthogonal matrices. We also consider stochastic and variance reduction algorithms when the cost function is an average of many functions. We demonstrate that all these methods have the same rate of convergence as their Riemannian counterparts that exactly enforce the constraint. Finally, our experiments demonstrate the promise of our approach to an array of machine-learning problems that involve orthogonality constraints.
△ Less
Submitted 29 March, 2023;
originally announced March 2023.
-
Riemannian preconditioned algorithms for tensor completion via tensor ring decomposition
Authors:
Bin Gao,
Renfeng Peng,
Ya-xiang Yuan
Abstract:
We propose Riemannian preconditioned algorithms for the tensor completion problem via tensor ring decomposition. A new Riemannian metric is developed on the product space of the mode-2 unfolding matrices of the core tensors in tensor ring decomposition. The construction of this metric aims to approximate the Hessian of the cost function by its diagonal blocks, paving the way for various Riemannian…
▽ More
We propose Riemannian preconditioned algorithms for the tensor completion problem via tensor ring decomposition. A new Riemannian metric is developed on the product space of the mode-2 unfolding matrices of the core tensors in tensor ring decomposition. The construction of this metric aims to approximate the Hessian of the cost function by its diagonal blocks, paving the way for various Riemannian optimization methods. Specifically, we propose the Riemannian gradient descent and Riemannian conjugate gradient algorithms. We prove that both algorithms globally converge to a stationary point. In the implementation, we exploit the tensor structure and adopt an economical procedure to avoid large matrix formulation and computation in gradients, which significantly reduces the computational cost. Numerical experiments on various synthetic and real-world datasets -- movie ratings, hyperspectral images, and high-dimensional functions -- suggest that the proposed algorithms are more efficient and have better reconstruction ability than other candidates.
△ Less
Submitted 14 November, 2023; v1 submitted 28 February, 2023;
originally announced February 2023.
-
Resilient distributed resource allocation algorithm under false data injection attacks
Authors:
Xin Cai,
Xinyuan Nan,
Binpeng Gao
Abstract:
A resilient distributed algorithm is proposed to solve the distributed resource allocation problem of a first-order nonlinear multi-agent system who is subject to false data injection (FDI) attacks. An intelligent attacker injects false data into agents' actuators and sensors such that agents execute the algorithm according to the compromised control inputs and interactive information. The goal of…
▽ More
A resilient distributed algorithm is proposed to solve the distributed resource allocation problem of a first-order nonlinear multi-agent system who is subject to false data injection (FDI) attacks. An intelligent attacker injects false data into agents' actuators and sensors such that agents execute the algorithm according to the compromised control inputs and interactive information. The goal of the attacker is to make the multi-agent system to be unstable and to cause the deviance of agents' decisions from the optimal resource allocation. At first, we analyze the robustness of a distributed resource allocation algorithm under FDI attacks. Then, the unknown nonlinear term and the false data injected in agents are considered as extended states which can be estimated by extended state observers. The estimation was used in the feedback control to suppress the effect of the FDI attacks. A resilient distributed resource allocation algorithm based on the extended state observer is proposed to ensure that it can converge to the optimal allocation without requiring any information about the nature of the attacker. An example is given to illustrate the results.
△ Less
Submitted 6 December, 2022;
originally announced December 2022.
-
Optimization on the symplectic Stiefel manifold: SR decomposition-based retraction and applications
Authors:
Bin Gao,
Nguyen Thanh Son,
Tatjana Stykel
Abstract:
Numerous problems in optics, quantum physics, stability analysis, and control of dynamical systems can be brought to an optimization problem with matrix variable subjected to the symplecticity constraint. As this constraint nicely forms a so-called symplectic Stiefel manifold, Riemannian optimization is preferred, because one can borrow ideas from unconstrained optimization methods after preparing…
▽ More
Numerous problems in optics, quantum physics, stability analysis, and control of dynamical systems can be brought to an optimization problem with matrix variable subjected to the symplecticity constraint. As this constraint nicely forms a so-called symplectic Stiefel manifold, Riemannian optimization is preferred, because one can borrow ideas from unconstrained optimization methods after preparing necessary geometric tools. Retraction is arguably the most important one which decides the way iterates are updated given a search direction. Two retractions have been constructed so far: one relies on the Cayley transform and the other is designed using quasi-geodesic curves. In this paper, we propose a new retraction which is based on an SR matrix decomposition. We prove that its domain contains the open unit ball which is essential in proving the global convergence of the associated gradient-based optimization algorithm. Moreover, we consider three applications--symplectic target matrix problem, symplectic eigenvalue computation, and symplectic model reduction of Hamiltonian systems--with various examples. The extensive numerical comparisons reveal the strengths of the proposed optimization algorithm.
△ Less
Submitted 17 November, 2022;
originally announced November 2022.
-
On the analysis of optimization with fixed-rank matrices: a quotient geometric view
Authors:
Shuyu Dong,
Bin Gao,
Wen Huang,
Kyle A. Gallivan
Abstract:
We study a type of Riemannian gradient descent (RGD) algorithm, designed through Riemannian preconditioning, for optimization on $\mathcal{M}_k^{m\times n}$ -- the set of $m\times n$ real matrices with a fixed rank $k$. Our analysis is based on a quotient geometric view of $\mathcal{M}_k^{m\times n}$: by identifying this set with the quotient manifold of a two-term product space…
▽ More
We study a type of Riemannian gradient descent (RGD) algorithm, designed through Riemannian preconditioning, for optimization on $\mathcal{M}_k^{m\times n}$ -- the set of $m\times n$ real matrices with a fixed rank $k$. Our analysis is based on a quotient geometric view of $\mathcal{M}_k^{m\times n}$: by identifying this set with the quotient manifold of a two-term product space $\mathbb{R}_*^{m\times k}\times \mathbb{R}_*^{n\times k}$ of matrices with full column rank via matrix factorization, we find an explicit form for the update rule of the RGD algorithm, which leads to a novel approach to analysing their convergence behavior in rank-constrained optimization. We then deduce some interesting properties that reflect how RGD distinguishes from other matrix factorization algorithms such as those based on the Euclidean geometry. In particular, we show that the RGD algorithm is not only faster than Euclidean gradient descent but also does not rely on balancing techniques to ensure its efficiency while the latter does. We further show that this RGD algorithm is guaranteed to solve matrix sensing and matrix completion problems with linear convergence rate under the restricted positive definiteness property. Numerical experiments on matrix sensing and completion are provided to demonstrate these properties.
△ Less
Submitted 13 August, 2024; v1 submitted 13 March, 2022;
originally announced March 2022.
-
Meta Mirror Descent: Optimiser Learning for Fast Convergence
Authors:
Boyan Gao,
Henry Gouk,
Hae Beom Lee,
Timothy M. Hospedales
Abstract:
Optimisers are an essential component for training machine learning models, and their design influences learning speed and generalisation. Several studies have attempted to learn more effective gradient-descent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing optimiser learning methods are in…
▽ More
Optimisers are an essential component for training machine learning models, and their design influences learning speed and generalisation. Several studies have attempted to learn more effective gradient-descent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing optimiser learning methods are intuitively motivated, without clear theoretical support. We take a different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergence. Within this paradigm, we formalise a novel meta-learning objective of minimising the regret bound of learning. The resulting framework, termed Meta Mirror Descent (MetaMD), learns to accelerate optimisation speed. Unlike many meta-learned optimisers, it also supports convergence and generalisation guarantees and uniquely does so without requiring validation data. We evaluate our framework on a variety of tasks and architectures in terms of convergence rate and generalisation error and demonstrate strong performance.
△ Less
Submitted 5 March, 2022;
originally announced March 2022.
-
Optimization flows landing on the Stiefel manifold
Authors:
Bin Gao,
Simon Vary,
Pierre Ablin,
P. -A. Absil
Abstract:
We study a continuous-time system that solves optimization problems over the set of orthonormal matrices, which is also known as the Stiefel manifold. The resulting optimization flow follows a path that is not always on the manifold but asymptotically lands on the manifold. We introduce a generalized Stiefel manifold to which we extend the canonical metric of the Stiefel manifold. We show that the…
▽ More
We study a continuous-time system that solves optimization problems over the set of orthonormal matrices, which is also known as the Stiefel manifold. The resulting optimization flow follows a path that is not always on the manifold but asymptotically lands on the manifold. We introduce a generalized Stiefel manifold to which we extend the canonical metric of the Stiefel manifold. We show that the vector field of the proposed flow can be interpreted as the sum of a Riemannian gradient on a generalized Stiefel manifold and a normal vector. Moreover, we prove that the proposed flow globally converges to the set of critical points, and any local minimum and isolated critical point is asymptotically stable.
△ Less
Submitted 31 July, 2022; v1 submitted 18 February, 2022;
originally announced February 2022.
-
A Random Active Set Method for Strictly Convex Quadratic Problem with Simple Bounds
Authors:
Ran Gu,
Bing Gao
Abstract:
Active set method aims to find the correct active set of the optimal solution and it is a powerful method for solving strictly convex quadratic problem with bound constraints. To guarantee the finite step convergence, the existing active set methods all need strict conditions or some additional strategies, which greatly affect the efficiency of the algorithm. In this paper, we propose a random act…
▽ More
Active set method aims to find the correct active set of the optimal solution and it is a powerful method for solving strictly convex quadratic problem with bound constraints. To guarantee the finite step convergence, the existing active set methods all need strict conditions or some additional strategies, which greatly affect the efficiency of the algorithm. In this paper, we propose a random active set method which introduces randomness in the update of active set. We prove that it can converge in finite iterations with probability one without any conditions on the problem or any additional strategies. Numerical results show that the algorithm obtains the correct active set within a few iterations, and compared with the existing methods, it has better robustness and efficiency.
△ Less
Submitted 27 November, 2021;
originally announced November 2021.
-
Second-Order Mirror Descent: Convergence in Games Beyond Averaging and Discounting
Authors:
Bolin Gao,
Lacra Pavel
Abstract:
In this paper, we propose a second-order extension of the continuous-time game-theoretic mirror descent (MD) dynamics, referred to as MD2, which provably converges to mere (but not necessarily strict) variationally stable states (VSS) without using common auxiliary techniques such as time-averaging or discounting. We show that MD2 enjoys no-regret as well as an exponential rate of convergence towa…
▽ More
In this paper, we propose a second-order extension of the continuous-time game-theoretic mirror descent (MD) dynamics, referred to as MD2, which provably converges to mere (but not necessarily strict) variationally stable states (VSS) without using common auxiliary techniques such as time-averaging or discounting. We show that MD2 enjoys no-regret as well as an exponential rate of convergence towards strong VSS upon a slight modification. MD2 can also be used to derive many novel continuous-time primal-space dynamics. We then use stochastic approximation techniques to provide a convergence guarantee of discrete-time MD2 with noisy observations towards interior mere VSS. Selected simulations are provided to illustrate our results.
△ Less
Submitted 30 June, 2023; v1 submitted 18 November, 2021;
originally announced November 2021.
-
Scalable adaptive PDE solvers in arbitrary domains
Authors:
Kumar Saurabh,
Masado Ishii,
Milinda Fernando,
Boshun Gao,
Kendrick Tan,
Ming-Chen Hsu,
Adarsh Krishnamurthy,
Hari Sundar,
Baskar Ganapathysubramanian
Abstract:
Efficiently and accurately simulating partial differential equations (PDEs) in and around arbitrarily defined geometries, especially with high levels of adaptivity, has significant implications for different application domains. A key bottleneck in the above process is the fast construction of a `good' adaptively-refined mesh. In this work, we present an efficient novel octree-based adaptive discr…
▽ More
Efficiently and accurately simulating partial differential equations (PDEs) in and around arbitrarily defined geometries, especially with high levels of adaptivity, has significant implications for different application domains. A key bottleneck in the above process is the fast construction of a `good' adaptively-refined mesh. In this work, we present an efficient novel octree-based adaptive discretization approach capable of carving out arbitrarily shaped void regions from the parent domain: an essential requirement for fluid simulations around complex objects. Carving out objects produces an $\textit{incomplete}$ octree. We develop efficient top-down and bottom-up traversal methods to perform finite element computations on $\textit{incomplete}$ octrees. We validate the framework by (a) showing appropriate convergence analysis and (b) computing the drag coefficient for flow past a sphere for a wide range of Reynolds numbers ($\mathcal{O}(1-10^6)$) encompassing the drag crisis regime. Finally, we deploy the framework on a realistic geometry on a current project to evaluate COVID-19 transmission risk in classrooms.
△ Less
Submitted 8 August, 2021;
originally announced August 2021.
-
A Riemannian rank-adaptive method for low-rank matrix completion
Authors:
Bin Gao,
P. -A. Absil
Abstract:
The low-rank matrix completion problem can be solved by Riemannian optimization on a fixed-rank manifold. However, a drawback of the known approaches is that the rank parameter has to be fixed a priori. In this paper, we consider the optimization problem on the set of bounded-rank matrices. We propose a Riemannian rank-adaptive method, which consists of fixed-rank optimization, rank increase step…
▽ More
The low-rank matrix completion problem can be solved by Riemannian optimization on a fixed-rank manifold. However, a drawback of the known approaches is that the rank parameter has to be fixed a priori. In this paper, we consider the optimization problem on the set of bounded-rank matrices. We propose a Riemannian rank-adaptive method, which consists of fixed-rank optimization, rank increase step and rank reduction step. We explore its performance applied to the low-rank matrix completion problem. Numerical experiments on synthetic and real-world datasets illustrate that the proposed rank-adaptive method compares favorably with state-of-the-art algorithms. In addition, it shows that one can incorporate each aspect of this rank-adaptive framework separately into existing algorithms for the purpose of improving performance.
△ Less
Submitted 26 March, 2021;
originally announced March 2021.
-
Geometry of the symplectic Stiefel manifold endowed with the Euclidean metric
Authors:
Bin Gao,
Nguyen Thanh Son,
P. -A. Absil,
Tatjana Stykel
Abstract:
The symplectic Stiefel manifold, denoted by $\mathrm{Sp}(2p,2n)$, is the set of linear symplectic maps between the standard symplectic spaces $\mathbb{R}^{2p}$ and $\mathbb{R}^{2n}$. When $p=n$, it reduces to the well-known set of $2n\times 2n$ symplectic matrices. We study the Riemannian geometry of this manifold viewed as a Riemannian submanifold of the Euclidean space…
▽ More
The symplectic Stiefel manifold, denoted by $\mathrm{Sp}(2p,2n)$, is the set of linear symplectic maps between the standard symplectic spaces $\mathbb{R}^{2p}$ and $\mathbb{R}^{2n}$. When $p=n$, it reduces to the well-known set of $2n\times 2n$ symplectic matrices. We study the Riemannian geometry of this manifold viewed as a Riemannian submanifold of the Euclidean space $\mathbb{R}^{2n\times 2p}$. The corresponding normal space and projections onto the tangent and normal spaces are investigated. Moreover, we consider optimization problems on the symplectic Stiefel manifold. We obtain the expression of the Riemannian gradient with respect to the Euclidean metric, which then used in optimization algorithms. Numerical experiments on the nearest symplectic matrix problem and the symplectic eigenvalue problem illustrate the effectiveness of Euclidean-based algorithms.
△ Less
Submitted 28 February, 2021;
originally announced March 2021.
-
New Riemannian preconditioned algorithms for tensor completion via polyadic decomposition
Authors:
Shuyu Dong,
Bin Gao,
Yu Guan,
François Glineur
Abstract:
We propose new Riemannian preconditioned algorithms for low-rank tensor completion via the polyadic decomposition of a tensor. These algorithms exploit a non-Euclidean metric on the product space of the factor matrices of the low-rank tensor in the polyadic decomposition form. This new metric is designed using an approximation of the diagonal blocks of the Hessian of the tensor completion cost fun…
▽ More
We propose new Riemannian preconditioned algorithms for low-rank tensor completion via the polyadic decomposition of a tensor. These algorithms exploit a non-Euclidean metric on the product space of the factor matrices of the low-rank tensor in the polyadic decomposition form. This new metric is designed using an approximation of the diagonal blocks of the Hessian of the tensor completion cost function, thus has a preconditioning effect on these algorithms. We prove that the proposed Riemannian gradient descent algorithm globally converges to a stationary point of the tensor completion problem, with convergence rate estimates using the $Ł$ojasiewicz property. Numerical results on synthetic and real-world data suggest that the proposed algorithms are more efficient in memory and time compared to state-of-the-art algorithms. Moreover, the proposed algorithms display a greater tolerance for overestimated rank parameters in terms of the tensor recovery performance, thus enable a flexible choice of the rank parameter.
△ Less
Submitted 2 June, 2022; v1 submitted 26 January, 2021;
originally announced January 2021.
-
Symplectic eigenvalue problem via trace minimization and Riemannian optimization
Authors:
Nguyen Thanh Son,
P. -A. Absil,
Bin Gao,
Tatjana Stykel
Abstract:
We address the problem of computing the smallest symplectic eigenvalues and the corresponding eigenvectors of symmetric positive-definite matrices in the sense of Williamson's theorem. It is formulated as minimizing a trace cost function over the symplectic Stiefel manifold. We first investigate various theoretical aspects of this optimization problem such as characterizing the sets of critical po…
▽ More
We address the problem of computing the smallest symplectic eigenvalues and the corresponding eigenvectors of symmetric positive-definite matrices in the sense of Williamson's theorem. It is formulated as minimizing a trace cost function over the symplectic Stiefel manifold. We first investigate various theoretical aspects of this optimization problem such as characterizing the sets of critical points, saddle points, and global minimizers as well as proving that non-global local minimizers do not exist. Based on our recent results on constructing Riemannian structures on the symplectic Stiefel manifold and the associated optimization algorithms, we then propose solving the symplectic eigenvalue problem in the framework of Riemannian optimization. Moreover, a connection of the sought solution with the eigenvalues of a special class of Hamiltonian matrices is discussed. Numerical examples are presented.
△ Less
Submitted 7 January, 2021;
originally announced January 2021.
-
Multipliers Correction Methods for Optimization Problems over the Stiefel Manifold
Authors:
Lei Wang,
Bin Gao,
Xin Liu
Abstract:
We propose a class of multipliers correction methods to minimize a differentiable function over the Stiefel manifold. The proposed methods combine a function value reduction step with a proximal correction step. The former one searches along an arbitrary descent direction in the Euclidean space instead of a vector in the tangent space of the Stiefel manifold. Meanwhile, the latter one minimizes a…
▽ More
We propose a class of multipliers correction methods to minimize a differentiable function over the Stiefel manifold. The proposed methods combine a function value reduction step with a proximal correction step. The former one searches along an arbitrary descent direction in the Euclidean space instead of a vector in the tangent space of the Stiefel manifold. Meanwhile, the latter one minimizes a first-order proximal approximation of the objective function in the range space of the current iterate to make Lagrangian multipliers associated with orthogonality constraints symmetric at any accumulation point. The global convergence has been established for the proposed methods. Preliminary numerical experiments demonstrate that the new methods significantly outperform other state-of-the-art first-order approaches in solving various kinds of testing problems.
△ Less
Submitted 30 November, 2020;
originally announced November 2020.
-
Continuous-Time Convergence Rates in Potential and Monotone Games
Authors:
Bolin Gao,
Lacra Pavel
Abstract:
In this paper, we provide exponential rates of convergence to the interior Nash equilibrium for continuous-time dual-space game dynamics such as mirror descent (MD) and actor-critic (AC). We perform our analysis in $N$-player continuous concave games that satisfy certain monotonicity assumptions while possibly also admitting potential functions. In the first part of this paper, we provide a novel…
▽ More
In this paper, we provide exponential rates of convergence to the interior Nash equilibrium for continuous-time dual-space game dynamics such as mirror descent (MD) and actor-critic (AC). We perform our analysis in $N$-player continuous concave games that satisfy certain monotonicity assumptions while possibly also admitting potential functions. In the first part of this paper, we provide a novel relative characterization of monotone games and show that MD and its discounted version converge with $\mathcal{O}(e^{-βt})$ in relatively strongly and relatively hypo-monotone games, respectively. In the second part of this paper, we specialize our results to games that admit a relatively strongly concave potential and show AC converges with $\mathcal{O}(e^{-βt})$. These rates extend their known convergence conditions. Simulations are performed which empirically back up our results.
△ Less
Submitted 2 February, 2022; v1 submitted 20 November, 2020;
originally announced November 2020.
-
Industrial scale large eddy simulations (LES) with adaptive octree meshes using immersogeometric analysis
Authors:
Kumar Saurabh,
Boshun Gao,
Milinda Fernando,
Songzhe Xu,
Makrand A. Khanwale,
Biswajit Khara,
Ming-Chen Hsu,
Adarsh Krishnamurthy,
Hari Sundar,
Baskar Ganapathysubramanian
Abstract:
We present a variant of the immersed boundary method integrated with octree meshes for highly efficient and accurate Large-Eddy Simulations (LES) of flows around complex geometries. We demonstrate the scalability of the proposed method up to $\mathcal{O}(32K)$ processors. This is achieved by (a) rapid in-out tests; (b) adaptive quadrature for an accurate evaluation of forces; (c) tensorized evalua…
▽ More
We present a variant of the immersed boundary method integrated with octree meshes for highly efficient and accurate Large-Eddy Simulations (LES) of flows around complex geometries. We demonstrate the scalability of the proposed method up to $\mathcal{O}(32K)$ processors. This is achieved by (a) rapid in-out tests; (b) adaptive quadrature for an accurate evaluation of forces; (c) tensorized evaluation during matrix assembly. We showcase this method on two non-trivial applications: accurately computing the drag coefficient of a sphere across Reynolds numbers $1-10^6$ encompassing the drag crisis regime; simulating flow features across a semi-truck for investigating the effect of platooning on efficiency.
△ Less
Submitted 9 February, 2021; v1 submitted 28 August, 2020;
originally announced September 2020.
-
Alternating minimization algorithms for graph regularized tensor completion
Authors:
Yu Guan,
Shuyu Dong,
Bin Gao,
P. -A. Absil,
François Glineur
Abstract:
We consider a Canonical Polyadic (CP) decomposition approach to low-rank tensor completion (LRTC) by incorporating external pairwise similarity relations through graph Laplacian regularization on the CP factor matrices. The usage of graph regularization entails benefits in the learning accuracy of LRTC, but at the same time, induces coupling graph Laplacian terms that hinder the optimization of th…
▽ More
We consider a Canonical Polyadic (CP) decomposition approach to low-rank tensor completion (LRTC) by incorporating external pairwise similarity relations through graph Laplacian regularization on the CP factor matrices. The usage of graph regularization entails benefits in the learning accuracy of LRTC, but at the same time, induces coupling graph Laplacian terms that hinder the optimization of the tensor completion model. In order to solve graph-regularized LRTC, we propose efficient alternating minimization algorithms by leveraging the block structure of the underlying CP decomposition-based model. For the subproblems of alternating minimization, a linear conjugate gradient subroutine is specifically adapted to graph-regularized LRTC. Alternatively, we circumvent the complicating coupling effects of graph Laplacian terms by using an alternating directions method of multipliers. Based on the Kurdyka-Łojasiewicz property, we show that the sequence generated by the proposed algorithms globally converges to a critical point of the objective function. Moreover, the complexity and convergence rate are also derived. In addition, numerical experiments including synthetic data and real data show that the graph regularized tensor completion model has improved recovery results compared to those without graph regularization, and that the proposed algorithms achieve gains in time efficiency over existing algorithms.
△ Less
Submitted 11 November, 2023; v1 submitted 28 August, 2020;
originally announced August 2020.
-
An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory
Authors:
Bin Gao,
Guanghui Hu,
Yang Kuang,
Xin Liu
Abstract:
All-electron calculations play an important role in density functional theory, in which improving computational efficiency is one of the most needed and challenging tasks. In the model formulations, both nonlinear eigenvalue problem and total energy minimization problem pursue orthogonal solutions. Most existing algorithms for solving these two models invoke orthogonalization process either explic…
▽ More
All-electron calculations play an important role in density functional theory, in which improving computational efficiency is one of the most needed and challenging tasks. In the model formulations, both nonlinear eigenvalue problem and total energy minimization problem pursue orthogonal solutions. Most existing algorithms for solving these two models invoke orthogonalization process either explicitly or implicitly in each iteration. Their efficiency suffers from this process in view of its cubic complexity and low parallel scalability in terms of the number of electrons for large scale systems. To break through this bottleneck, we propose an orthogonalization-free algorithm framework based on the total energy minimization problem. It is shown that the desired orthogonality can be gradually achieved without invoking orthogonalization in each iteration. Moreover, this framework fully consists of Basic Linear Algebra Subprograms (BLAS) operations and thus can be naturally parallelized. The global convergence of the proposed algorithm is established. We also present a precondition technique which can dramatically accelerate the convergence of the algorithm. The numerical experiments on all-electron calculations show the efficiency and high scalability of the proposed algorithm.
△ Less
Submitted 28 July, 2020;
originally announced July 2020.
-
On fair entropy of the tent family
Authors:
Bing Gao,
Rui Gao
Abstract:
The notions of fair measure and fair entropy were introduced by Misiurewicz and Rodrigues recently, and discussed in detail for piecewise monotone interval maps. In particular, they showed that the fair entropy $h(a)$ of the tent map $f_a$, as a function of the parameter $a=\exp(h_{top}(f_a))$, is continuous and strictly increasing on $[\sqrt{2},2]$. In this short note, we extend the last result a…
▽ More
The notions of fair measure and fair entropy were introduced by Misiurewicz and Rodrigues recently, and discussed in detail for piecewise monotone interval maps. In particular, they showed that the fair entropy $h(a)$ of the tent map $f_a$, as a function of the parameter $a=\exp(h_{top}(f_a))$, is continuous and strictly increasing on $[\sqrt{2},2]$. In this short note, we extend the last result and characterize regularity of the function $h$ precisely. We prove that $h$ is $\frac{1}{2}$-Hölder continuous on $[\sqrt{2},2]$ and identify its best Hölder exponent on each subinterval of $[\sqrt{2},2]$. On the other hand, parallel to a recent result on topological entropy of the quadratic family due to Dobbs and Mihalache, we give a formula of pointwise Hölder exponents of $h$ at parameters chosen in an explicitly constructed set of full measure. This formula particularly implies that the derivative of $h$ vanishes almost everywhere.
△ Less
Submitted 23 July, 2020;
originally announced July 2020.
-
Riemannian Optimization on the Symplectic Stiefel Manifold
Authors:
Bin Gao,
Nguyen Thanh Son,
P. -A. Absil,
Tatjana Stykel
Abstract:
The symplectic Stiefel manifold, denoted by $\mathrm{Sp}(2p,2n)$, is the set of linear symplectic maps between the standard symplectic spaces $\mathbb{R}^{2p}$ and $\mathbb{R}^{2n}$. When $p=n$, it reduces to the well-known set of $2n\times 2n$ symplectic matrices. Optimization problems on $\mathrm{Sp}(2p,2n)$ find applications in various areas, such as optics, quantum physics, numerical linear al…
▽ More
The symplectic Stiefel manifold, denoted by $\mathrm{Sp}(2p,2n)$, is the set of linear symplectic maps between the standard symplectic spaces $\mathbb{R}^{2p}$ and $\mathbb{R}^{2n}$. When $p=n$, it reduces to the well-known set of $2n\times 2n$ symplectic matrices. Optimization problems on $\mathrm{Sp}(2p,2n)$ find applications in various areas, such as optics, quantum physics, numerical linear algebra and model order reduction of dynamical systems. The purpose of this paper is to propose and analyze gradient-descent methods on $\mathrm{Sp}(2p,2n)$, where the notion of gradient stems from a Riemannian metric. We consider a novel Riemannian metric on $\mathrm{Sp}(2p,2n)$ akin to the canonical metric of the (standard) Stiefel manifold. In order to perform a feasible step along the antigradient, we develop two types of search strategies: one is based on quasi-geodesic curves, and the other one on the symplectic Cayley transform. The resulting optimization algorithms are proved to converge globally to critical points of the objective function. Numerical experiments illustrate the efficiency of the proposed methods.
△ Less
Submitted 26 June, 2020;
originally announced June 2020.
-
Shapes of hyperbolic triangles and once-punctured torus groups
Authors:
Sang-hyun Kim,
Thomas Koberda,
Jaejeong Lee,
Ken'ichi Ohshika,
Ser Peow Tan,
with an appendix by Xinghua Gao
Abstract:
Let $Δ$ be a hyperbolic triangle with a fixed area $\varphi$. We prove that for all but countably many $\varphi$, generic choices of $Δ$ have the property that the group generated by the $π$--rotations about the midpoints of the sides of the triangle admits no nontrivial relations. By contrast, we show for all $\varphi\in(0,π)\setminus\mathbb{Q}π$, a dense set of triangles does afford nontrivial r…
▽ More
Let $Δ$ be a hyperbolic triangle with a fixed area $\varphi$. We prove that for all but countably many $\varphi$, generic choices of $Δ$ have the property that the group generated by the $π$--rotations about the midpoints of the sides of the triangle admits no nontrivial relations. By contrast, we show for all $\varphi\in(0,π)\setminus\mathbb{Q}π$, a dense set of triangles does afford nontrivial relations, which in the generic case map to hyperbolic translations. To establish this fact, we study the deformation space $\mathfrak{C}_θ$ of singular hyperbolic metrics on a torus with a single cone point of angle $θ=2(π-\varphi)$, and answer an analogous question for the holonomy map $ρ_ξ$ of such a hyperbolic structure $ξ$. In an appendix by X.~Gao, concrete examples of $θ$ and $ξ\in\mathfrak{C}_θ$ are given where the image of each $ρ_ξ$ is finitely presented, non-free and torsion-free; in fact, those images will be isomorphic to the fundamental groups of closed hyperbolic 3--manifolds.
△ Less
Submitted 1 April, 2021; v1 submitted 3 June, 2020;
originally announced June 2020.
-
Continuous-time Discounted Mirror-Descent Dynamics in Monotone Concave Games
Authors:
Bolin Gao,
Lacra Pavel
Abstract:
In this paper, we consider concave continuous-kernel games characterized by monotonicity properties and propose discounted mirror descent-type dynamics. We introduce two classes of dynamics whereby the associated mirror map is constructed based on a strongly convex or a Legendre regularizer. Depending on the properties of the regularizer we show that these new dynamics can converge asymptotically…
▽ More
In this paper, we consider concave continuous-kernel games characterized by monotonicity properties and propose discounted mirror descent-type dynamics. We introduce two classes of dynamics whereby the associated mirror map is constructed based on a strongly convex or a Legendre regularizer. Depending on the properties of the regularizer we show that these new dynamics can converge asymptotically in concave games with monotone (negative) pseudo-gradient. Furthermore, we show that when the regularizer enjoys strong convexity, the resulting dynamics can converge even in games with hypo-monotone (negative) pseudo-gradient, which corresponds to a shortage of monotonicity.
△ Less
Submitted 7 December, 2019;
originally announced December 2019.
-
Phase retrieval for sub-Gaussian measurements
Authors:
Bing Gao,
Haixia Liu,
Yang Wang
Abstract:
Generally, phase retrieval problem can be viewed as the reconstruction of a function/signal from only the magnitude of the linear measurements. These measurements can be, for example, the Fourier transform of the density function. Computationally the phase retrieval problem is very challenging. Many algorithms for phase retrieval are based on i.i.d. Gaussian random measurements. However, Gaussian…
▽ More
Generally, phase retrieval problem can be viewed as the reconstruction of a function/signal from only the magnitude of the linear measurements. These measurements can be, for example, the Fourier transform of the density function. Computationally the phase retrieval problem is very challenging. Many algorithms for phase retrieval are based on i.i.d. Gaussian random measurements. However, Gaussian random measurements remain one of the very few classes of measurements. In this paper, we develop an efficient phase retrieval algorithm for sub-gaussian random frames. We provide a general condition for measurements and develop a modified spectral initialization. In the algorithm, we first obtain a good approximation of the solution through the initialization, and from there we useWirtinger Flow to solve for the solution. We prove that the algorithm converges to the global minimizer linearly.
△ Less
Submitted 20 November, 2019;
originally announced November 2019.
-
Frobenius-Perron theory for projective schemes
Authors:
J. M. Chen,
Z. B. Gao,
E. Wicks,
J. J. Zhang,
X-. H. Zhang,
H. Zhu
Abstract:
The Frobenius-Perron theory of an endofunctor of a $\Bbbk$-linear category (recently introduced in [CG]) provides new invariants for abelian and triangulated categories. Here we study Frobenius-Perron type invariants for derived categories of commutative and noncommutative projective schemes. In particular, we calculate the Frobenius-Perron dimension for domestic and tubular weighted projective li…
▽ More
The Frobenius-Perron theory of an endofunctor of a $\Bbbk$-linear category (recently introduced in [CG]) provides new invariants for abelian and triangulated categories. Here we study Frobenius-Perron type invariants for derived categories of commutative and noncommutative projective schemes. In particular, we calculate the Frobenius-Perron dimension for domestic and tubular weighted projective lines, define Frobenius-Perron generalizations of Calabi-Yau and Kodaira dimensions, and provide examples. We apply this theory to the derived categories associated to certain Artin-Schelter regular and finite-dimensional algebras.
△ Less
Submitted 16 December, 2021; v1 submitted 4 July, 2019;
originally announced July 2019.
-
Perturbed Amplitude Flow for Phase Retrieval
Authors:
Bing Gao,
Xinwei Sun,
Yang Wang,
Zhiqiang Xu
Abstract:
In this paper, we propose a new non-convex algorithm for solving the phase retrieval problem, i.e., the reconstruction of a signal $ \vx\in\H^n $ ($\H=\R$ or $\C$) from phaseless samples $ b_j=\abs{\langle \va_j, \vx\rangle } $, $ j=1,\ldots,m $. The proposed algorithm solves a new proposed model, perturbed amplitude-based model, for phase retrieval and is correspondingly named as {\em Perturbed A…
▽ More
In this paper, we propose a new non-convex algorithm for solving the phase retrieval problem, i.e., the reconstruction of a signal $ \vx\in\H^n $ ($\H=\R$ or $\C$) from phaseless samples $ b_j=\abs{\langle \va_j, \vx\rangle } $, $ j=1,\ldots,m $. The proposed algorithm solves a new proposed model, perturbed amplitude-based model, for phase retrieval and is correspondingly named as {\em Perturbed Amplitude Flow} (PAF). We prove that PAF can recover $c\vx$ ($\abs{c} = 1$) under $\mathcal{O}(n)$ Gaussian random measurements (optimal order of measurements). Starting with a designed initial point, our PAF algorithm iteratively converges to the true solution at a linear rate for both real and complex signals. Besides, PAF algorithm needn't any truncation or re-weighted procedure, so it enjoys simplicity for implementation. The effectiveness and benefit of the proposed method are validated by both the simulation studies and the experiment of recovering natural images.
△ Less
Submitted 14 October, 2020; v1 submitted 23 April, 2019;
originally announced April 2019.
-
Quenched asymptotics for a 1-d stochastic heat equation driven by a rough spatial noise
Authors:
Prakash Chakraborty,
Xia Chen,
Bo Gao,
Samy Tindel
Abstract:
In this note we consider the parabolic Anderson model in one dimension with time-independent fractional noise $\dot{W}$ in space. We consider the case $H<\frac{1}{2}$ and get existence and uniqueness of solution. In order to find the quenched asymptotics for the solution we consider its Feynman-Kac representation and explore the asymptotics of the principal eigenvalue for a random operator of the…
▽ More
In this note we consider the parabolic Anderson model in one dimension with time-independent fractional noise $\dot{W}$ in space. We consider the case $H<\frac{1}{2}$ and get existence and uniqueness of solution. In order to find the quenched asymptotics for the solution we consider its Feynman-Kac representation and explore the asymptotics of the principal eigenvalue for a random operator of the form $\frac{1}{2} Δ+ \dot{W}$.
△ Less
Submitted 9 October, 2018;
originally announced October 2018.
-
Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints
Authors:
Bin Gao,
Xin Liu,
Ya-xiang Yuan
Abstract:
To construct a parallel approach for solving optimization problems with orthogonality constraints is usually regarded as an extremely difficult mission, due to the low scalability of the orthonormalization procedure. However, such demand is particularly huge in some application areas such as materials computation. In this paper, we propose a proximal linearized augmented Lagrangian algorithm (PLAM…
▽ More
To construct a parallel approach for solving optimization problems with orthogonality constraints is usually regarded as an extremely difficult mission, due to the low scalability of the orthonormalization procedure. However, such demand is particularly huge in some application areas such as materials computation. In this paper, we propose a proximal linearized augmented Lagrangian algorithm (PLAM) for solving optimization problems with orthogonality constraints. Unlike the classical augmented Lagrangian methods, in our algorithm, the prime variables are updated by minimizing a proximal linearized approximation of the augmented Lagrangian function, meanwhile the dual variables are updated by a closed-form expression which holds at any first-order stationary point. The orthonormalization procedure is only invoked once at the last step of the above mentioned algorithm if high-precision feasibility is needed. Consequently, the main parts of the proposed algorithm can be parallelized naturally. We establish global subsequence convergence, worst-case complexity and local convergence rate for PLAM under some mild assumptions. To reduce the sensitivity of the penalty parameter, we put forward a modification of PLAM, which is called parallelizable column-wise block minimization of PLAM (PCAL). Numerical experiments in serial illustrate that the novel updating rule for the Lagrangian multipliers significantly accelerates the convergence of PLAM and makes it comparable with the existent feasible solvers for optimization problems with orthogonality constraints, and the performance of PCAL does not highly rely on the choice of the penalty parameter. Numerical experiments under parallel environment demonstrate that PCAL attains good performance and high scalability in solving discretized Kohn-Sham total energy minimization problems.
△ Less
Submitted 24 October, 2018; v1 submitted 9 October, 2018;
originally announced October 2018.
-
On Passivity, Reinforcement Learning and Higher-Order Learning in Multi-Agent Finite Games
Authors:
Bolin Gao,
Lacra Pavel
Abstract:
In this paper, we propose a passivity-based methodology for analysis and design of reinforcement learning in multi-agent finite games. Starting from a known exponentially-discounted reinforcement learning scheme, we show that convergence to a Nash distribution can be shown in the class of games characterized by the monotonicity property of their (negative) payoff. We further exploit passivity to p…
▽ More
In this paper, we propose a passivity-based methodology for analysis and design of reinforcement learning in multi-agent finite games. Starting from a known exponentially-discounted reinforcement learning scheme, we show that convergence to a Nash distribution can be shown in the class of games characterized by the monotonicity property of their (negative) payoff. We further exploit passivity to propose a class of higher-order schemes that preserve convergence properties, can improve the speed of convergence and can even converge in cases whereby their first-order counterpart fail to converge. We demonstrate these properties through numerical simulations for several representative games.
△ Less
Submitted 13 August, 2018;
originally announced August 2018.
-
On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
Authors:
Bolin Gao,
Lacra Pavel
Abstract:
In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax function is the monotone gradient map of the log-sum-exp function. By exploiting this connection, we show that the inverse temperature parameter determines the L…
▽ More
In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax function is the monotone gradient map of the log-sum-exp function. By exploiting this connection, we show that the inverse temperature parameter determines the Lipschitz and co-coercivity properties of the softmax function. We then demonstrate the usefulness of these properties through an application in game-theoretic reinforcement learning.
△ Less
Submitted 20 August, 2018; v1 submitted 3 April, 2017;
originally announced April 2017.
-
On the Łojasiewicz Exponent of the Quadratic Sphere Constrained Optimization Problem
Authors:
Bin Gao,
Xin Liu,
Xiaojun Chen,
Ya-xiang Yuan
Abstract:
In this paper, we prove that the global version of the $Ł$ojasiewicz gradient inequality holds for quadratic sphere constrained optimization problem with exponent $θ=\frac{3}{4}$. An example from Ting Kei Pong shows that $θ=\frac{3}{4}$ is tight. This is the first $Ł$ojasiewicz gradient inequality established for the sphere constrained optimization problem with a linear term.
In this paper, we prove that the global version of the $Ł$ojasiewicz gradient inequality holds for quadratic sphere constrained optimization problem with exponent $θ=\frac{3}{4}$. An example from Ting Kei Pong shows that $θ=\frac{3}{4}$ is tight. This is the first $Ł$ojasiewicz gradient inequality established for the sphere constrained optimization problem with a linear term.
△ Less
Submitted 22 February, 2021; v1 submitted 26 November, 2016;
originally announced November 2016.
-
Phaseless Rcovery using Gauss-Newton Method
Authors:
Bing Gao,
Zhiqiang Xu
Abstract:
In this paper, we develop a concrete algorithm for phase retrieval, which we refer to as Gauss-Newton algorithm. In short, this algorithm starts with a good initial estimation, which is obtained by a modified spectral method, and then update the iteration point by a Gauss-Newton iteration step. We prove that a re-sampled version of this algorithm quadratically converges to the solution for the rea…
▽ More
In this paper, we develop a concrete algorithm for phase retrieval, which we refer to as Gauss-Newton algorithm. In short, this algorithm starts with a good initial estimation, which is obtained by a modified spectral method, and then update the iteration point by a Gauss-Newton iteration step. We prove that a re-sampled version of this algorithm quadratically converges to the solution for the real case with the number of random measurements being nearly minimal. Numerical experiments also show that Gauss-Newton method has better performance over the other algorithms.
△ Less
Submitted 8 February, 2018; v1 submitted 27 June, 2016;
originally announced June 2016.
-
The $ \ell_1 $-analysis with redundant dictionary in phase retrieval
Authors:
Bing Gao
Abstract:
This article presents new results concerning the recovery of a signal from magnitude only measurements where the signal is not sparse in an orthonormal basis but in a redundant dictionary. To solve this phaseless problem, we analyze the $ \ell_1 $-analysis model. Firstly we investigate the noiseless case with presenting a null space property of the measurement matrix under which the $ \ell_1 $-ana…
▽ More
This article presents new results concerning the recovery of a signal from magnitude only measurements where the signal is not sparse in an orthonormal basis but in a redundant dictionary. To solve this phaseless problem, we analyze the $ \ell_1 $-analysis model. Firstly we investigate the noiseless case with presenting a null space property of the measurement matrix under which the $ \ell_1 $-analysis model provide an exact recovery. Secondly we introduce a new property (S-DRIP) of the measurement matrix. By solving the $ \ell_1 $-analysis model, we prove that this property can guarantee a stable recovery of real signals that are nearly sparse in highly overcomplete dictionaries.
△ Less
Submitted 2 June, 2016;
originally announced June 2016.
-
Stable Signal Recovery from Phaseless Measurements
Authors:
Bing Gao,
Yang Wang,
Zhiqiang Xu
Abstract:
The aim of this paper is to study the stability of the $\ell_1$ minimization for the compressive phase retrieval and to extend the instance-optimality in compressed sensing to the real phase retrieval setting. We first show that the $m={\mathcal O}(k\log(N/k))$ measurements is enough to guarantee the $\ell_1$ minimization to recover $k$-sparse signals stably provided the measurement matrix $A$ sat…
▽ More
The aim of this paper is to study the stability of the $\ell_1$ minimization for the compressive phase retrieval and to extend the instance-optimality in compressed sensing to the real phase retrieval setting. We first show that the $m={\mathcal O}(k\log(N/k))$ measurements is enough to guarantee the $\ell_1$ minimization to recover $k$-sparse signals stably provided the measurement matrix $A$ satisfies the strong RIP property. We second investigate the phaseless instance-optimality with presenting a null space property of the measurement matrix $A$ under which there exists a decoder $Δ$ so that the phaseless instance-optimality holds. We use the result to study the phaseless instance-optimality for the $\ell_1$ norm. The results build a parallel for compressive phase retrieval with the classical compressive sensing.
△ Less
Submitted 9 April, 2015; v1 submitted 5 April, 2015;
originally announced April 2015.
-
Integral with respect to the $G$-Brownian local time
Authors:
Litan Yan,
Xichao Sun,
Bo Gao
Abstract:
Let ${\mathscr L}$ be the local time of $G$-Brownian motion $B$. In this paper, we prove the existence of the quadratic covariation $<f(B),B>_{t}$ and the integral $\int_{\mathbb R}f(x){\mathscr L}(dx,t)$. Moreover, a sublinear version of the Bouleau-Yor identity $$ \int_{\mathbb R}f(x){\mathscr L}(dx,t)=-<f(B),B>_{t} $$ is showed to hold under some suitable conditions. These allow us to write the…
▽ More
Let ${\mathscr L}$ be the local time of $G$-Brownian motion $B$. In this paper, we prove the existence of the quadratic covariation $<f(B),B>_{t}$ and the integral $\int_{\mathbb R}f(x){\mathscr L}(dx,t)$. Moreover, a sublinear version of the Bouleau-Yor identity $$ \int_{\mathbb R}f(x){\mathscr L}(dx,t)=-<f(B),B>_{t} $$ is showed to hold under some suitable conditions. These allow us to write the Itô's formula for $C^1$-functions.
△ Less
Submitted 27 December, 2012;
originally announced December 2012.
-
The Bouleau-Yor identity for a bi-fractional Brownian motion
Authors:
Litan Yan,
Bo Gao,
Junfeng Liu
Abstract:
Let $B$ be a bi-fractional Brownian motion with indices $H\in (0,1),K\in (0,1]$, $2HK=1$ and let ${\mathscr L}(x,t)$ be its local time process. We construct a Banach space ${\mathscr H}$ of measurable functions such that the quadratic covariation $[f(B),B]$ and the integral $\int_{\mathbb R}f(x){\mathscr L}(dx,t)$ exist provided $f\in {\mathscr H}$. Moreover, the Bouleau-Yor identity…
▽ More
Let $B$ be a bi-fractional Brownian motion with indices $H\in (0,1),K\in (0,1]$, $2HK=1$ and let ${\mathscr L}(x,t)$ be its local time process. We construct a Banach space ${\mathscr H}$ of measurable functions such that the quadratic covariation $[f(B),B]$ and the integral $\int_{\mathbb R}f(x){\mathscr L}(dx,t)$ exist provided $f\in {\mathscr H}$. Moreover, the Bouleau-Yor identity $$ [f(B),B]_t=-2^{1-K}\int_{\mathbb R}f(x){\mathscr L}(dx,t),\qquad t\geq 0, $$ holds for all $f\in {\mathscr H}$.
△ Less
Submitted 27 December, 2012;
originally announced December 2012.
-
Limit Cycle Bifurcations from Centers of Symmetric Hamiltonian Systems Perturbing by Cubic Polynomials
Authors:
Zhaoping Hu,
Bin Gao,
Valery G. Romanovski
Abstract:
In this paper, we consider some cubic near-Hamiltonian systems obtained from perturbing the symmetric cubic Hamiltonian system with two symmetric singular points by cubic polynomials. First, following Han [2012] we develop a method to study the analytical property of the Melnikov function near the origin for near-Hamiltonian system having the origin as its elementary center or nilpotent center. Ba…
▽ More
In this paper, we consider some cubic near-Hamiltonian systems obtained from perturbing the symmetric cubic Hamiltonian system with two symmetric singular points by cubic polynomials. First, following Han [2012] we develop a method to study the analytical property of the Melnikov function near the origin for near-Hamiltonian system having the origin as its elementary center or nilpotent center. Based on the method, a computationally efficient algorithm is established to systematically compute the coefficients of Melnikov function. Then, we consider the symmetric singular points and present the conditions for one of them to be elementary center or nilpotent center. Under the condition for the singular point to be a center, we obtain the normal form of the Hamiltonian systems near the center. Moreover, perturbing the symmetric cubic Hamiltonian systems by cubic polynomials, we consider limit cycles bifurcating from the center using the algorithm to compute the coefficients of Melnikov function. Finally, perturbing the symmetric hamiltonian system by symmetric cubic polynomials, we consider the number of limit cycles near one of the symmetric centers of the symmetric near-Hamiltonian system, which is same to that of another center.
△ Less
Submitted 12 January, 2012;
originally announced January 2012.
-
Summability implies Collet-Eckmann almost surely
Authors:
Bing Gao,
Weixiao Shen
Abstract:
We provide a strengthened version of the famous Jakobson's theorem. Consider an interval map $f$ satisfying a summability condition. For a generic one-parameter family $f_t$ of maps with $f_0=f$, we prove that $t=0$ is a Lebesgue density point of the set of parameters for which $f_t$ satisfies both the Collect-Eckmann condition and a strong polynomial recurrence condition.
We provide a strengthened version of the famous Jakobson's theorem. Consider an interval map $f$ satisfying a summability condition. For a generic one-parameter family $f_t$ of maps with $f_0=f$, we prove that $t=0$ is a Lebesgue density point of the set of parameters for which $f_t$ satisfies both the Collect-Eckmann condition and a strong polynomial recurrence condition.
△ Less
Submitted 8 November, 2012; v1 submitted 16 November, 2011;
originally announced November 2011.
