Showing 1–17 of 17 results for author: Haddock, J

Block Matrix and Tensor Randomized Kaczmarz Methods for Linear Feasibility Problems

Authors: Minxin Zhang, Jamie Haddock, Deanna Needell

Abstract: The randomized Kaczmarz methods are a popular and effective family of iterative methods for solving large-scale linear systems of equations, which have also been applied to linear feasibility problems. In this work, we propose a new block variant of the randomized Kaczmarz method, B-MRK, for solving linear feasibility problems defined by matrices. We show that B-MRK converges linearly in expectati… ▽ More The randomized Kaczmarz methods are a popular and effective family of iterative methods for solving large-scale linear systems of equations, which have also been applied to linear feasibility problems. In this work, we propose a new block variant of the randomized Kaczmarz method, B-MRK, for solving linear feasibility problems defined by matrices. We show that B-MRK converges linearly in expectation to the feasible region.Furthermore, we extend the method to solve tensor linear feasibility problems defined under the tensor t-product. A tensor randomized Kaczmarz (TRK) method, TRK-L, is proposed for solving linear feasibility problems that involve mixed equality and inequality constraints. Additionally, we introduce another TRK method, TRK-LB, specifically tailored for cases where the feasible region is defined by linear equality constraints coupled with bound constraints on the variables. We show that both of the TRK methods converge linearly in expectation to the feasible region. Moreover, the effectiveness of our methods is demonstrated through numerical experiments on various Gaussian random data and applications in image deblurring. △ Less

Submitted 17 June, 2024; originally announced June 2024.

On Quantile Randomized Kaczmarz for Linear Systems with Time-Varying Noise and Corruption

Authors: Nestor Coria, Jamie Haddock, Jaime Pacheco

Abstract: Large-scale systems of linear equations arise in machine learning, medical imaging, sensor networks, and in many areas of data science. When the scale of the systems are extreme, it is common for a fraction of the data or measurements to be corrupted. The Quantile Randomized Kaczmarz (QRK) method is known to converge on large-scale systems of linear equations Ax=b that are inconsistent due to stat… ▽ More Large-scale systems of linear equations arise in machine learning, medical imaging, sensor networks, and in many areas of data science. When the scale of the systems are extreme, it is common for a fraction of the data or measurements to be corrupted. The Quantile Randomized Kaczmarz (QRK) method is known to converge on large-scale systems of linear equations Ax=b that are inconsistent due to static corruptions in the measurement vector b. We prove that QRK converges even for systems corrupted by time-varying perturbations. Additionally, we prove that QRK converges up to a convergence horizon on systems affected by time-varying noise and demonstrate that the noise affects only the convergence horizon of the method, and not the rate of convergence. Finally, we utilize Markov's inequality to prove a lower bound on the probability that the largest entries of the QRK residual reveal the time-varying corruption in each iteration. We present numerical experiments which illustrate our theoretical results. △ Less

Submitted 28 March, 2024; originally announced March 2024.

On Subsampled Quantile Randomized Kaczmarz

Authors: Jamie Haddock, Anna Ma, Elizaveta Rebrova

Abstract: When solving noisy linear systems Ax = b + c, the theoretical and empirical performance of stochastic iterative methods, such as the Randomized Kaczmarz algorithm, depends on the noise level. However, if there are a small number of highly corrupt measurements, one can instead use quantile-based methods to guarantee convergence to the solution x of the system, despite the presence of noise. Such me… ▽ More When solving noisy linear systems Ax = b + c, the theoretical and empirical performance of stochastic iterative methods, such as the Randomized Kaczmarz algorithm, depends on the noise level. However, if there are a small number of highly corrupt measurements, one can instead use quantile-based methods to guarantee convergence to the solution x of the system, despite the presence of noise. Such methods require the computation of the entire residual vector, which may not be desirable or even feasible in some cases. In this work, we analyze the sub-sampled quantile Randomized Kaczmarz (sQRK) algorithm for solving large-scale linear systems which utilize a sub-sampled residual to approximate the quantile threshold. We prove that this method converges to the unique solution to the linear system and provide numerical experiments that support our theoretical findings. We additionally remark on the extremely small sample size case and demonstrate the importance of interplay between the choice of quantile and subset size. △ Less

Submitted 15 August, 2023; originally announced August 2023.

Zero Forcing with Random Sets

Authors: Bryan Curtis, Luyining Gan, Jamie Haddock, Rachel Lawrence, Sam Spiro

Abstract: Given a graph $G$ and a real number $0\le p\le 1$, we define the random set $B_p(G)\subset V(G)$ by including each vertex independently and with probability $p$. We investigate the probability that the random set $B_p(G)$ is a zero forcing set of $G$. In particular, we prove that for large $n$, this probability for trees is upper bounded by the corresponding probability for a path graph. Given a m… ▽ More Given a graph $G$ and a real number $0\le p\le 1$, we define the random set $B_p(G)\subset V(G)$ by including each vertex independently and with probability $p$. We investigate the probability that the random set $B_p(G)$ is a zero forcing set of $G$. In particular, we prove that for large $n$, this probability for trees is upper bounded by the corresponding probability for a path graph. Given a minimum degree condition, we also prove a conjecture of Boyer et.\ al.\ regarding the number of zero forcing sets of a given size that a graph can have. △ Less

Submitted 26 August, 2022; originally announced August 2022.

Comments: 22 pages, 5 figures

MSC Class: 05C15; 05C05; 05C31

Nonbacktracking spectral clustering of nonuniform hypergraphs

Authors: Philip Chodrow, Nicole Eikmeier, Jamie Haddock

Abstract: Spectral methods offer a tractable, global framework for clustering in graphs via eigenvector computations on graph matrices. Hypergraph data, in which entities interact on edges of arbitrary size, poses challenges for matrix representations and therefore for spectral clustering. We study spectral clustering for nonuniform hypergraphs based on the hypergraph nonbacktracking operator. After reviewi… ▽ More Spectral methods offer a tractable, global framework for clustering in graphs via eigenvector computations on graph matrices. Hypergraph data, in which entities interact on edges of arbitrary size, poses challenges for matrix representations and therefore for spectral clustering. We study spectral clustering for nonuniform hypergraphs based on the hypergraph nonbacktracking operator. After reviewing the definition of this operator and its basic properties, we prove a theorem of Ihara-Bass type which allows eigenpair computations to take place on a smaller matrix, often enabling faster computation. We then propose an alternating algorithm for inference in a hypergraph stochastic blockmodel via linearized belief-propagation which involves a spectral clustering step again using nonbacktracking operators. We provide proofs related to this algorithm that both formalize and extend several previous results. We pose several conjectures about the limits of spectral methods and detectability in hypergraph stochastic blockmodels in general, supporting these with in-expectation analysis of the eigeinpairs of our studied operators. We perform experiments in real and synthetic data that demonstrate the benefits of hypergraph methods over graph-based ones when interactions of different sizes carry different information about cluster structure. △ Less

Submitted 3 September, 2022; v1 submitted 26 April, 2022; originally announced April 2022.

Comments: Main text: 26 pages, 6 figures. Appendix and references: 23 pages, 4 figures

MSC Class: 05C50; 05C65; 15A18; 62H30; 62R07; 91D30

Semi-supervised Nonnegative Matrix Factorization for Document Classification

Authors: Jamie Haddock, Lara Kassab, Sixian Li, Alona Kryshchenko, Rachel Grotheer, Elena Sizikova, Chuntian Wang, Thomas Merkh, RWMA Madushani, Miju Ahn, Deanna Needell, Kathryn Leonard

Abstract: We propose new semi-supervised nonnegative matrix factorization (SSNMF) models for document classification and provide motivation for these models as maximum likelihood estimators. The proposed SSNMF models simultaneously provide both a topic model and a model for classification, thereby offering highly interpretable classification results. We derive training methods using multiplicative updates f… ▽ More We propose new semi-supervised nonnegative matrix factorization (SSNMF) models for document classification and provide motivation for these models as maximum likelihood estimators. The proposed SSNMF models simultaneously provide both a topic model and a model for classification, thereby offering highly interpretable classification results. We derive training methods using multiplicative updates for each new model, and demonstrate the application of these models to single-label and multi-label document classification, although the models are flexible to other supervised learning tasks such as regression. We illustrate the promise of these models and training methods on document classification datasets (e.g., 20 Newsgroups, Reuters). △ Less

Submitted 28 February, 2022; originally announced March 2022.

Comments: arXiv admin note: substantial text overlap with arXiv:2010.07956

Paving the Way for Consensus: Convergence of Block Gossip Algorithms

Authors: Jamie Haddock, Benjamin Jarman, Chen Yap

Abstract: Gossip protocols are popular methods for average consensus problems in distributed computing. We prove new convergence guarantees for a variety of such protocols, including path, clique, and synchronous pairwise gossip. These arise by exploiting the connection between these protocols and the block randomized Kaczmarz method for solving linear systems. Moreover, we extend existing convergence resul… ▽ More Gossip protocols are popular methods for average consensus problems in distributed computing. We prove new convergence guarantees for a variety of such protocols, including path, clique, and synchronous pairwise gossip. These arise by exploiting the connection between these protocols and the block randomized Kaczmarz method for solving linear systems. Moreover, we extend existing convergence results for block randomized Kaczmarz to allow for a more general choice of blocks, rank-deficient systems, and provide a tighter convergence rate guarantee. We furthermore apply this analysis to inconsistent consensus models and obtain similar guarantees. An extensive empirical analysis of these methods is provided for a variety of synthetic networks. △ Less

Submitted 27 October, 2021; originally announced October 2021.

Comments: 21 pages, 19 figures

On Application of Block Kaczmarz Methods in Matrix Factorization

Authors: Edwin Chau, Jamie Haddock

Abstract: Matrix factorization techniques compute low-rank product approximations of high dimensional data matrices and as a result, are often employed in recommender systems and collaborative filtering applications. However, many algorithms for this task utilize an exact least-squares solver whose computation is time consuming and memory-expensive. In this paper we discuss and test a block Kaczmarz solver… ▽ More Matrix factorization techniques compute low-rank product approximations of high dimensional data matrices and as a result, are often employed in recommender systems and collaborative filtering applications. However, many algorithms for this task utilize an exact least-squares solver whose computation is time consuming and memory-expensive. In this paper we discuss and test a block Kaczmarz solver that replaces the least-squares subroutine in the common alternating scheme for matrix factorization. This variant trades a small increase in factorization error for significantly faster algorithmic performance. In doing so we find block sizes that produce a solution comparable to that of the least-squares solver for only a fraction of the runtime and working memory requirement. △ Less

Submitted 20 October, 2020; originally announced October 2020.

Comments: 13 pages, 5 figures

Semi-supervised NMF Models for Topic Modeling in Learning Tasks

Authors: Jamie Haddock, Lara Kassab, Sixian Li, Alona Kryshchenko, Rachel Grotheer, Elena Sizikova, Chuntian Wang, Thomas Merkh, R. W. M. A. Madushani, Miju Ahn, Deanna Needell, Kathryn Leonard

Abstract: We propose several new models for semi-supervised nonnegative matrix factorization (SSNMF) and provide motivation for SSNMF models as maximum likelihood estimators given specific distributions of uncertainty. We present multiplicative updates training methods for each new model, and demonstrate the application of these models to classification, although they are flexible to other supervised learni… ▽ More We propose several new models for semi-supervised nonnegative matrix factorization (SSNMF) and provide motivation for SSNMF models as maximum likelihood estimators given specific distributions of uncertainty. We present multiplicative updates training methods for each new model, and demonstrate the application of these models to classification, although they are flexible to other supervised learning tasks. We illustrate the promise of these models and training methods on both synthetic and real data, and achieve high classification accuracy on the 20 Newsgroups dataset. △ Less

Submitted 15 October, 2020; originally announced October 2020.

Comments: 4 figures, 12 tables

Quantile-based Iterative Methods for Corrupted Systems of Linear Equations

Authors: Jamie Haddock, Deanna Needell, Elizaveta Rebrova, William Swartworth

Abstract: Often in applications ranging from medical imaging and sensor networks to error correction and data science (and beyond), one needs to solve large-scale linear systems in which a fraction of the measurements have been corrupted. We consider solving such large-scale systems of linear equations $\mathbf{A}\mathbf{x}=\mathbf{b}$ that are inconsistent due to corruptions in the measurement vector… ▽ More Often in applications ranging from medical imaging and sensor networks to error correction and data science (and beyond), one needs to solve large-scale linear systems in which a fraction of the measurements have been corrupted. We consider solving such large-scale systems of linear equations $\mathbf{A}\mathbf{x}=\mathbf{b}$ that are inconsistent due to corruptions in the measurement vector $\mathbf{b}$. We develop several variants of iterative methods that converge to the solution of the uncorrupted system of equations, even in the presence of large corruptions. These methods make use of a quantile of the absolute values of the residual vector in determining the iterate update. We present both theoretical and empirical results that demonstrate the promise of these iterative approaches. △ Less

Submitted 7 July, 2021; v1 submitted 17 September, 2020; originally announced September 2020.

MSC Class: 65F10; 68W20; 60B20

Greed Works: An Improved Analysis of Sampling Kaczmarz-Motzkin

Authors: Jamie Haddock, Anna Ma

Abstract: Stochastic iterative algorithms have gained recent interest in machine learning and signal processing for solving large-scale systems of equations, $Ax=b$. One such example is the Randomized Kaczmarz (RK) algorithm, which acts only on single rows of the matrix $A$ at a time. While RK randomly selects a row of $A$ to work with, Motzkin's Method (MM) employs a greedy row selection. Connections betwe… ▽ More Stochastic iterative algorithms have gained recent interest in machine learning and signal processing for solving large-scale systems of equations, $Ax=b$. One such example is the Randomized Kaczmarz (RK) algorithm, which acts only on single rows of the matrix $A$ at a time. While RK randomly selects a row of $A$ to work with, Motzkin's Method (MM) employs a greedy row selection. Connections between the two algorithms resulted in the Sampling Kaczmarz-Motzkin (SKM) algorithm which samples a random subset of $β$ rows of $A$ and then greedily selects the best row of the subset. Despite their variable computational costs, all three algorithms have been proven to have the same theoretical upper bound on the convergence rate. In this work, an improved analysis of the range of random (RK) to greedy (MM) methods is presented. This analysis improves upon previous known convergence bounds for SKM, capturing the benefit of partially greedy selection schemes. This work also further generalizes previous known results, removing the theoretical assumptions that $β$ must be fixed at every iteration and that $A$ must have normalized rows. △ Less

Submitted 24 July, 2020; v1 submitted 7 December, 2019; originally announced December 2019.

On Inferences from Completed Data

Authors: Jamie Haddock, Denali Molitor, Deanna Needell, Sneha Sambandam, Joy Song, Simon Sun

Abstract: Matrix completion has become an extremely important technique as data scientists are routinely faced with large, incomplete datasets on which they wish to perform statistical inferences. We investigate how error introduced via matrix completion affects statistical inference. Furthermore, we prove recovery error bounds which depend upon the matrix recovery error for several common statistical infer… ▽ More Matrix completion has become an extremely important technique as data scientists are routinely faced with large, incomplete datasets on which they wish to perform statistical inferences. We investigate how error introduced via matrix completion affects statistical inference. Furthermore, we prove recovery error bounds which depend upon the matrix recovery error for several common statistical inferences. We consider matrix recovery via nuclear norm minimization and a variant, $\ell_1$-regularized nuclear norm minimization for data with a structured sampling pattern. Finally, we run a series of numerical experiments on synthetic data and real patient surveys from MyLymeData, which illustrate the relationship between inference recovery error and matrix recovery error. These results indicate that exact matrix recovery is often not necessary to achieve small inference recovery error. △ Less

Submitted 5 July, 2019; originally announced July 2019.

Data-driven Algorithm Selection and Parameter Tuning: Two Case studies in Optimization and Signal Processing

Authors: Jesus A. De Loera, Jamie Haddock, Anna Ma, Deanna Needell

Abstract: Machine learning algorithms typically rely on optimization subroutines and are well-known to provide very effective outcomes for many types of problems. Here, we flip the reliance and ask the reverse question: can machine learning algorithms lead to more effective outcomes for optimization problems? Our goal is to train machine learning methods to automatically improve the performance of optimizat… ▽ More Machine learning algorithms typically rely on optimization subroutines and are well-known to provide very effective outcomes for many types of problems. Here, we flip the reliance and ask the reverse question: can machine learning algorithms lead to more effective outcomes for optimization problems? Our goal is to train machine learning methods to automatically improve the performance of optimization and signal processing algorithms. As a proof of concept, we use our approach to improve two popular data processing subroutines in data science: stochastic gradient descent and greedy methods in compressed sensing. We provide experimental results that demonstrate the answer is ``yes'', machine learning algorithms do lead to more effective outcomes for optimization problems, and show the future potential for this research direction. △ Less

Submitted 26 July, 2019; v1 submitted 30 May, 2019; originally announced May 2019.

Randomized Projection Methods for Linear Systems with Arbitrarily Large Sparse Corruptions

Authors: Jamie Haddock, Deanna Needell

Abstract: In applications like medical imaging, error correction, and sensor networks, one needs to solve large-scale linear systems that may be corrupted by a small number of arbitrarily large corruptions. We consider solving such large-scale systems of linear equations $A\mathbf{x}=\mathbf{b}$ that are inconsistent due to corruptions in the measurement vector $\mathbf{b}$. With this as our motivating exam… ▽ More In applications like medical imaging, error correction, and sensor networks, one needs to solve large-scale linear systems that may be corrupted by a small number of arbitrarily large corruptions. We consider solving such large-scale systems of linear equations $A\mathbf{x}=\mathbf{b}$ that are inconsistent due to corruptions in the measurement vector $\mathbf{b}$. With this as our motivating example, we develop an approach for this setting that allows detection of the corrupted entries and thus convergence to the "true" solution of the original system. We provide analytical justification for our approaches as well as experimental evidence on real and synthetic systems. △ Less

Submitted 22 December, 2018; v1 submitted 21 March, 2018; originally announced March 2018.

MSC Class: 65F10; 65F20; 65F22

On Motzkin's Method for Inconsistent Linear Systems

Authors: Jamie Haddock, Deanna Needell

Abstract: Iterative linear solvers have gained recent popularity due to their computational efficiency and low memory footprint for large-scale linear systems. The relaxation method, or Motzkin's method, can be viewed as an iterative method that projects the current estimation onto the solution hyperplane corresponding to the most violated constraint. Although this leads to an optimal selection strategy for… ▽ More Iterative linear solvers have gained recent popularity due to their computational efficiency and low memory footprint for large-scale linear systems. The relaxation method, or Motzkin's method, can be viewed as an iterative method that projects the current estimation onto the solution hyperplane corresponding to the most violated constraint. Although this leads to an optimal selection strategy for consistent systems, for inconsistent least square problems, the strategy presents a tradeoff between convergence rate and solution accuracy. We provide a theoretical analysis that shows Motzkin's method offers an initially accelerated convergence rate and this acceleration depends on the dynamic range of the residual. We quantify this acceleration for Gaussian systems as a concrete example. Lastly, we include experimental evidence on real and synthetic systems that support the analysis. △ Less

Submitted 26 October, 2018; v1 submitted 8 February, 2018; originally announced February 2018.

MSC Class: 15A06; 65F10; 65F20; 65F22

The Minimum Euclidean-Norm Point on a Convex Polytope: Wolfe's Combinatorial Algorithm is Exponential

Authors: Jesus De Loera, Jamie Haddock, Luis Rademacher

Abstract: The complexity of Philip Wolfe's method for the minimum Euclidean-norm point problem over a convex polytope has remained unknown since he proposed the method in 1974. The method is important because it is used as a subroutine for one of the most practical algorithms for submodular function minimization. We present the first example that Wolfe's method takes exponential time. Additionally, we impro… ▽ More The complexity of Philip Wolfe's method for the minimum Euclidean-norm point problem over a convex polytope has remained unknown since he proposed the method in 1974. The method is important because it is used as a subroutine for one of the most practical algorithms for submodular function minimization. We present the first example that Wolfe's method takes exponential time. Additionally, we improve previous results to show that linear programming reduces in strongly-polynomial time to the minimum norm point problem over a simplex. △ Less

Submitted 3 November, 2017; v1 submitted 6 October, 2017; originally announced October 2017.

MSC Class: 90C20; 90C27; 90C60

A Sampling Kaczmarz-Motzkin Algorithm for Linear Feasibility

Authors: Jesus De Loera, Jamie Haddock, Deanna Needell

Abstract: We combine two iterative algorithms for solving large-scale systems of linear inequalities, the relaxation method of Agmon, Motzkin et al. and the randomized Kaczmarz method. In doing so, we obtain a family of algorithms that generalize and extend both techniques. We prove several convergence results, and our computational experiments show our algorithms often outperform the original methods. We combine two iterative algorithms for solving large-scale systems of linear inequalities, the relaxation method of Agmon, Motzkin et al. and the randomized Kaczmarz method. In doing so, we obtain a family of algorithms that generalize and extend both techniques. We prove several convergence results, and our computational experiments show our algorithms often outperform the original methods. △ Less

Submitted 3 June, 2019; v1 submitted 4 May, 2016; originally announced May 2016.