Computation
See recent articles
- [1] arXiv:2409.00013 [pdf, html, other]
-
Title: CEopt: A MATLAB Package for Non-convex Optimization with the Cross-Entropy MethodSubjects: Computation (stat.CO); Mathematical Software (cs.MS); Optimization and Control (math.OC); Methodology (stat.ME)
This paper introduces CEopt (this https URL), a MATLAB tool leveraging the Cross-Entropy method for non-convex optimization. Due to the relative simplicity of the algorithm, it provides a kind of transparent ``gray-box'' optimization solver, with intuitive control parameters. Unique in its approach, CEopt effectively handles both equality and inequality constraints using an augmented Lagrangian method, offering robustness and scalability for moderately sized complex problems. Through select case studies, the package's applicability and effectiveness in various optimization scenarios are showcased, marking CEopt as a practical addition to optimization research and application toolsets.
- [2] arXiv:2409.00407 [pdf, html, other]
-
Title: Response probability distribution estimation of expensive computer simulators: A Bayesian active learning perspective using Gaussian process regressionSubjects: Computation (stat.CO)
Estimation of the response probability distributions of computer simulators in the presence of randomness is a crucial task in many fields. However, achieving this task with guaranteed accuracy remains an open computational challenge, especially for expensive-to-evaluate computer simulators. In this work, a Bayesian active learning perspective is presented to address the challenge, which is based on the use of the Gaussian process (GP) regression. First, estimation of the response probability distributions is conceptually interpreted as a Bayesian inference problem, as opposed to frequentist inference. This interpretation provides several important benefits: (1) it quantifies and propagates discretization error probabilistically; (2) it incorporates prior knowledge of the computer simulator, and (3) it enables the effective reduction of numerical uncertainty in the solution to a prescribed level. The conceptual Bayesian idea is then realized by using the GP regression, where we derive the posterior statistics of the response probability distributions in semi-analytical form and also provide a numerical solution scheme. Based on the practical Bayesian approach, a Bayesian active learning (BAL) method is further proposed for estimating the response probability distributions. In this context, the key contribution lies in the development of two crucial components for active learning, i.e., stopping criterion and learning function, by taking advantage of posterior statistics. It is empirically demonstrated by five numerical examples that the proposed BAL method can efficiently estimate the response probability distributions with desired accuracy.
- [3] arXiv:2409.01293 [pdf, other]
-
Title: Extracting Signal out of Chaos: Advancements on MAGI for Bayesian Analysis of Dynamical SystemsComments: An honors thesis presented to the Harvard University Departments of Statistics and Mathematics. Advised by Professor Samuel Kou, Department of StatisticsSubjects: Computation (stat.CO); Machine Learning (cs.LG); Dynamical Systems (math.DS); Machine Learning (stat.ML)
This work builds off the manifold-constrained Gaussian process inference (MAGI) method for Bayesian parameter inference and trajectory reconstruction of ODE-based dynamical systems, focusing primarily on sparse and noisy data conditions. First, we introduce Pilot MAGI (pMAGI), a novel methodological upgrade on the base MAGI method that confers significantly-improved numerical stability, parameter inference, and trajectory reconstruction. Second, we demonstrate, for the first time to our knowledge, how one can combine MAGI-based methods with dynamical systems theory to provide probabilistic classifications of whether a system is stable or chaotic. Third, we demonstrate how pMAGI performs favorably in many settings against much more computationally-expensive and overparameterized methods. Fourth, we introduce Pilot MAGI Sequential Prediction (PMSP), a novel method building upon pMAGI that allows one to predict the trajectory of ODE-based dynamical systems multiple time steps into the future, given only sparse and noisy observations. We show that PMSP can output accurate future predictions even on chaotic dynamical systems and significantly outperform PINN-based methods. Overall, we contribute to the literature two novel methods, pMAGI and PMSP, that serve as Bayesian, uncertainty-quantified competitors to the Physics-Informed Neural Network.
- [4] arXiv:2409.01316 [pdf, html, other]
-
Title: Plasmode simulation for the evaluation of causal inference methods in homophilous social networksSubjects: Computation (stat.CO); Applications (stat.AP)
Typical simulation approaches for evaluating the performance of statistical methods on populations embedded in social networks may fail to capture important features of real-world networks. It can therefore be unclear whether inference methods for causal effects due to interference that have been shown to perform well in such synthetic networks are applicable to social networks which arise in the real world. Plasmode simulation studies use a real dataset created from natural processes, but with part of the data-generation mechanism known. However, given the sensitivity of relational data, many network data are protected from unauthorized access or disclosure. In such case, plasmode simulations cannot use released versions of real datasets which often omit the network links, and instead can only rely on parameters estimated from them. A statistical framework for creating replicated simulation datasets from private social network data is developed and validated. The approach consists of simulating from a parametric exponential family random graph model fitted to the network data and resampling from the observed exposure and covariate distributions to preserve the associations among these variables.
- [5] arXiv:2409.01547 [pdf, html, other]
-
Title: The R package psvmSDR: A Unified Algorithm for Sufficient Dimension Reduction via Principal MachinesSubjects: Computation (stat.CO); Machine Learning (stat.ML)
Sufficient dimension reduction (SDR), which seeks a lower-dimensional subspace of the predictors containing regression or classification information has been popular in a machine learning community. In this work, we present a new R software package psvmSDR that implements a new class of SDR estimators, which we call the principal machine (PM) generalized from the principal support vector machine (PSVM). The package covers both linear and nonlinear SDR and provides a function applicable to realtime update scenarios. The package implements the descent algorithm for the PMs to efficiently compute the SDR estimators in various situations. This easy-to-use package will be an attractive alternative to the dr R package that implements classical SDR methods.
- [6] arXiv:2409.01574 [pdf, html, other]
-
Title: Policy Gradients for Optimal Parallel Tempering MCMCComments: 11 pages, 5 figures, accepted to ICML 2024 Workshop on Structured Probabilistic Inference & Generative ModelingSubjects: Computation (stat.CO); Machine Learning (cs.LG); Machine Learning (stat.ML)
Parallel tempering is meta-algorithm for Markov Chain Monte Carlo that uses multiple chains to sample from tempered versions of the target distribution, enhancing mixing in multi-modal distributions that are challenging for traditional methods. The effectiveness of parallel tempering is heavily influenced by the selection of chain temperatures. Here, we present an adaptive temperature selection algorithm that dynamically adjusts temperatures during sampling using a policy gradient approach. Experiments demonstrate that our method can achieve lower integrated autocorrelation times compared to traditional geometrically spaced temperatures and uniform acceptance rate schemes on benchmark distributions.
New submissions for Wednesday, 4 September 2024 (showing 6 of 6 entries )
- [7] arXiv:2409.00470 (cross-list from stat.ME) [pdf, other]
-
Title: Examining the robustness of a model selection procedure in the binary latent block model through a language placement test data setComments: 23 pages, 6 figures, 8 tables, soumissionSubjects: Methodology (stat.ME); Computation (stat.CO)
When entering French university, the students' foreign language level is assessed through a placement test. In this work, we model the placement test results using binary latent block models which allow to simultaneously form homogeneous groups of students and of items. However, a major difficulty in latent block models is to select correctly the number of groups of rows and the number of groups of columns. The first purpose of this paper is to tune the number of initializations needed to limit the initial values problem in the estimation algorithm in order to propose a model selection procedure in the placement test context. Computational studies based on simulated data sets and on two placement test data sets are investigated. The second purpose is to investigate the robustness of the proposed model selection procedure in terms of stability of the students groups when the number of students varies.
- [8] arXiv:2409.00568 (cross-list from cs.MS) [pdf, html, other]
-
Title: Armadillo and Eigen: A Tale of Two Linear Algebra LibrariesComments: 11 pages, 0 figures, 4 tablesSubjects: Mathematical Software (cs.MS); Programming Languages (cs.PL); Computation (stat.CO)
This article introduces `cpp11eigen`, a new R package that integrates the powerful Eigen C++ library for linear algebra into the R programming environment. This article provides a detailed comparison between Armadillo and Eigen speed and syntax. The `cpp11eigen` package simplifies a part of the process of using C++ within R by offering additional ease of integration for those who require high-performance linear algebra operations in their R workflows. This work aims to discuss the tradeoff between computational efficiency and accessibility.
- [9] arXiv:2409.01410 (cross-list from cs.LG) [pdf, html, other]
-
Title: Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful LearningVyacheslav Kungurtsev, Yuanfang Peng, Jianyang Gu, Saeed Vahidian, Anthony Quinn, Fadwa Idlahcen, Yiran ChenSubjects: Machine Learning (cs.LG); Computation (stat.CO)
Dataset distillation (DD) is an increasingly important technique that focuses on constructing a synthetic dataset capable of capturing the core information in training data to achieve comparable performance in models trained on the latter. While DD has a wide range of applications, the theory supporting it is less well evolved. New methods of DD are compared on a common set of benchmarks, rather than oriented towards any particular learning task. In this work, we present a formal model of DD, arguing that a precise characterization of the underlying optimization problem must specify the inference task associated with the application of interest. Without this task-specific focus, the DD problem is under-specified, and the selection of a DD algorithm for a particular task is merely heuristic. Our formalization reveals novel applications of DD across different modeling environments. We analyze existing DD methods through this broader lens, highlighting their strengths and limitations in terms of accuracy and faithfulness to optimal DD operation. Finally, we present numerical results for two case studies important in contemporary settings. Firstly, we address a critical challenge in medical data analysis: merging the knowledge from different datasets composed of intersecting, but not identical, sets of features, in order to construct a larger dataset in what is usually a small sample setting. Secondly, we consider out-of-distribution error across boundary conditions for physics-informed neural networks (PINNs), showing the potential for DD to provide more physically faithful data. By establishing this general formulation of DD, we aim to establish a new research paradigm by which DD can be understood and from which new DD techniques can arise.
- [10] arXiv:2409.02086 (cross-list from q-bio.QM) [pdf, html, other]
-
Title: Taming Randomness in Agent-Based Models using Common Random NumbersComments: 26 pages, including figures and supplement. Thanks!Subjects: Quantitative Methods (q-bio.QM); Populations and Evolution (q-bio.PE); Computation (stat.CO); Methodology (stat.ME)
Random numbers are at the heart of every agent-based model (ABM) of health and disease. By representing each individual in a synthetic population, agent-based models enable detailed analysis of intervention impact and parameter sensitivity. Yet agent-based modeling has a fundamental signal-to-noise problem, in which small differences between simulations cannot be reliably differentiated from stochastic noise resulting from misaligned random number realizations. We introduce a novel methodology that eliminates noise due to misaligned random numbers, a first for agent-based modeling. Our approach enables meaningful individual-level analysis between ABM scenarios because all differences are driven by mechanistic effects rather than random number noise. A key result is that many fewer simulations are needed for some applications. We demonstrate the benefits of our approach on three disparate examples and discuss limitations.
Cross submissions for Wednesday, 4 September 2024 (showing 4 of 4 entries )
- [11] arXiv:2210.04482 (replaced) [pdf, html, other]
-
Title: Leave-group-out cross-validation for latent Gaussian modelsSubjects: Computation (stat.CO); Applications (stat.AP); Methodology (stat.ME)
Evaluating the predictive performance of a statistical model is commonly done using cross-validation. Although the leave-one-out method is frequently employed, its application is justified primarily for independent and identically distributed observations. However, this method tends to mimic interpolation rather than prediction when dealing with dependent observations. This paper proposes a modified cross-validation for dependent observations. This is achieved by excluding an automatically determined set of observations from the training set to mimic a more reasonable prediction scenario. Also, within the framework of latent Gaussian models, we illustrate a method to adjust the joint posterior for this modified cross-validation to avoid model refitting. This new approach is accessible in the R-INLA package (this http URL).
- [12] arXiv:2211.02990 (replaced) [pdf, html, other]
-
Title: Efficient convex PCA with applications to Wasserstein GPCA and ranked dataComments: 43 pages, 10 figures, 4 tables, Code available at: this https URLSubjects: Computation (stat.CO); Statistical Finance (q-fin.ST)
Convex PCA, which was introduced in Bigot et al. (2017), modifies Euclidean PCA by restricting the data and the principal components to lie in a given convex subset of a Hilbert space. This setting arises naturally in many applications, including distributional data in the Wasserstein space of an interval, and ranked compositional data under the Aitchison geometry. Our contribution in this paper is threefold. First, we present several new theoretical results including consistency as well as continuity and differentiability of the objective function in the finite dimensional case. Second, we develop a numerical implementation of finite dimensional convex PCA when the convex set is polyhedral, and show that this provides a natural approximation of Wasserstein GPCA. Third, we illustrate our results with two financial applications, namely distributions of stock returns ranked by size and the capital distribution curve, both of which are of independent interest in stochastic portfolio theory. Supplementary materials for this article are available online.
- [13] arXiv:2312.13021 (replaced) [pdf, html, other]
-
Title: Taming numerical imprecision by adapting the KL divergence to negative probabilitiesComments: 11 pages, 6 figures, minor clarifications, published versionJournal-ref: Statistics and Computing 34 (2024) 168Subjects: Computation (stat.CO)
The Kullback-Leibler (KL) divergence is frequently used in data science. For discrete distributions on large state spaces, approximations of probability vectors may result in a few small negative entries, rendering the KL divergence undefined. We address this problem by introducing a parameterized family of substitute divergence measures, the shifted KL (sKL) divergence measures. Our approach is generic and does not increase the computational overhead. We show that the sKL divergence shares important theoretical properties with the KL divergence and discuss how its shift parameters should be chosen. If Gaussian noise is added to a probability vector, we prove that the average sKL divergence converges to the KL divergence for small enough noise. We also show that our method solves the problem of negative entries in an application from computational oncology, the optimization of Mutual Hazard Networks for cancer progression using tensor-train approximations.
- [14] arXiv:2404.16528 (replaced) [pdf, html, other]
-
Title: Generalized Posterior Calibration via Sequential Monte Carlo SamplerComments: Accepted for publication in Proceedings of the 2024 6th Asia Conference on Machine Learning and ComputingSubjects: Computation (stat.CO)
As the amount and complexity of available data increases, the need for robust statistical learning becomes more pressing. To enhance resilience against model misspecification, the generalized posterior inference method adjusts the likelihood term by exponentiating it with a learning rate, thereby fine-tuning the dispersion of the posterior distribution. This study proposes a computationally efficient strategy for selecting an appropriate learning rate. The proposed approach builds upon the generalized posterior calibration (GPC) algorithm, which is designed to select a learning rate that ensures nominal frequentist coverage. This algorithm, which evaluates the coverage probability using bootstrap samples, has high computational costs because of the repeated posterior simulations needed for bootstrap samples. To address this limitation, the study proposes an algorithm that combines elements of the GPC algorithm with the sequential Monte Carlo (SMC) sampler. By leveraging the similarity between the learning rate in generalized posterior inference and the inverse temperature in SMC sampling, the proposed algorithm efficiently calibrates the posterior distribution with a reduced computational cost. For demonstration, the proposed algorithm was applied to several statistical learning models and shown to be significantly faster than the original GPC.
- [15] arXiv:2408.09618 (replaced) [pdf, html, other]
-
Title: kendallknight: An R Package for Efficient Implementation of Kendall's Correlation Coefficient ComputationComments: 9 pages with references, 2 tables, 2 figuresSubjects: Computation (stat.CO); Data Structures and Algorithms (cs.DS); Econometrics (econ.EM)
The kendallknight package introduces an efficient implementation of Kendall's correlation coefficient computation, significantly improving the processing time for large datasets without sacrificing accuracy. The kendallknight package, following Knight (1966) and posterior literature, reduces the computational complexity resulting in drastic reductions in computation time, transforming operations that would take minutes or hours into milliseconds or minutes, while maintaining precision and correctly handling edge cases and errors. The package is particularly advantageous in econometric and statistical contexts where rapid and accurate calculation of Kendall's correlation coefficient is desirable. Benchmarks demonstrate substantial performance gains over the base R implementation, especially for large datasets.
- [16] arXiv:2408.17102 (replaced) [pdf, html, other]
-
Title: Stochastic Vector Approximate Message Passing with applications to phase retrievalSubjects: Computation (stat.CO)
Phase retrieval refers to the problem of recovering a high-dimensional vector $\boldsymbol{x} \in \mathbb{C}^N$ from the magnitude of its linear transform $\boldsymbol{z} = A \boldsymbol{x}$, observed through a noisy channel. To improve the ill-posed nature of the inverse problem, it is a common practice to observe the magnitude of linear measurements $\boldsymbol{z}^{(1)} = A^{(1)} \boldsymbol{x},..., \boldsymbol{z}^{(L)} = A^{(L)}\boldsymbol{x}$ using multiple sensing matrices $A^{(1)},..., A^{(L)}$, with ptychographic imaging being a remarkable example of such strategies. Inspired by existing algorithms for ptychographic reconstruction, we introduce stochasticity to Vector Approximate Message Passing (VAMP), a computationally efficient algorithm applicable to a wide range of Bayesian inverse problems. By testing our approach in the setup of phase retrieval, we show the superior convergence speed of the proposed algorithm.
- [17] arXiv:2308.12470 (replaced) [pdf, html, other]
-
Title: Scalable Estimation of Multinomial Response Models with Random Consideration SetsSubjects: Methodology (stat.ME); Econometrics (econ.EM); Applications (stat.AP); Computation (stat.CO)
A common assumption in the fitting of unordered multinomial response models for $J$ mutually exclusive categories is that the responses arise from the same set of $J$ categories across subjects. However, when responses measure a choice made by the subject, it is more appropriate to condition the distribution of multinomial responses on a subject-specific consideration set, drawn from the power set of $\{1,2,\ldots,J\}$. This leads to a mixture of multinomial response models governed by a probability distribution over the $J^{\ast} = 2^J -1$ consideration sets. We introduce a novel method for estimating such generalized multinomial response models based on the fundamental result that any mass distribution over $J^{\ast}$ consideration sets can be represented as a mixture of products of $J$ component-specific inclusion-exclusion probabilities. Moreover, under time-invariant consideration sets, the conditional posterior distribution of consideration sets is sparse. These features enable a scalable MCMC algorithm for sampling the posterior distribution of parameters, random effects, and consideration sets. Under regularity conditions, the posterior distributions of the marginal response probabilities and the model parameters satisfy consistency. The methodology is demonstrated in a longitudinal data set on weekly cereal purchases that cover $J = 101$ brands, a dimension substantially beyond the reach of existing methods.
- [18] arXiv:2408.11074 (replaced) [pdf, html, other]
-
Title: cpp11armadillo: An R Package to Use the Armadillo C++ LibraryComments: 23 pages, 0 figuresSubjects: Mathematical Software (cs.MS); Programming Languages (cs.PL); Computation (stat.CO)
This article introduces 'cpp11armadillo', a new R package that integrates the powerful Armadillo C++ library for linear algebra into the R programming environment. Targeted primarily at social scientists and other non-programmers, this article explains the computational benefits of moving code to C++ in terms of speed and syntax. We provide a comprehensive overview of Armadillo's capabilities, highlighting its user-friendly syntax akin to MATLAB and its efficiency for computationally intensive tasks. The 'cpp11armadillo' package simplifies a part of the process of using C++ within R by offering additional ease of integration for those who require high-performance linear algebra operations in their R workflows. This work aims to bridge the gap between computational efficiency and accessibility, making advanced linear algebra operations more approachable for R users without extensive programming backgrounds.