Showing 1–42 of 42 results for author: Stern, M

Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function

Authors: Matias Oscar Volman Stern, Dominic Hohs, Andreas Jansche, Timo Bernthaler, Gerhard Schneider

Abstract: Training of semantic segmentation models for material analysis requires micrographs and their corresponding masks. It is quite unlikely that perfect masks will be drawn, especially at the edges of objects, and sometimes the amount of data that can be obtained is small, since only a few samples are available. These aspects make it very problematic to train a robust model. We demonstrate a workflow… ▽ More Training of semantic segmentation models for material analysis requires micrographs and their corresponding masks. It is quite unlikely that perfect masks will be drawn, especially at the edges of objects, and sometimes the amount of data that can be obtained is small, since only a few samples are available. These aspects make it very problematic to train a robust model. We demonstrate a workflow for the improvement of semantic segmentation models of micrographs through the generation of synthetic microstructural images in conjunction with masks. The workflow only requires joining a few micrographs with their respective masks to create the input for a Vector Quantised-Variational AutoEncoder model that includes an embedding space, which is trained such that a generative model (PixelCNN) learns the distribution of each input, transformed into discrete codes, and can be used to sample new codes. The latter will eventually be decoded by VQ-VAE to generate images alongside corresponding masks for semantic segmentation. To evaluate the synthetic data, we have trained U-Net models with different amounts of these synthetic data in conjunction with real data. These models were then evaluated using non-synthetic images only. Additionally, we introduce a customized metric derived from the mean Intersection over Union (mIoU). The proposed metric prevents a few falsely predicted pixels from greatly reducing the value of the mIoU. We have achieved a reduction in sample preparation and acquisition times, as well as the efforts, needed for image processing and labeling tasks, are less when it comes to training semantic segmentation model. The approach could be generalized to various types of image data such that it serves as a user-friendly solution for training models with a small number of real images. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Contextual modulation of language comprehension in a dynamic neural model of lexical meaning

Authors: Michael C. Stern, Maria M. Piñango

Abstract: We propose and computationally implement a dynamic neural model of lexical meaning, and experimentally test its behavioral predictions. We demonstrate the architecture and behavior of the model using as a test case the English lexical item 'have', focusing on its polysemous use. In the model, 'have' maps to a semantic space defined by two continuous conceptual dimensions, connectedness and control… ▽ More We propose and computationally implement a dynamic neural model of lexical meaning, and experimentally test its behavioral predictions. We demonstrate the architecture and behavior of the model using as a test case the English lexical item 'have', focusing on its polysemous use. In the model, 'have' maps to a semantic space defined by two continuous conceptual dimensions, connectedness and control asymmetry, previously proposed to parameterize the conceptual system for language. The mapping is modeled as coupling between a neural node representing the lexical item and neural fields representing the conceptual dimensions. While lexical knowledge is modeled as a stable coupling pattern, real-time lexical meaning retrieval is modeled as the motion of neural activation patterns between metastable states corresponding to semantic interpretations or readings. Model simulations capture two previously reported empirical observations: (1) contextual modulation of lexical semantic interpretation, and (2) individual variation in the magnitude of this modulation. Simulations also generate a novel prediction that the by-trial relationship between sentence reading time and acceptability should be contextually modulated. An experiment combining self-paced reading and acceptability judgments replicates previous results and confirms the new model prediction. Altogether, results support a novel perspective on lexical polysemy: that the many related meanings of a word are metastable neural activation states that arise from the nonlinear dynamics of neural populations governing interpretation on continuous semantic dimensions. △ Less

Submitted 19 July, 2024; originally announced July 2024.

Investigating the impact of virtual element misalignment in collaborative Augmented Reality experiences

Authors: Francesco Vona, Sina Hinzmann, Michael Stern, Tanja Kojić, Navid Ashrafi, David Grieshammer, Jan-Niklas Voigt-Antons

Abstract: The collaboration in co-located shared environments has sparked an increased interest in immersive technologies, including Augmented Reality (AR). Since research in this field has primarily focused on individual user experiences in AR, the collaborative aspects within shared AR spaces remain less explored, and fewer studies can provide guidelines for designing this type of experience. This article… ▽ More The collaboration in co-located shared environments has sparked an increased interest in immersive technologies, including Augmented Reality (AR). Since research in this field has primarily focused on individual user experiences in AR, the collaborative aspects within shared AR spaces remain less explored, and fewer studies can provide guidelines for designing this type of experience. This article investigates how the user experience in a collaborative shared AR space is affected by divergent perceptions of virtual objects and the effects of positional synchrony and avatars. For this purpose, we developed an AR app and used two distinct experimental conditions to study the influencing factors. Forty-eight participants, organized into 24 pairs, participated in the experiment and jointly interacted with shared virtual objects. Results indicate that divergent perceptions of virtual objects did not directly influence communication and collaboration dynamics. Conversely, positional synchrony emerged as a critical factor, significantly enhancing the quality of the collaborative experience. On the contrary, while not negligible, avatars played a relatively less pronounced role in influencing these dynamics. The findings can potentially offer valuable practical insights, guiding the development of future collaborative AR/VR environments. △ Less

Submitted 25 April, 2024; v1 submitted 14 April, 2024; originally announced April 2024.

Comments: Paper accepted for publication/presentation at QoMEX 2024

Modeling Analog Dynamic Range Compressors using Deep Learning and State-space Models

Authors: Hanzhi Yin, Gang Cheng, Christian J. Steinmetz, Ruibin Yuan, Richard M. Stern, Roger B. Dannenberg

Abstract: We describe a novel approach for developing realistic digital models of dynamic range compressors for digital audio production by analyzing their analog prototypes. While realistic digital dynamic compressors are potentially useful for many applications, the design process is challenging because the compressors operate nonlinearly over long time scales. Our approach is based on the structured stat… ▽ More We describe a novel approach for developing realistic digital models of dynamic range compressors for digital audio production by analyzing their analog prototypes. While realistic digital dynamic compressors are potentially useful for many applications, the design process is challenging because the compressors operate nonlinearly over long time scales. Our approach is based on the structured state space sequence model (S4), as implementing the state-space model (SSM) has proven to be efficient at learning long-range dependencies and is promising for modeling dynamic range compressors. We present in this paper a deep learning model with S4 layers to model the Teletronix LA-2A analog dynamic range compressor. The model is causal, executes efficiently in real time, and achieves roughly the same quality as previous deep-learning models but with fewer parameters. △ Less

Submitted 24 March, 2024; originally announced March 2024.

Applying statistical learning theory to deep learning

Authors: Cédric Gerbelot, Avetik Karagulyan, Stefani Karp, Kavya Ravichandran, Menachem Stern, Nathan Srebro

Abstract: Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear, in particular how different architectures may lead to inductive bias when trained using gradient based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep l… ▽ More Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear, in particular how different architectures may lead to inductive bias when trained using gradient based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias, in particular transitioning between kernel or feature learning. △ Less

Submitted 25 March, 2024; v1 submitted 26 November, 2023; originally announced November 2023.

Comments: 66 pages, 20 figures

Machine Learning Without a Processor: Emergent Learning in a Nonlinear Electronic Metamaterial

Authors: Sam Dillavou, Benjamin D Beyer, Menachem Stern, Andrea J Liu, Marc Z Miskin, Douglas J Durian

Abstract: Standard deep learning algorithms require differentiating large nonlinear networks, a process that is slow and power-hungry. Electronic learning metamaterials offer potentially fast, efficient, and fault-tolerant hardware for analog machine learning, but existing implementations are linear, severely limiting their capabilities. These systems differ significantly from artificial neural networks as… ▽ More Standard deep learning algorithms require differentiating large nonlinear networks, a process that is slow and power-hungry. Electronic learning metamaterials offer potentially fast, efficient, and fault-tolerant hardware for analog machine learning, but existing implementations are linear, severely limiting their capabilities. These systems differ significantly from artificial neural networks as well as the brain, so the feasibility and utility of incorporating nonlinear elements have not been explored. Here we introduce a nonlinear learning metamaterial -- an analog electronic network made of self-adjusting nonlinear resistive elements based on transistors. We demonstrate that the system learns tasks unachievable in linear systems, including XOR and nonlinear regression, without a computer. We find our nonlinear learning metamaterial reduces modes of training error in order (mean, slope, curvature), similar to spectral bias in artificial neural networks. The circuitry is robust to damage, retrainable in seconds, and performs learned tasks in microseconds while dissipating only picojoules of energy across each transistor. This suggests enormous potential for fast, low-power computing in edge systems like sensors, robotic controllers, and medical devices, as well as manufacturability at scale for performing and studying emergent learning. △ Less

Submitted 5 April, 2024; v1 submitted 1 November, 2023; originally announced November 2023.

Comments: 11 pages 8 figures

Journal ref: Proc. Nat. Acad. Sci. 121 (28), e2319718121 (2024)

Online Active Learning For Sound Event Detection

Authors: Mark Lindsey, Ankit Shah, Francis Kubala, Richard M. Stern

Abstract: Data collection and annotation is a laborious, time-consuming prerequisite for supervised machine learning tasks. Online Active Learning (OAL) is a paradigm that addresses this issue by simultaneously minimizing the amount of annotation required to train a classifier and adapting to changes in the data over the duration of the data collection process. Prior work has indicated that fluctuating clas… ▽ More Data collection and annotation is a laborious, time-consuming prerequisite for supervised machine learning tasks. Online Active Learning (OAL) is a paradigm that addresses this issue by simultaneously minimizing the amount of annotation required to train a classifier and adapting to changes in the data over the duration of the data collection process. Prior work has indicated that fluctuating class distributions and data drift are still common problems for OAL. This work presents new loss functions that address these challenges when OAL is applied to Sound Event Detection (SED). Experimental results from the SONYC dataset and two Voice-Type Discrimination (VTD) corpora indicate that OAL can reduce the time and effort required to train SED classifiers by a factor of 5 for SONYC, and that the new methods presented here successfully resolve issues present in existing OAL methods. △ Less

Submitted 25 September, 2023; originally announced September 2023.

Comments: Submitted to ICASSP 2024. Publication will belong to IEEE

Learnable Front Ends Based on Temporal Modulation for Music Tagging

Authors: Yinghao Ma, Richard M. Stern

Abstract: While end-to-end systems are becoming popular in auditory signal processing including automatic music tagging, models using raw audio as input needs a large amount of data and computational resources without domain knowledge. Inspired by the fact that temporal modulation is regarded as an essential component in auditory perception, we introduce the Temporal Modulation Neural Network (TMNN) that co… ▽ More While end-to-end systems are becoming popular in auditory signal processing including automatic music tagging, models using raw audio as input needs a large amount of data and computational resources without domain knowledge. Inspired by the fact that temporal modulation is regarded as an essential component in auditory perception, we introduce the Temporal Modulation Neural Network (TMNN) that combines Mel-like data-driven front ends and temporal modulation filters with a simple ResNet back end. The structure includes a set of temporal modulation filters to capture long-term patterns in all frequency channels. Experimental results show that the proposed front ends surpass state-of-the-art (SOTA) methods on the MagnaTagATune dataset in automatic music tagging, and they are also helpful for keyword spotting on speech commands. Moreover, the model performance for each tag suggests that genre or instrument tags with complex rhythm and mood tags can especially be improved with temporal modulation. △ Less

Submitted 28 November, 2022; originally announced November 2022.

Comments: Submitted to ICASSP 2023

Neural inhibition during speech planning contributes to contrastive hyperarticulation

Authors: Michael C. Stern, Jason A. Shaw

Abstract: Previous work has demonstrated that words are hyperarticulated on dimensions of speech that differentiate them from a minimal pair competitor. This phenomenon has been termed contrastive hyperarticulation (CH). We present a dynamic neural field (DNF) model of voice onset time (VOT) planning that derives CH from an inhibitory influence of the minimal pair competitor during planning. We test some pr… ▽ More Previous work has demonstrated that words are hyperarticulated on dimensions of speech that differentiate them from a minimal pair competitor. This phenomenon has been termed contrastive hyperarticulation (CH). We present a dynamic neural field (DNF) model of voice onset time (VOT) planning that derives CH from an inhibitory influence of the minimal pair competitor during planning. We test some predictions of the model with a novel experiment investigating CH of voiceless stop consonant VOT in pseudowords. The results demonstrate a CH effect in pseudowords, consistent with a basis for the effect in the real-time planning and production of speech. The scope and magnitude of CH in pseudowords was reduced compared to CH in real words, consistent with a role for interactive activation between lexical and phonological levels of planning. We discuss the potential of our model to unify an apparently disparate set of phenomena, from CH to phonological neighborhood effects to phonetic trace effects in speech errors. △ Less

Submitted 14 March, 2023; v1 submitted 25 September, 2022; originally announced September 2022.

Quantitative probing: Validating causal models using quantitative domain knowledge

Authors: Daniel Grünbaum, Maike L. Stern, Elmar W. Lang

Abstract: We present quantitative probing as a model-agnostic framework for validating causal models in the presence of quantitative domain knowledge. The method is constructed as an analogue of the train/test split in correlation-based machine learning and as an enhancement of current causal validation strategies that are consistent with the logic of scientific discovery. The effectiveness of the method is… ▽ More We present quantitative probing as a model-agnostic framework for validating causal models in the presence of quantitative domain knowledge. The method is constructed as an analogue of the train/test split in correlation-based machine learning and as an enhancement of current causal validation strategies that are consistent with the logic of scientific discovery. The effectiveness of the method is illustrated using Pearl's sprinkler example, before a thorough simulation-based investigation is conducted. Limits of the technique are identified by studying exemplary failing scenarios, which are furthermore used to propose a list of topics for future research and improvements of the presented version of quantitative probing. The code for integrating quantitative probing into causal analysis, as well as the code for the presented simulation-based studies of the effectiveness of quantitative probing is provided in two separate open-source Python packages. △ Less

Submitted 7 September, 2022; originally announced September 2022.

Comments: submitted to the Journal of Causal Inference

MSC Class: 62D20

Journal ref: Journal of Causal Inference, vol. 11, no. 1, 2023

Desynchronous Learning in a Physics-Driven Learning Network

Authors: Jacob F Wycoff, Sam Dillavou, Menachem Stern, Andrea J Liu, Douglas J Durian

Abstract: In a neuron network, synapses update individually using local information, allowing for entirely decentralized learning. In contrast, elements in an artificial neural network (ANN) are typically updated simultaneously using a central processor. Here we investigate the feasibility and effect of desynchronous learning in a recently introduced decentralized, physics-driven learning network. We show t… ▽ More In a neuron network, synapses update individually using local information, allowing for entirely decentralized learning. In contrast, elements in an artificial neural network (ANN) are typically updated simultaneously using a central processor. Here we investigate the feasibility and effect of desynchronous learning in a recently introduced decentralized, physics-driven learning network. We show that desynchronizing the learning process does not degrade performance for a variety of tasks in an idealized simulation. In experiment, desynchronization actually improves performance by allowing the system to better explore the discretized state space of solutions. We draw an analogy between desynchronization and mini-batching in stochastic gradient descent, and show that they have similar effects on the learning process. Desynchronizing the learning process establishes physics-driven learning networks as truly fully distributed learning machines, promoting better performance and scalability in deployment. △ Less

Submitted 1 December, 2022; v1 submitted 10 January, 2022; originally announced January 2022.

Comments: 6 pages 4 figures

Playing Ping Pong with Light: Directional Emission of White Light

Authors: Heribert Wankerl, Christopher Wiesmann, Laura Kreiner, Rainer Butendeich, Alexander Luce, Sandra Sobczyk, Maike Lorena Stern, Elmar Wolfgang Lang

Abstract: Over the last decades, light-emitting diodes (LED) have replaced common light bulbs in almost every application, from flashlights in smartphones to automotive headlights. Illuminating nightly streets requires LEDs to emit a light spectrum that is perceived as pure white by the human eye. The power associated with such a white light spectrum is not only distributed over the contributing wavelengths… ▽ More Over the last decades, light-emitting diodes (LED) have replaced common light bulbs in almost every application, from flashlights in smartphones to automotive headlights. Illuminating nightly streets requires LEDs to emit a light spectrum that is perceived as pure white by the human eye. The power associated with such a white light spectrum is not only distributed over the contributing wavelengths but also over the angles of vision. For many applications, the usable light rays are required to exit the LED in forward direction, namely under small angles to the perpendicular. In this work, we demonstrate that a specifically designed multi-layer thin film on top of a white LED increases the power of pure white light emitted in forward direction. Therefore, the deduced multi-objective optimization problem is reformulated via a real-valued physics-guided objective function that represents the hierarchical structure of our engineering problem. Variants of Bayesian optimization are employed to maximize this non-deterministic objective function based on ray tracing simulations. Eventually, the investigation of optical properties of suitable multi-layer thin films allowed to identify the mechanism behind the increased directionality of white light: angle and wavelength selective filtering causes the multi-layer thin film to play ping pong with rays of light. △ Less

Submitted 30 November, 2021; originally announced November 2021.

Comments: Under review for publication

Tipping the Scales: A Corpus-Based Reconstruction of Adjective Scales in the McGill Pain Questionnaire

Authors: Miriam Stern

Abstract: Modern medical diagnosis relies on precise pain assessment tools in translating clinical information from patient to physician. The McGill Pain Questionnaire (MPQ) is a clinical pain assessment technique that utilizes 78 adjectives of different intensities in 20 different categories to quantity a patient's pain. The questionnaire's efficacy depends on a predictable pattern of adjective use by pati… ▽ More Modern medical diagnosis relies on precise pain assessment tools in translating clinical information from patient to physician. The McGill Pain Questionnaire (MPQ) is a clinical pain assessment technique that utilizes 78 adjectives of different intensities in 20 different categories to quantity a patient's pain. The questionnaire's efficacy depends on a predictable pattern of adjective use by patients experiencing pain. In this study, I recreate the MPQ's adjective intensity orderings using data gathered from patient forums and modern NLP techniques. I extract adjective intensity relationships by searching for key linguistic contexts, and then combine the relationship information to form robust adjective scales. Of 17 adjective relationships predicted by this research, only 4 diverge from the MPQ's orderings, which is statistically significant at the 0.1 alpha level. The results suggest predictable patterns of adjective use by people experiencing pain, but call into question the MPQ's categories for grouping adjectives. △ Less

Submitted 5 October, 2021; v1 submitted 29 September, 2021; originally announced September 2021.

Comments: 16 pages, 1 figure

Dynamic Posted-Price Mechanisms for the Blockchain Transaction Fee Market

Authors: Matheus V. X. Ferreira, Daniel J. Moroz, David C. Parkes, Mitchell Stern

Abstract: In recent years, prominent blockchain systems such as Bitcoin and Ethereum have experienced explosive growth in transaction volume, leading to frequent surges in demand for limited block space and causing transaction fees to fluctuate by orders of magnitude. Existing systems sell space using first-price auctions; however, users find it difficult to estimate how much they need to bid in order to ge… ▽ More In recent years, prominent blockchain systems such as Bitcoin and Ethereum have experienced explosive growth in transaction volume, leading to frequent surges in demand for limited block space and causing transaction fees to fluctuate by orders of magnitude. Existing systems sell space using first-price auctions; however, users find it difficult to estimate how much they need to bid in order to get their transactions accepted onto the chain. If they bid too low, their transactions can have long confirmation times. If they bid too high, they pay larger fees than necessary. In light of these issues, new transaction fee mechanisms have been proposed, most notably EIP-1559, aiming to provide better usability. EIP-1559 is a history-dependent mechanism that relies on block utilization to adjust a base fee. We propose an alternative design -- a {\em dynamic posted-price mechanism} -- which uses not only block utilization but also observable bids from past blocks to compute a posted price for subsequent blocks. We show its potential to reduce price volatility by providing examples for which the prices of EIP-1559 are unstable while the prices of the proposed mechanism are stable. More generally, whenever the demand for the blockchain stabilizes, we ask if our mechanism is able to converge to a stable state. Our main result provides sufficient conditions in a probabilistic setting for which the proposed mechanism is approximately welfare optimal and the prices are stable. Our main technical contribution towards establishing stability is an iterative algorithm that, given oracle access to a Lipschitz continuous and strictly concave function $f$, converges to a fixed point of $f$. △ Less

Submitted 16 November, 2021; v1 submitted 25 March, 2021; originally announced March 2021.

Journal ref: AFT '21: Proceedings of the 3rd ACM Conference on Advances in Financial Technologies, 2021, 86-99

Towards End-to-End In-Image Neural Machine Translation

Authors: Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Firat, Jakob Uszkoreit, Puneet Jain

Abstract: In this paper, we offer a preliminary investigation into the task of in-image machine translation: transforming an image containing text in one language into an image containing the same text in another language. We propose an end-to-end neural model for this task inspired by recent approaches to neural machine translation, and demonstrate promising initial results based purely on pixel-level supe… ▽ More In this paper, we offer a preliminary investigation into the task of in-image machine translation: transforming an image containing text in one language into an image containing the same text in another language. We propose an end-to-end neural model for this task inspired by recent approaches to neural machine translation, and demonstrate promising initial results based purely on pixel-level supervision. We then offer a quantitative and qualitative evaluation of our system outputs and discuss some common failure modes. Finally, we conclude with directions for future work. △ Less

Submitted 20 October, 2020; originally announced October 2020.

Comments: Accepted as an oral presentation at EMNLP, NLP Beyond Text workshop, 2020

Parameterized Reinforcement Learning for Optical System Optimization

Authors: Heribert Wankerl, Maike L. Stern, Ali Mahdavi, Christoph Eichler, Elmar W. Lang

Abstract: Designing a multi-layer optical system with designated optical characteristics is an inverse design problem in which the resulting design is determined by several discrete and continuous parameters. In particular, we consider three design parameters to describe a multi-layer stack: Each layer's dielectric material and thickness as well as the total number of layers. Such a combination of both, dis… ▽ More Designing a multi-layer optical system with designated optical characteristics is an inverse design problem in which the resulting design is determined by several discrete and continuous parameters. In particular, we consider three design parameters to describe a multi-layer stack: Each layer's dielectric material and thickness as well as the total number of layers. Such a combination of both, discrete and continuous parameters is a challenging optimization problem that often requires a computationally expensive search for an optimal system design. Hence, most methods merely determine the optimal thicknesses of the system's layers. To incorporate layer material and the total number of layers as well, we propose a method that considers the stacking of consecutive layers as parameterized actions in a Markov decision process. We propose an exponentially transformed reward signal that eases policy optimization and adapt a recent variant of Q-learning for inverse design optimization. We demonstrate that our method outperforms human experts and a naive reinforcement learning algorithm concerning the achieved optical characteristics. Moreover, the learned Q-values contain information about the optical properties of multi-layer optical systems, thereby allowing physical interpretation or what-if analysis. △ Less

Submitted 25 November, 2020; v1 submitted 9 October, 2020; originally announced October 2020.

Comments: Presented as a poster at the workshop on machine learning for engineering modeling, simulation and design @ NeurIPS 2020

Journal ref: J. Phys. D: Appl. Phys. 54 305104 (2021)

Non causal deep learning based dereverberation

Authors: Jorge Wuth, Richard M. Stern, Nestor Becerra Yoma

Abstract: In this paper we demonstrate the effectiveness of non-causal context for mitigating the effects of reverberation in deep-learning-based automatic speech recognition (ASR) systems. First, the value of non-causal context using a non-causal FIR filter is shown by comparing the contributions of previous vs. future information. Second, MLP- and LSTM-based dereverberation networks were trained to confir… ▽ More In this paper we demonstrate the effectiveness of non-causal context for mitigating the effects of reverberation in deep-learning-based automatic speech recognition (ASR) systems. First, the value of non-causal context using a non-causal FIR filter is shown by comparing the contributions of previous vs. future information. Second, MLP- and LSTM-based dereverberation networks were trained to confirm the effects of causal and non-causal context when used in ASR systems trained with clean speech. The non-causal deep-learning-based dereverberation provides a 45% relative reduction in word error rate (WER) compared to the popular weighted prediction error (WPE) method in experiments with clean training in the REVERB challenge. Finally, an expanded multicondition training procedure used in combination with a semi-enhanced test utterance generation based on combinations of reverberated and dereverberated signals is proposed to reduce any artifacts or distortion that may be introduced by the non-causal dereverberation methods. The combination of both approaches provided average relative reductions in WER equal to 10.9% and 6.0% when compared to the baseline system obtained with the most recent REVERB challenge recipe without and with WPE, respectively. △ Less

Submitted 6 September, 2020; originally announced September 2020.

Comments: 33 pages

A Fair, Traceable, Auditable and Participatory Randomization Tool for Legal Systems

Authors: Marcos Vinicius M. Silva, Marcos Antonio Simplicio Jr., Roberto Augusto Castellanos Pfeiffer, Julio Michael Stern

Abstract: Many real-world scenarios require the random selection of one or more individuals from a pool of eligible candidates. One example of especial social relevance refers to the legal system, in which the jurors and judges are commonly picked according to some probability distribution aiming to avoid biased decisions. In this scenario, ensuring auditability of the random drawing procedure is imperative… ▽ More Many real-world scenarios require the random selection of one or more individuals from a pool of eligible candidates. One example of especial social relevance refers to the legal system, in which the jurors and judges are commonly picked according to some probability distribution aiming to avoid biased decisions. In this scenario, ensuring auditability of the random drawing procedure is imperative to promote confidence in its fairness. With this goal in mind, this article describes a protocol for random drawings specially designed for use in legal systems. The proposed design combines the following properties: security by design, ensuring the fairness of the random draw as long as at least one participant behaves honestly; auditability by any interested party, even those having no technical background, using only public information; and statistical robustness, supporting drawings where candidates may have distinct probability distributions. Moreover, it is capable of inviting and engaging as participating stakeholders the main interested parties of a legal process, in a way that promotes process transparency, public trust and institutional resilience. An open-source implementation is also provided as supplementary material. △ Less

Submitted 4 June, 2020; originally announced June 2020.

MSC Class: 64-04 (Primary); 62D99 (Secondary)

Otimizacao e Processos Estocasticos Aplicados a Economia e Financas

Authors: Julio Michael Stern, Carlos Alberto de Braganca Pereira, Celma de Oliveira Ribeiro, Cibele Dunder, Fabio Nakano, Marcelo Lauretto

Abstract: Optimization and Stochastic Processes Applied to Economy and Finance -- is the name of this book translated to English; It has been used at the IME-USP - The Institute of Mathematics and Statistics of the University of Sao Paulo, since 1993. Contents: Ch.1: Linear Programming; Ch.2: Non-Linear Programming; Ch.3: Quadratic Programming; Ch.4: Markowitz Model; Ch.5: Dynamic Programming; Ch.6: LQG E… ▽ More Optimization and Stochastic Processes Applied to Economy and Finance -- is the name of this book translated to English; It has been used at the IME-USP - The Institute of Mathematics and Statistics of the University of Sao Paulo, since 1993. Contents: Ch.1: Linear Programming; Ch.2: Non-Linear Programming; Ch.3: Quadratic Programming; Ch.4: Markowitz Model; Ch.5: Dynamic Programming; Ch.6: LQG Estimation and Control; Ch.7: Decision Trees; Ch.8: Pension Funds; Ch.9: Mixed Portfolios Including Derivative Contracts; Appendices: App.A: Matlab; App.B: Critical-Point Software; App.C: Computational Linear Algebra; App.D: Probability; App.E: Computer Codes. This book is written in Portuguese language. △ Less

Submitted 25 May, 2020; originally announced May 2020.

Comments: in Portuguese

Semantic Scaffolds for Pseudocode-to-Code Generation

Authors: Ruiqi Zhong, Mitchell Stern, Dan Klein

Abstract: We propose a method for program generation based on semantic scaffolds, lightweight structures representing the high-level semantic and syntactic composition of a program. By first searching over plausible scaffolds then using these as constraints for a beam search over programs, we achieve better coverage of the search space when compared with existing techniques. We apply our hierarchical search… ▽ More We propose a method for program generation based on semantic scaffolds, lightweight structures representing the high-level semantic and syntactic composition of a program. By first searching over plausible scaffolds then using these as constraints for a beam search over programs, we achieve better coverage of the search space when compared with existing techniques. We apply our hierarchical search method to the SPoC dataset for pseudocode-to-code generation, in which we are given line-level natural language pseudocode annotations and aim to produce a program satisfying execution-based test cases. By using semantic scaffolds during inference, we achieve a 10% absolute improvement in top-100 accuracy over the previous state-of-the-art. Additionally, we require only 11 candidates to reach the top-3000 performance of the previous best approach when tested against unseen problems, demonstrating a substantial improvement in efficiency. △ Less

Submitted 12 May, 2020; originally announced May 2020.

Imitation Attacks and Defenses for Black-box Machine Translation Systems

Authors: Eric Wallace, Mitchell Stern, Dawn Song

Abstract: Adversaries may look to steal or attack black-box NLP systems, either for financial gain or to exploit model errors. One setting of particular interest is machine translation (MT), where models have high commercial value and errors can be costly. We investigate possible exploits of black-box MT systems and explore a preliminary defense against such threats. We first show that MT systems can be sto… ▽ More Adversaries may look to steal or attack black-box NLP systems, either for financial gain or to exploit model errors. One setting of particular interest is machine translation (MT), where models have high commercial value and errors can be costly. We investigate possible exploits of black-box MT systems and explore a preliminary defense against such threats. We first show that MT systems can be stolen by querying them with monolingual sentences and training models to imitate their outputs. Using simulated experiments, we demonstrate that MT model stealing is possible even when imitation models have different input data or architectures than their target models. Applying these ideas, we train imitation models that reach within 0.6 BLEU of three production MT systems on both high-resource and low-resource language pairs. We then leverage the similarity of our imitation models to transfer adversarial examples to the production systems. We use gradient-based attacks that expose inputs which lead to semantically-incorrect translations, dropped content, and vulgar model outputs. To mitigate these vulnerabilities, we propose a defense that modifies translation outputs in order to misdirect the optimization of imitation models. This defense degrades the adversary's BLEU score and attack success rate at some cost in the defender's BLEU and inference speed. △ Less

Submitted 3 January, 2021; v1 submitted 30 April, 2020; originally announced April 2020.

Comments: EMNLP 2020

Rethinking Fully Convolutional Networks for the Analysis of Photoluminescence Wafer Images

Authors: Maike Lorena Stern, Hans Lindberg, Klaus Meyer-Wegener

Abstract: The manufacturing of light-emitting diodes is a complex semiconductor-manufacturing process, interspersed with different measurements. Among the employed measurements, photoluminescence imaging has several advantages, namely being a non-destructive, fast and thus cost-effective measurement. On a photoluminescence measurement image of an LED wafer, every pixel corresponds to an LED chip's brightnes… ▽ More The manufacturing of light-emitting diodes is a complex semiconductor-manufacturing process, interspersed with different measurements. Among the employed measurements, photoluminescence imaging has several advantages, namely being a non-destructive, fast and thus cost-effective measurement. On a photoluminescence measurement image of an LED wafer, every pixel corresponds to an LED chip's brightness after photo-excitation, revealing chip performance information. However, generating a chip-fine defect map of the LED wafer, based on photoluminescence images, proves challenging for multiple reasons: on the one hand, the measured brightness values vary from image to image, in addition to local spots of differing brightness. On the other hand, certain defect structures may assume multiple shapes, sizes and brightness gradients, where salient brightness values may correspond to defective LED chips, measurement artefacts or non-defective structures. In this work, we revisit the creation of chip-fine defect maps using fully convolutional networks and show that the problem of segmenting objects at multiple scales can be improved by the incorporation of densely connected convolutional blocks and atrous spatial pyramid pooling modules. We also share implementation details and our experiences with training networks with small datasets of measurement images. The proposed architecture significantly improves the segmentation accuracy of highly variable defect structures over our previous version. △ Less

Submitted 1 March, 2020; originally announced March 2020.

Insertion-Deletion Transformer

Authors: Laura Ruis, Mitchell Stern, Julia Proskurnia, William Chan

Abstract: We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution… ▽ More We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis. The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on-policy from the insertion model output. We demonstrate the effectiveness of our Insertion-Deletion Transformer on synthetic translation tasks, obtaining significant BLEU score improvement over an insertion-only model. △ Less

Submitted 15 January, 2020; originally announced January 2020.

Comments: Accepted as an Extended Abstract at the Workshop of Neural Generation and Translation (WNGT 2019) at EMNLP 2019

Robo-advising: Learning Investors' Risk Preferences via Portfolio Choices

Authors: Humoud Alsabah, Agostino Capponi, Octavio Ruiz Lacedelli, Matt Stern

Abstract: We introduce a reinforcement learning framework for retail robo-advising. The robo-advisor does not know the investor's risk preference, but learns it over time by observing her portfolio choices in different market environments. We develop an exploration-exploitation algorithm which trades off costly solicitations of portfolio choices by the investor with autonomous trading decisions based on sta… ▽ More We introduce a reinforcement learning framework for retail robo-advising. The robo-advisor does not know the investor's risk preference, but learns it over time by observing her portfolio choices in different market environments. We develop an exploration-exploitation algorithm which trades off costly solicitations of portfolio choices by the investor with autonomous trading decisions based on stale estimates of investor's risk aversion. We show that the algorithm's value function converges to the optimal value function of an omniscient robo-advisor over a number of periods that is polynomial in the state and action space. By correcting for the investor's mistakes, the robo-advisor may outperform a stand-alone investor, regardless of the investor's opportunity cost for making portfolio decisions. △ Less

Submitted 16 November, 2019; v1 submitted 5 November, 2019; originally announced November 2019.

MSC Class: 68T01 ACM Class: I.2.6

An Empirical Study of Generation Order for Machine Translation

Authors: William Chan, Mitchell Stern, Jamie Kiros, Jakob Uszkoreit

Abstract: In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-bas… ▽ More In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-based orders, frequency-based orders, content-based orders, and model-based orders. Curiously, we find that for the WMT'14 English $\to$ German translation task, order does not have a substantial impact on output quality, with unintuitive orderings such as alphabetical and shortest-first matching the performance of a standard Transformer. This demonstrates that traditional left-to-right generation is not strictly necessary to achieve high performance. On the other hand, results on the WMT'18 English $\to$ Chinese task tend to vary more widely, suggesting that translation for less well-aligned language pairs may be more sensitive to generation order. △ Less

Submitted 29 October, 2019; originally announced October 2019.

Fully Convolutional Networks for Chip-wise Defect Detection Employing Photoluminescence Images

Authors: Maike Lorena Stern, Martin Schellenberger

Abstract: Efficient quality control is inevitable in the manufacturing of light-emitting diodes (LEDs). Because defective LED chips may be traced back to different causes, a time and cost-intensive electrical and optical contact measurement is employed. Fast photoluminescence measurements, on the other hand, are commonly used to detect wafer separation damages but also hold the potential to enable an effici… ▽ More Efficient quality control is inevitable in the manufacturing of light-emitting diodes (LEDs). Because defective LED chips may be traced back to different causes, a time and cost-intensive electrical and optical contact measurement is employed. Fast photoluminescence measurements, on the other hand, are commonly used to detect wafer separation damages but also hold the potential to enable an efficient detection of all kinds of defective LED chips. On a photoluminescence image, every pixel corresponds to an LED chip's brightness after photoexcitation, revealing performance information. But due to unevenly distributed brightness values and varying defect patterns, photoluminescence images are not yet employed for a comprehensive defect detection. In this work, we show that fully convolutional networks can be used for chip-wise defect detection, trained on a small data-set of photoluminescence images. Pixel-wise labels allow us to classify each and every chip as defective or not. Being measurement-based, labels are easy to procure and our experiments show that existing discrepancies between training images and labels do not hinder network training. Using weighted loss calculation, we were able to equalize our highly unbalanced class categories. Due to the consistent use of skip connections and residual shortcuts, our network is able to predict a variety of structures, from extensive defect clusters up to single defective LED chips. △ Less

Submitted 6 October, 2019; originally announced October 2019.

Comments: 14 pages, 12 figures

ACM Class: J.2; I.2.10

Journal ref: J Intell Manuf (2020) 1-14

On combining features for single-channel robust speech recognition in reverberant environments

Authors: José Novoa, Josué Fredes, Jorge Wuth, Fernando Huenupán, Richard M. Stern, Nestor Becerra Yoma

Abstract: This paper addresses the combination of complementary parallel speech recognition systems to reduce the error rate of speech recognition systems operating in real highly-reverberant environments. First, the testing environment consists of recordings of speech in a calibrated real room with reverberation times from 0.47 to 1.77 seconds and speaker-to-microphone distances of 0.16 to 2.56 meters. We… ▽ More This paper addresses the combination of complementary parallel speech recognition systems to reduce the error rate of speech recognition systems operating in real highly-reverberant environments. First, the testing environment consists of recordings of speech in a calibrated real room with reverberation times from 0.47 to 1.77 seconds and speaker-to-microphone distances of 0.16 to 2.56 meters. We combined systems both at the level of the DNN outputs and at the level of the final ASR outputs. Second, recognition experiments with the reverb challenge are also reported. The results presented here show that the combination of features can lead to WER improvements between 7% and 18% with speech recorded in real reverberant environments. Also, the combination at DNN-output level is much more effective than at the system-output level. However, cascading both schemes can still lead to smaller reductions in WER. △ Less

Submitted 17 June, 2019; originally announced June 2019.

KERMIT: Generative Insertion-Based Modeling for Sequences

Authors: William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, Jakob Uszkoreit

Abstract: We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to lea… ▽ More We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation. △ Less

Submitted 4 June, 2019; originally announced June 2019.

Comments: William Chan, Nikita Kitaev, Kelvin Guu, and Mitchell Stern contributed equally

Auditable Blockchain Randomization Tool

Authors: Olivia Saa, Julio Michael Stern

Abstract: Randomization is an integral part of well-designed statistical trials, and is also a required procedure in legal systems, see Marcondes et al. (2019) This paper presents an easy to implement randomization protocol that assures, in a formal mathematical setting, a statistically sound, computationally efficient, cryptographically secure, traceable and auditable randomization procedure that is also r… ▽ More Randomization is an integral part of well-designed statistical trials, and is also a required procedure in legal systems, see Marcondes et al. (2019) This paper presents an easy to implement randomization protocol that assures, in a formal mathematical setting, a statistically sound, computationally efficient, cryptographically secure, traceable and auditable randomization procedure that is also resistant to collusion and manipulation by participating agents. △ Less

Submitted 20 April, 2019; originally announced April 2019.

Comments: 7 pages

Journal ref: MaxEnt 2019 - Proceedings of the 39th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Garching, Germany, 30 June - 5 July 2019

Insertion Transformer: Flexible Sequence Generation via Insertion Operations

Authors: Mitchell Stern, William Chan, Jamie Kiros, Jakob Uszkoreit

Abstract: We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a numbe… ▽ More We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a number of advantages: for instance, not only can our model be trained to follow specific orderings such as left-to-right generation or a binary tree traversal, but it can also be trained to maximize entropy over all valid insertions for robustness. In addition, our model seamlessly accommodates both fully autoregressive generation (one insertion at a time) and partially autoregressive generation (simultaneous insertions at multiple locations). We validate our approach by analyzing its performance on the WMT 2014 English-German machine translation task under various settings for training and decoding. We find that the Insertion Transformer outperforms many prior non-autoregressive approaches to translation at comparable or better levels of parallelism, and successfully recovers the performance of the original Transformer while requiring only logarithmically many iterations during decoding. △ Less

Submitted 8 February, 2019; originally announced February 2019.

Blockwise Parallel Decoding for Deep Autoregressive Models

Authors: Mitchell Stern, Noam Shazeer, Jakob Uszkoreit

Abstract: Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the length of the critical path at training time, generation still remains an inherent… ▽ More Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the length of the critical path at training time, generation still remains an inherently sequential process. To overcome this limitation, we propose a novel blockwise parallel decoding scheme in which we make predictions for multiple time steps in parallel then back off to the longest prefix validated by a scoring model. This allows for substantial theoretical improvements in generation speed when applied to architectures that can process output sequences in parallel. We verify our approach empirically through a series of experiments using state-of-the-art self-attention models for machine translation and image super-resolution, achieving iteration reductions of up to 2x over a baseline greedy decoder with no loss in quality, or up to 7x in exchange for a slight decrease in performance. In terms of wall-clock time, our fastest models exhibit real-time speedups of up to 4x over standard greedy decoding. △ Less

Submitted 7 November, 2018; originally announced November 2018.

Comments: NIPS 2018

Reliable counting of weakly labeled concepts by a single spiking neuron model

Authors: Hannes Rapp, Martin Paul Nawrot, Merav Stern

Abstract: Making an informed, correct and quick decision can be life-saving. It's crucial for animals during an escape behaviour or for autonomous cars during driving. The decision can be complex and may involve an assessment of the amount of threats present and the nature of each threat. Thus, we should expect early sensory processing to supply classification information fast and accurately, even before re… ▽ More Making an informed, correct and quick decision can be life-saving. It's crucial for animals during an escape behaviour or for autonomous cars during driving. The decision can be complex and may involve an assessment of the amount of threats present and the nature of each threat. Thus, we should expect early sensory processing to supply classification information fast and accurately, even before relying the information to higher brain areas or more complex system components downstream. Today, advanced convolutional artificial neural networks can successfully solve visual detection and classification tasks and are commonly used to build complex decision making systems. However, in order to perform well on these tasks they require increasingly complex, "very deep" model structure, which is costly in inference run-time, energy consumption and number of training samples, only trainable on cloud-computing clusters. A single spiking neuron has been shown to be able to solve recognition tasks for homogeneous Poisson input statistics, a commonly used model for spiking activity in the neocortex. When modeled as leaky integrate and fire with gradient decent learning algorithm it was shown to posses a variety of complex computational capabilities. Here we improve its implementation. We also account for more natural stimulus generated inputs that deviate from this homogeneous Poisson spiking. The improved gradient-based local learning rule allows for significantly better and stable generalization. We also show that with its improved capabilities it can count weakly labeled concepts by applying our model to a problem of multiple instance learning (MIL) with counting where labels are only available for collections of concepts. In this counting MNIST task the neuron exploits the improved implementation and outperforms conventional ConvNet architecture under similar condtions. △ Less

Submitted 16 November, 2018; v1 submitted 19 May, 2018; originally announced May 2018.

What's Going On in Neural Constituency Parsers? An Analysis

Authors: David Gaddy, Mitchell Stern, Dan Klein

Abstract: A number of differences have emerged between modern and classic approaches to constituency parsing in recent years, with structural components like grammars and feature-rich lexicons becoming less central while recurrent neural network representations rise in popularity. The goal of this work is to analyze the extent to which information provided directly by the model structure in classical system… ▽ More A number of differences have emerged between modern and classic approaches to constituency parsing in recent years, with structural components like grammars and feature-rich lexicons becoming less central while recurrent neural network representations rise in popularity. The goal of this work is to analyze the extent to which information provided directly by the model structure in classical systems is still being captured by neural methods. To this end, we propose a high-performance neural model (92.08 F1 on PTB) that is representative of recent work and perform a series of investigative experiments. We find that our model implicitly learns to encode much of the same information that was explicitly provided by grammars and lexicons in the past, indicating that this scaffolding can largely be subsumed by powerful general-purpose neural machinery. △ Less

Submitted 20 April, 2018; originally announced April 2018.

Comments: NAACL 2018

Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

Authors: Noam Shazeer, Mitchell Stern

Abstract: In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-… ▽ More In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves. △ Less

Submitted 11 April, 2018; originally announced April 2018.

Stochastic Cubic Regularization for Fast Nonconvex Optimization

Authors: Nilesh Tripuraneni, Mitchell Stern, Chi Jin, Jeffrey Regier, Michael I. Jordan

Abstract: This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only $\mathcal{\tilde{O}}(ε^{-3.5})$ stochastic gradient and stochastic Hessian-vector product evaluations. The latter can be computed… ▽ More This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only $\mathcal{\tilde{O}}(ε^{-3.5})$ stochastic gradient and stochastic Hessian-vector product evaluations. The latter can be computed as efficiently as stochastic gradients. This improves upon the $\mathcal{\tilde{O}}(ε^{-4})$ rate of stochastic gradient descent. Our rate matches the best-known result for finding local minima without requiring any delicate acceleration or variance-reduction techniques. △ Less

Submitted 5 December, 2017; v1 submitted 8 November, 2017; originally announced November 2017.

Comments: The first two authors contributed equally

Effective Inference for Generative Neural Parsing

Authors: Mitchell Stern, Daniel Fried, Dan Klein

Abstract: Generative neural models have recently achieved state-of-the-art results for constituency parsing. However, without a feasible search procedure, their use has so far been limited to reranking the output of external parsers in which decoding is more tractable. We describe an alternative to the conventional action-level beam search used for discriminative neural models that enables us to decode dire… ▽ More Generative neural models have recently achieved state-of-the-art results for constituency parsing. However, without a feasible search procedure, their use has so far been limited to reranking the output of external parsers in which decoding is more tractable. We describe an alternative to the conventional action-level beam search used for discriminative neural models that enables us to decode directly in these generative models. We then show that by improving our basic candidate selection strategy and using a coarse pruning function, we can improve accuracy while exploring significantly less of the search space. Applied to the model of Choe and Charniak (2016), our inference procedure obtains 92.56 F1 on section 23 of the Penn Treebank, surpassing prior state-of-the-art results for single-model systems. △ Less

Submitted 27 July, 2017; originally announced July 2017.

Comments: EMNLP 2017

Improving Neural Parsing by Disentangling Model Combination and Reranking Effects

Authors: Daniel Fried, Mitchell Stern, Dan Klein

Abstract: Recent work has proposed several generative neural models for constituency parsing that achieve state-of-the-art results. Since direct search in these generative models is difficult, they have primarily been used to rescore candidate outputs from base parsers in which decoding is more straightforward. We first present an algorithm for direct search in these generative models. We then demonstrate t… ▽ More Recent work has proposed several generative neural models for constituency parsing that achieve state-of-the-art results. Since direct search in these generative models is difficult, they have primarily been used to rescore candidate outputs from base parsers in which decoding is more straightforward. We first present an algorithm for direct search in these generative models. We then demonstrate that the rescoring results are at least partly due to implicit model combination rather than reranking effects. Finally, we show that explicit model combination can improve performance even further, resulting in new state-of-the-art numbers on the PTB of 94.25 F1 when training only on gold data and 94.66 F1 when using external data. △ Less

Submitted 10 July, 2017; originally announced July 2017.

Comments: ACL 2017. The first two authors contributed equally

Kernel Feature Selection via Conditional Covariance Minimization

Authors: Jianbo Chen, Mitchell Stern, Martin J. Wainwright, Michael I. Jordan

Abstract: We propose a method for feature selection that employs kernel-based measures of independence to find a subset of covariates that is maximally predictive of the response. Building on past work in kernel dimension reduction, we show how to perform feature selection via a constrained optimization problem involving the trace of the conditional covariance operator. We prove various consistency results… ▽ More We propose a method for feature selection that employs kernel-based measures of independence to find a subset of covariates that is maximally predictive of the response. Building on past work in kernel dimension reduction, we show how to perform feature selection via a constrained optimization problem involving the trace of the conditional covariance operator. We prove various consistency results for this procedure, and also demonstrate that our method compares favorably with other state-of-the-art algorithms on a variety of synthetic and real data sets. △ Less

Submitted 20 October, 2018; v1 submitted 4 July, 2017; originally announced July 2017.

Comments: The first two authors contributed equally

Risk-Sensitive Cooperative Games for Human-Machine Systems

Authors: Agostino Capponi, Reza Ghanadan, Matt Stern

Abstract: Autonomous systems can substantially enhance a human's efficiency and effectiveness in complex environments. Machines, however, are often unable to observe the preferences of the humans that they serve. Despite the fact that the human's and machine's objectives are aligned, asymmetric information, along with heterogeneous sensitivities to risk by the human and machine, make their joint optimizatio… ▽ More Autonomous systems can substantially enhance a human's efficiency and effectiveness in complex environments. Machines, however, are often unable to observe the preferences of the humans that they serve. Despite the fact that the human's and machine's objectives are aligned, asymmetric information, along with heterogeneous sensitivities to risk by the human and machine, make their joint optimization process a game with strategic interactions. We propose a framework based on risk-sensitive dynamic games; the human seeks to optimize her risk-sensitive criterion according to her true preferences, while the machine seeks to adaptively learn the human's preferences and at the same time provide a good service to the human. We develop a class of performance measures for the proposed framework based on the concept of regret. We then evaluate their dependence on the risk-sensitivity and the degree of uncertainty. We present applications of our framework to self-driving taxis, and robo-financial advising. △ Less

Submitted 26 May, 2017; originally announced May 2017.

Comments: 15 pages, 10 figures

MSC Class: 97R40

The Marginal Value of Adaptive Gradient Methods in Machine Learning

Authors: Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht

Abstract: Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient desc… ▽ More Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks. △ Less

Submitted 21 May, 2018; v1 submitted 23 May, 2017; originally announced May 2017.

A Minimal Span-Based Neural Constituency Parser

Authors: Mitchell Stern, Jacob Andreas, Dan Klein

Abstract: In this work, we present a minimal neural model for constituency parsing based on independent scoring of labels and spans. We show that this model is not only compatible with classical dynamic programming techniques, but also admits a novel greedy top-down inference algorithm based on recursive partitioning of the input. We demonstrate empirically that both prediction schemes are competitive with… ▽ More In this work, we present a minimal neural model for constituency parsing based on independent scoring of labels and spans. We show that this model is not only compatible with classical dynamic programming techniques, but also admits a novel greedy top-down inference algorithm based on recursive partitioning of the input. We demonstrate empirically that both prediction schemes are competitive with recent work, and when combined with basic extensions to the scoring model are capable of achieving state-of-the-art single-model performance on the Penn Treebank (91.79 F1) and strong performance on the French Treebank (82.23 F1). △ Less

Submitted 10 May, 2017; originally announced May 2017.

Comments: To appear in ACL 2017

Abstract Syntax Networks for Code Generation and Semantic Parsing

Authors: Maxim Rabinovich, Mitchell Stern, Dan Klein

Abstract: Tasks like code generation and semantic parsing require mapping unstructured (or partially structured) inputs to well-formed, executable outputs. We introduce abstract syntax networks, a modeling framework for these problems. The outputs are represented as abstract syntax trees (ASTs) and constructed by a decoder with a dynamically-determined modular structure paralleling the structure of the outp… ▽ More Tasks like code generation and semantic parsing require mapping unstructured (or partially structured) inputs to well-formed, executable outputs. We introduce abstract syntax networks, a modeling framework for these problems. The outputs are represented as abstract syntax trees (ASTs) and constructed by a decoder with a dynamically-determined modular structure paralleling the structure of the output tree. On the benchmark Hearthstone dataset for code generation, our model obtains 79.2 BLEU and 22.7% exact match accuracy, compared to previous state-of-the-art values of 67.1 and 6.1%. Furthermore, we perform competitively on the Atis, Jobs, and Geo semantic parsing datasets with no task-specific engineering. △ Less

Submitted 25 April, 2017; originally announced April 2017.

Comments: ACL 2017. MR and MS contributed equally