Showing 1–50 of 159 results for author: Walters, R

OrbitGrasp: $SE(3)$-Equivariant Grasp Learning

Authors: Boce Hu, Xupeng Zhu, Dian Wang, Zihao Dong, Haojie Huang, Chenghao Wang, Robin Walters, Robert Platt

Abstract: While grasp detection is an important part of any robotic manipulation pipeline, reliable and accurate grasp detection in $SE(3)$ remains a research challenge. Many robotics applications in unstructured environments such as the home or warehouse would benefit a lot from better grasp performance. This paper proposes a novel framework for detecting $SE(3)$ grasp poses based on point cloud input. Our… ▽ More While grasp detection is an important part of any robotic manipulation pipeline, reliable and accurate grasp detection in $SE(3)$ remains a research challenge. Many robotics applications in unstructured environments such as the home or warehouse would benefit a lot from better grasp performance. This paper proposes a novel framework for detecting $SE(3)$ grasp poses based on point cloud input. Our main contribution is to propose an $SE(3)$-equivariant model that maps each point in the cloud to a continuous grasp quality function over the 2-sphere $S^2$ using a spherical harmonic basis. Compared with reasoning about a finite set of samples, this formulation improves the accuracy and efficiency of our model when a large number of samples would otherwise be needed. In order to accomplish this, we propose a novel variation on EquiFormerV2 that leverages a UNet-style backbone to enlarge the number of points the model can handle. Our resulting method, which we name $\textit{OrbitGrasp}$, significantly outperforms baselines in both simulation and physical experiments. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Equivariant Diffusion Policy

Authors: Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, Robert Platt

Abstract: Recent work has shown diffusion models are an effective approach to learning the multimodal distributions arising from demonstration data in behavior cloning. However, a drawback of this approach is the need to learn a denoising function, which is significantly more complex than learning an explicit policy. In this work, we propose Equivariant Diffusion Policy, a novel diffusion policy learning me… ▽ More Recent work has shown diffusion models are an effective approach to learning the multimodal distributions arising from demonstration data in behavior cloning. However, a drawback of this approach is the need to learn a denoising function, which is significantly more complex than learning an explicit policy. In this work, we propose Equivariant Diffusion Policy, a novel diffusion policy learning method that leverages domain symmetries to obtain better sample efficiency and generalization in the denoising function. We theoretically analyze the $\mathrm{SO}(2)$ symmetry of full 6-DoF control and characterize when a diffusion model is $\mathrm{SO}(2)$-equivariant. We furthermore evaluate the method empirically on a set of 12 simulation tasks in MimicGen, and show that it obtains a success rate that is, on average, 21.9% higher than the baseline Diffusion Policy. We also evaluate the method on a real-world system to show that effective policies can be learned with relatively few training samples, whereas the baseline Diffusion Policy cannot. △ Less

Submitted 1 July, 2024; originally announced July 2024.

Open-vocabulary Pick and Place via Patch-level Semantic Maps

Authors: Mingxi Jia, Haojie Huang, Zhewen Zhang, Chenghao Wang, Linfeng Zhao, Dian Wang, Jason Xinyu Liu, Robin Walters, Robert Platt, Stefanie Tellex

Abstract: Controlling robots through natural language instructions in open-vocabulary scenarios is pivotal for enhancing human-robot collaboration and complex robot behavior synthesis. However, achieving this capability poses significant challenges due to the need for a system that can generalize from limited data to a wide range of tasks and environments. Existing methods rely on large, costly datasets and… ▽ More Controlling robots through natural language instructions in open-vocabulary scenarios is pivotal for enhancing human-robot collaboration and complex robot behavior synthesis. However, achieving this capability poses significant challenges due to the need for a system that can generalize from limited data to a wide range of tasks and environments. Existing methods rely on large, costly datasets and struggle with generalization. This paper introduces Grounded Equivariant Manipulation (GEM), a novel approach that leverages the generative capabilities of pre-trained vision-language models and geometric symmetries to facilitate few-shot and zero-shot learning for open-vocabulary robot manipulation tasks. Our experiments demonstrate GEM's high sample efficiency and superior generalization across diverse pick-and-place tasks in both simulation and real-world experiments, showcasing its ability to adapt to novel instructions and unseen objects with minimal data requirements. GEM advances a significant step forward in the domain of language-conditioned robot control, bridging the gap between semantic understanding and action generation in robotic systems. △ Less

Submitted 21 June, 2024; originally announced June 2024.

Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies

Authors: Haojie Huang, Karl Schmeckpeper, Dian Wang, Ondrej Biza, Yaoyao Qian, Haotian Liu, Mingxi Jia, Robert Platt, Robin Walters

Abstract: Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation.… ▽ More Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines. △ Less

Submitted 17 June, 2024; originally announced June 2024.

The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof

Authors: Derek Lim, Moe Putterman, Robin Walters, Haggai Maron, Stefanie Jegelka

Abstract: Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries -- transformations of neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization or loss-landscapes. However, theoreti… ▽ More Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries -- transformations of neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization or loss-landscapes. However, theoretical analysis of the relationship between parameter space symmetries and these phenomena is difficult. In this work, we empirically investigate the impact of neural parameter symmetries by introducing new neural network architectures that have reduced parameter space symmetries. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries. With these new methods, we conduct a comprehensive experimental study consisting of multiple tasks aimed at assessing the effect of removing parameter symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries; for instance, we observe linear mode connectivity between our networks without alignment of weight spaces, and we find that our networks allow for faster and more effective Bayesian neural network training. △ Less

Submitted 20 June, 2024; v1 submitted 30 May, 2024; originally announced May 2024.

Comments: 27 pages. Preparing code for release. v2: added / updated some citations

Symmetry-Informed Governing Equation Discovery

Authors: Jianke Yang, Wang Rao, Nima Dehmamy, Robin Walters, Rose Yu

Abstract: Despite the advancements in learning governing differential equations from observations of dynamical systems, data-driven methods are often unaware of fundamental physical laws, such as frame invariance. As a result, these algorithms may search an unnecessarily large space and discover equations that are less accurate or overly complex. In this paper, we propose to leverage symmetry in automated e… ▽ More Despite the advancements in learning governing differential equations from observations of dynamical systems, data-driven methods are often unaware of fundamental physical laws, such as frame invariance. As a result, these algorithms may search an unnecessarily large space and discover equations that are less accurate or overly complex. In this paper, we propose to leverage symmetry in automated equation discovery to compress the equation search space and improve the accuracy and simplicity of the learned equations. Specifically, we derive equivariance constraints from the time-independent symmetries of ODEs. Depending on the types of symmetries, we develop a pipeline for incorporating symmetry constraints into various equation discovery algorithms, including sparse regression and genetic programming. In experiments across a diverse range of dynamical systems, our approach demonstrates better robustness against noise and recovers governing equations with significantly higher probability than baselines without symmetry. △ Less

Submitted 26 May, 2024; originally announced May 2024.

Learning Galaxy Intrinsic Alignment Correlations

Authors: Sneh Pandya, Yuanyuan Yang, Nicholas Van Alfen, Jonathan Blazek, Robin Walters

Abstract: The intrinsic alignments (IA) of galaxies, regarded as a contaminant in weak lensing analyses, represents the correlation of galaxy shapes due to gravitational tidal interactions and galaxy formation processes. As such, understanding IA is paramount for accurate cosmological inferences from weak lensing surveys; however, one limitation to our understanding and mitigation of IA is expensive simulat… ▽ More The intrinsic alignments (IA) of galaxies, regarded as a contaminant in weak lensing analyses, represents the correlation of galaxy shapes due to gravitational tidal interactions and galaxy formation processes. As such, understanding IA is paramount for accurate cosmological inferences from weak lensing surveys; however, one limitation to our understanding and mitigation of IA is expensive simulation-based modeling. In this work, we present a deep learning approach to emulate galaxy position-position ($ξ$), position-orientation ($ω$), and orientation-orientation ($η$) correlation function measurements and uncertainties from halo occupation distribution-based mock galaxy catalogs. We find strong Pearson correlation values with the model across all three correlation functions and further predict aleatoric uncertainties through a mean-variance estimation training procedure. $ξ(r)$ predictions are generally accurate to $\leq10\%$. Our model also successfully captures the underlying signal of the noisier correlations $ω(r)$ and $η(r)$, although with a lower average accuracy. We find that the model performance is inhibited by the stochasticity of the data, and will benefit from correlations averaged over multiple data realizations. Our code will be made open source upon journal publication. △ Less

Submitted 21 April, 2024; originally announced April 2024.

Comments: 15 pages, 6 figures, 1 table. Accepted at the Data-centric Machine Learning Research (DMLR) Workshop at ICLR 2024

TopoX: A Suite of Python Packages for Machine Learning on Topological Domains

Authors: Mustafa Hajij, Mathilde Papillon, Florian Frantzen, Jens Agerberg, Ibrahem AlJabea, Ruben Ballester, Claudio Battiloro, Guillermo Bernárdez, Tolga Birdal, Aiden Brent, Peter Chin, Sergio Escalera, Simone Fiorellino, Odin Hoff Gardaa, Gurusankar Gopalakrishnan, Devendra Govil, Josef Hoppe, Maneel Reddy Karri, Jude Khouja, Manuel Lecha, Neal Livesay, Jan Meißner, Soham Mukherjee, Alexander Nikitin, Theodore Papamarkou , et al. (18 additional authors not shown)

Abstract: We introduce TopoX, a Python software suite that provides reliable and user-friendly building blocks for computing and machine learning on topological domains that extend graphs: hypergraphs, simplicial, cellular, path and combinatorial complexes. TopoX consists of three packages: TopoNetX facilitates constructing and computing on these domains, including working with nodes, edges and higher-order… ▽ More We introduce TopoX, a Python software suite that provides reliable and user-friendly building blocks for computing and machine learning on topological domains that extend graphs: hypergraphs, simplicial, cellular, path and combinatorial complexes. TopoX consists of three packages: TopoNetX facilitates constructing and computing on these domains, including working with nodes, edges and higher-order cells; TopoEmbedX provides methods to embed topological domains into vector spaces, akin to popular graph-based embedding algorithms such as node2vec; TopoModelx is built on top of PyTorch and offers a comprehensive toolbox of higher-order message passing functions for neural networks on topological domains. The extensively documented and unit-tested source code of TopoX is available under MIT license at https://pyt-team.github.io/. △ Less

Submitted 17 February, 2024; v1 submitted 4 February, 2024; originally announced February 2024.

Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D

Authors: Haojie Huang, Owen Howell, Dian Wang, Xupeng Zhu, Robin Walters, Robert Platt

Abstract: Many complex robotic manipulation tasks can be decomposed as a sequence of pick and place actions. Training a robotic agent to learn this sequence over many different starting conditions typically requires many iterations or demonstrations, especially in 3D environments. In this work, we propose Fourier Transporter (FourTran) which leverages the two-fold SE(d)xSE(d) symmetry in the pick-place prob… ▽ More Many complex robotic manipulation tasks can be decomposed as a sequence of pick and place actions. Training a robotic agent to learn this sequence over many different starting conditions typically requires many iterations or demonstrations, especially in 3D environments. In this work, we propose Fourier Transporter (FourTran) which leverages the two-fold SE(d)xSE(d) symmetry in the pick-place problem to achieve much higher sample efficiency. FourTran is an open-loop behavior cloning method trained using expert demonstrations to predict pick-place actions on new environments. FourTran is constrained to incorporate symmetries of the pick and place actions independently. Our method utilizes a fiber space Fourier transformation that allows for memory-efficient construction. We test our proposed network on the RLbench benchmark and achieve state-of-the-art results across various tasks. △ Less

Submitted 15 March, 2024; v1 submitted 22 January, 2024; originally announced January 2024.

Comments: ICLR 2024

Topological Obstructions and How to Avoid Them

Authors: Babak Esmaeili, Robin Walters, Heiko Zimmermann, Jan-Willem van de Meent

Abstract: Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, we theoretically and empirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima can arise due to singularities (e.g. self… ▽ More Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, we theoretically and empirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima can arise due to singularities (e.g. self-intersection) or due to an incorrect degree or winding number. We then discuss how normalizing flows can potentially circumvent these obstructions by defining multimodal variational distributions. Inspired by this observation, we propose a new flow-based model that maps data points to multimodal distributions over geometric spaces and empirically evaluate our model on 2 domains. We observe improved stability during training and a higher chance of converging to a homeomorphic encoder. △ Less

Submitted 12 December, 2023; originally announced December 2023.

Fast and Expressive Gesture Recognition using a Combination-Homomorphic Electromyogram Encoder

Authors: Niklas Smedemark-Margulies, Yunus Bicer, Elifnur Sunger, Tales Imbiriba, Eugene Tunik, Deniz Erdogmus, Mathew Yarossi, Robin Walters

Abstract: We study the task of gesture recognition from electromyography (EMG), with the goal of enabling expressive human-computer interaction at high accuracy, while minimizing the time required for new subjects to provide calibration data. To fulfill these goals, we define combination gestures consisting of a direction component and a modifier component. New subjects only demonstrate the single component… ▽ More We study the task of gesture recognition from electromyography (EMG), with the goal of enabling expressive human-computer interaction at high accuracy, while minimizing the time required for new subjects to provide calibration data. To fulfill these goals, we define combination gestures consisting of a direction component and a modifier component. New subjects only demonstrate the single component gestures and we seek to extrapolate from these to all possible single or combination gestures. We extrapolate to unseen combination gestures by combining the feature vectors of real single gestures to produce synthetic training data. This strategy allows us to provide a large and flexible gesture vocabulary, while not requiring new subjects to demonstrate combinatorially many example gestures. We pre-train an encoder and a combination operator using self-supervision, so that we can produce useful synthetic training data for unseen test subjects. To evaluate the proposed method, we collect a real-world EMG dataset, and measure the effect of augmented supervision against two baselines: a partially-supervised model trained with only single gesture data from the unseen subject, and a fully-supervised model trained with real single and real combination gesture data from the unseen subject. We find that the proposed method provides a dramatic improvement over the partially-supervised model, and achieves a useful classification accuracy that in some cases approaches the performance of the fully-supervised model. △ Less

Submitted 29 November, 2023; v1 submitted 30 October, 2023; originally announced November 2023.

Comments: 24 pages, 7 figures, 6 tables V2: add link to code, fix bibliography

Modeling Dynamics over Meshes with Gauge Equivariant Nonlinear Message Passing

Authors: Jung Yeon Park, Lawson L. S. Wong, Robin Walters

Abstract: Data over non-Euclidean manifolds, often discretized as surface meshes, naturally arise in computer graphics and biological and physical systems. In particular, solutions to partial differential equations (PDEs) over manifolds depend critically on the underlying geometry. While graph neural networks have been successfully applied to PDEs, they do not incorporate surface geometry and do not conside… ▽ More Data over non-Euclidean manifolds, often discretized as surface meshes, naturally arise in computer graphics and biological and physical systems. In particular, solutions to partial differential equations (PDEs) over manifolds depend critically on the underlying geometry. While graph neural networks have been successfully applied to PDEs, they do not incorporate surface geometry and do not consider local gauge symmetries of the manifold. Alternatively, recent works on gauge equivariant convolutional and attentional architectures on meshes leverage the underlying geometry but underperform in modeling surface PDEs with complex nonlinear dynamics. To address these issues, we introduce a new gauge equivariant architecture using nonlinear message passing. Our novel architecture achieves higher performance than either convolutional or attentional networks on domains with highly complex and nonlinear dynamics. However, similar to the non-mesh case, design trade-offs favor convolutional, attentional, or message passing networks for different tasks; we investigate in which circumstances our message passing method provides the most benefit. △ Less

Submitted 2 November, 2023; v1 submitted 30 October, 2023; originally announced October 2023.

Comments: Accepted to NeurIPS 2023

Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution

Authors: Rui Wang, Elyssa Hofgard, Han Gao, Robin Walters, Tess E. Smidt

Abstract: Modeling symmetry breaking is essential for understanding the fundamental changes in the behaviors and properties of physical systems, from microscopic particle interactions to macroscopic phenomena like fluid dynamics and cosmic structures. Thus, identifying sources of asymmetry is an important tool for understanding physical systems. In this paper, we focus on learning asymmetries of data using… ▽ More Modeling symmetry breaking is essential for understanding the fundamental changes in the behaviors and properties of physical systems, from microscopic particle interactions to macroscopic phenomena like fluid dynamics and cosmic structures. Thus, identifying sources of asymmetry is an important tool for understanding physical systems. In this paper, we focus on learning asymmetries of data using relaxed group convolutions. We provide both theoretical and empirical evidence that this flexible convolution technique allows the model to maintain the highest level of equivariance that is consistent with data and discover the subtle symmetry-breaking factors in various physical systems. We employ various relaxed group convolution architectures to uncover various symmetry-breaking factors that are interpretable and physically meaningful in different physical systems, including the phase transition of crystal structure, the isotropy and homogeneity breaking in turbulent flow, and the time-reversal symmetry breaking in pendulum systems. △ Less

Submitted 1 June, 2024; v1 submitted 3 October, 2023; originally announced October 2023.

Latent Space Symmetry Discovery

Authors: Jianke Yang, Nima Dehmamy, Robin Walters, Rose Yu

Abstract: Equivariant neural networks require explicit knowledge of the symmetry group. Automatic symmetry discovery methods aim to relax this constraint and learn invariance and equivariance from data. However, existing symmetry discovery methods are limited to simple linear symmetries and cannot handle the complexity of real-world data. We propose a novel generative model, Latent LieGAN (LaLiGAN), which c… ▽ More Equivariant neural networks require explicit knowledge of the symmetry group. Automatic symmetry discovery methods aim to relax this constraint and learn invariance and equivariance from data. However, existing symmetry discovery methods are limited to simple linear symmetries and cannot handle the complexity of real-world data. We propose a novel generative model, Latent LieGAN (LaLiGAN), which can discover symmetries of nonlinear group actions. It learns a mapping from the data space to a latent space where the symmetries become linear and simultaneously discovers symmetries in the latent space. Theoretically, we show that our method can express any nonlinear symmetry under some conditions about the group action. Experimentally, we demonstrate that our method can accurately discover the intrinsic symmetry in high-dimensional dynamical systems. LaLiGAN also results in a well-structured latent space that is useful for downstream tasks including equation discovery and long-term forecasting. △ Less

Submitted 23 April, 2024; v1 submitted 29 September, 2023; originally announced October 2023.

ICML 2023 Topological Deep Learning Challenge : Design and Results

Authors: Mathilde Papillon, Mustafa Hajij, Helen Jenne, Johan Mathe, Audun Myers, Theodore Papamarkou, Tolga Birdal, Tamal Dey, Tim Doster, Tegan Emerson, Gurusankar Gopalakrishnan, Devendra Govil, Aldo Guzmán-Sáenz, Henry Kvinge, Neal Livesay, Soham Mukherjee, Shreyas N. Samaga, Karthikeyan Natesan Ramamurthy, Maneel Reddy Karri, Paul Rosen, Sophia Sanborn, Robin Walters, Jens Agerberg, Sadrodin Barikbin, Claudio Battiloro , et al. (31 additional authors not shown)

Abstract: This paper presents the computational challenge on topological deep learning that was hosted within the ICML 2023 Workshop on Topology and Geometry in Machine Learning. The competition asked participants to provide open-source implementations of topological neural networks from the literature by contributing to the python packages TopoNetX (data processing) and TopoModelX (deep learning). The chal… ▽ More This paper presents the computational challenge on topological deep learning that was hosted within the ICML 2023 Workshop on Topology and Geometry in Machine Learning. The competition asked participants to provide open-source implementations of topological neural networks from the literature by contributing to the python packages TopoNetX (data processing) and TopoModelX (deep learning). The challenge attracted twenty-eight qualifying submissions in its two-month duration. This paper describes the design of the challenge and summarizes its main findings. △ Less

Submitted 18 January, 2024; v1 submitted 26 September, 2023; originally announced September 2023.

Leveraging Symmetries in Pick and Place

Authors: Haojie Huang, Dian Wang, Arsh Tangri, Robin Walters, Robert Platt

Abstract: Robotic pick and place tasks are symmetric under translations and rotations of both the object to be picked and the desired place pose. For example, if the pick object is rotated or translated, then the optimal pick action should also rotate or translate. The same is true for the place pose; if the desired place pose changes, then the place action should also transform accordingly. A recently prop… ▽ More Robotic pick and place tasks are symmetric under translations and rotations of both the object to be picked and the desired place pose. For example, if the pick object is rotated or translated, then the optimal pick action should also rotate or translate. The same is true for the place pose; if the desired place pose changes, then the place action should also transform accordingly. A recently proposed pick and place framework known as Transporter Net captures some of these symmetries, but not all. This paper analytically studies the symmetries present in planar robotic pick and place and proposes a method of incorporating equivariant neural models into Transporter Net in a way that captures all symmetries. The new model, which we call Equivariant Transporter Net, is equivariant to both pick and place symmetries and can immediately generalize pick and place knowledge to different pick and place poses. We evaluate the new model empirically and show that it is much more sample efficient than the non-symmetric version, resulting in a system that can imitate demonstrated pick and place behavior using very few human demonstrations on a variety of imitation learning tasks. △ Less

Submitted 22 December, 2023; v1 submitted 15 August, 2023; originally announced August 2023.

Comments: International Journal of Robotics Research. arXiv admin note: substantial text overlap with arXiv:2202.09400

Disentangling Node Attributes from Graph Topology for Improved Generalizability in Link Prediction

Authors: Ayan Chatterjee, Robin Walters, Giulia Menichetti, Tina Eliassi-Rad

Abstract: Link prediction is a crucial task in graph machine learning with diverse applications. We explore the interplay between node attributes and graph topology and demonstrate that incorporating pre-trained node attributes improves the generalization power of link prediction models. Our proposed method, UPNA (Unsupervised Pre-training of Node Attributes), solves the inductive link prediction problem by… ▽ More Link prediction is a crucial task in graph machine learning with diverse applications. We explore the interplay between node attributes and graph topology and demonstrate that incorporating pre-trained node attributes improves the generalization power of link prediction models. Our proposed method, UPNA (Unsupervised Pre-training of Node Attributes), solves the inductive link prediction problem by learning a function that takes a pair of node attributes and predicts the probability of an edge, as opposed to Graph Neural Networks (GNN), which can be prone to topological shortcuts in graphs with power-law degree distribution. In this manner, UPNA learns a significant part of the latent graph generation mechanism since the learned function can be used to add incoming nodes to a growing graph. By leveraging pre-trained node attributes, we overcome observational bias and make meaningful predictions about unobserved nodes, surpassing state-of-the-art performance (3X to 34X improvement on benchmark datasets). UPNA can be applied to various pairwise learning tasks and integrated with existing link prediction models to enhance their generalizability and bolster graph generative models. △ Less

Submitted 17 July, 2023; originally announced July 2023.

Comments: 17 pages, 6 figures

Can Euclidean Symmetry be Leveraged in Reinforcement Learning and Planning?

Authors: Linfeng Zhao, Owen Howell, Jung Yeon Park, Xupeng Zhu, Robin Walters, Lawson L. S. Wong

Abstract: In robotic tasks, changes in reference frames typically do not influence the underlying physical properties of the system, which has been known as invariance of physical laws.These changes, which preserve distance, encompass isometric transformations such as translations, rotations, and reflections, collectively known as the Euclidean group. In this work, we delve into the design of improved learn… ▽ More In robotic tasks, changes in reference frames typically do not influence the underlying physical properties of the system, which has been known as invariance of physical laws.These changes, which preserve distance, encompass isometric transformations such as translations, rotations, and reflections, collectively known as the Euclidean group. In this work, we delve into the design of improved learning algorithms for reinforcement learning and planning tasks that possess Euclidean group symmetry. We put forth a theory on that unify prior work on discrete and continuous symmetry in reinforcement learning, planning, and optimal control. Algorithm side, we further extend the 2D path planning with value-based planning to continuous MDPs and propose a pipeline for constructing equivariant sampling-based planning algorithms. Our work is substantiated with empirical evidence and illustrated through examples that explain the benefits of equivariance to Euclidean symmetry in tackling natural control problems. △ Less

Submitted 17 July, 2023; originally announced July 2023.

Comments: Preprint. Website: http://lfzhao.com/SymCtrl

Equivariant Single View Pose Prediction Via Induced and Restricted Representations

Authors: Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, Robin Walters

Abstract: Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three dimensions to make predictions about novel images. However, imposing SO(3)-equivariance on two-dimensional inputs is difficult because the group of three-di… ▽ More Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three dimensions to make predictions about novel images. However, imposing SO(3)-equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of SO(3) will rotate an image out of plane. We show that an algorithm that learns a three-dimensional representation of the world from two dimensional images must satisfy certain geometric consistency properties which we formulate as SO(2)-equivariance constraints. We use the induced and restricted representations of SO(2) on SO(3) to construct and classify architectures which satisfy these geometric consistency constraints. We prove that any architecture which respects said consistency constraints can be realized as an instance of our construction. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new algorithm that is a learnable generalization of previously considered methods. We test our architecture on three pose predictions task and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks. △ Less

Submitted 7 July, 2023; originally announced July 2023.

A Discrimination Report Card

Authors: Patrick Kline, Evan K. Rose, Christopher R. Walters

Abstract: We develop an Empirical Bayes grading scheme that balances the informativeness of the assigned grades against the expected frequency of ranking errors. Applying the method to a massive correspondence experiment, we grade the racial biases of 97 U.S. employers. A four-grade ranking limits the chances that a randomly selected pair of firms is mis-ranked to 5% while explaining nearly half of the vari… ▽ More We develop an Empirical Bayes grading scheme that balances the informativeness of the assigned grades against the expected frequency of ranking errors. Applying the method to a massive correspondence experiment, we grade the racial biases of 97 U.S. employers. A four-grade ranking limits the chances that a randomly selected pair of firms is mis-ranked to 5% while explaining nearly half of the variation in firms' racial contact gaps. The grades are presented alongside measures of uncertainty about each firm's contact gap in an accessible rubric that is easily adapted to other settings where ranks and levels are of simultaneous interest. △ Less

Submitted 22 June, 2023; originally announced June 2023.

One-shot Imitation Learning via Interaction Warping

Authors: Ondrej Biza, Skye Thompson, Kishore Reddy Pagidi, Abhinav Kumar, Elise van der Pol, Robin Walters, Thomas Kipf, Jan-Willem van de Meent, Lawson L. S. Wong, Robert Platt

Abstract: Imitation learning of robot policies from few demonstrations is crucial in open-ended applications. We propose a new method, Interaction Warping, for learning SE(3) robotic manipulation policies from a single demonstration. We infer the 3D mesh of each object in the environment using shape warping, a technique for aligning point clouds across object instances. Then, we represent manipulation actio… ▽ More Imitation learning of robot policies from few demonstrations is crucial in open-ended applications. We propose a new method, Interaction Warping, for learning SE(3) robotic manipulation policies from a single demonstration. We infer the 3D mesh of each object in the environment using shape warping, a technique for aligning point clouds across object instances. Then, we represent manipulation actions as keypoints on objects, which can be warped with the shape of the object. We show successful one-shot imitation learning on three simulated and real-world object re-arrangement tasks. We also demonstrate the ability of our method to predict object meshes and robot grasps in the wild. △ Less

Submitted 4 November, 2023; v1 submitted 21 June, 2023; originally announced June 2023.

Comments: CoRL 2023

On Robot Grasp Learning Using Equivariant Models

Authors: Xupeng Zhu, Dian Wang, Guanang Su, Ondrej Biza, Robin Walters, Robert Platt

Abstract: Real-world grasp detection is challenging due to the stochasticity in grasp dynamics and the noise in hardware. Ideally, the system would adapt to the real world by training directly on physical systems. However, this is generally difficult due to the large amount of training data required by most grasp learning models. In this paper, we note that the planar grasp function is $\SE(2)$-equivariant… ▽ More Real-world grasp detection is challenging due to the stochasticity in grasp dynamics and the noise in hardware. Ideally, the system would adapt to the real world by training directly on physical systems. However, this is generally difficult due to the large amount of training data required by most grasp learning models. In this paper, we note that the planar grasp function is $\SE(2)$-equivariant and demonstrate that this structure can be used to constrain the neural network used during learning. This creates an inductive bias that can significantly improve the sample efficiency of grasp learning and enable end-to-end training from scratch on a physical robot with as few as $600$ grasp attempts. We call this method Symmetric Grasp learning (SymGrasp) and show that it can learn to grasp ``from scratch'' in less that 1.5 hours of physical robot time. △ Less

Submitted 10 June, 2023; originally announced June 2023.

Comments: Accepted in Autonomous Robot. arXiv admin note: substantial text overlap with arXiv:2202.09468

Improving Convergence and Generalization Using Parameter Symmetries

Authors: Bo Zhao, Robert M. Gower, Robin Walters, Rose Yu

Abstract: In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only sp… ▽ More In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only speeds up optimization in the short-term, but gives overall faster time to convergence. Additionally, teleporting to minima with different curvatures improves generalization, which suggests a connection between the curvature of the minimum and generalization ability. Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence. Our results showcase the versatility of teleportation and demonstrate the potential of incorporating symmetry in optimization. △ Less

Submitted 13 April, 2024; v1 submitted 22 May, 2023; originally announced May 2023.

Comments: 28 pages, 13 figures, ICLR 2024

A General Theory of Correct, Incorrect, and Extrinsic Equivariance

Authors: Dian Wang, Xupeng Zhu, Jung Yeon Park, Mingxi Jia, Guanang Su, Robert Platt, Robin Walters

Abstract: Although equivariant machine learning has proven effective at many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in an equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially in the domain. In this work, w… ▽ More Although equivariant machine learning has proven effective at many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in an equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially in the domain. In this work, we present a general theory for such a situation. We propose pointwise definitions of correct, incorrect, and extrinsic equivariance, which allow us to quantify continuously the degree of each type of equivariance a function displays. We then study the impact of various degrees of incorrect or extrinsic symmetry on model error. We prove error lower bounds for invariant or equivariant networks in classification or regression settings with partially incorrect symmetry. We also analyze the potentially harmful effects of extrinsic equivariance. Experiments validate these results in three different environments. △ Less

Submitted 28 October, 2023; v1 submitted 8 March, 2023; originally announced March 2023.

Comments: Published at NeurIPS 2023

Image to Sphere: Learning Equivariant Features for Efficient Pose Prediction

Authors: David M. Klee, Ondrej Biza, Robert Platt, Robin Walters

Abstract: Predicting the pose of objects from a single image is an important but difficult computer vision problem. Methods that predict a single point estimate do not predict the pose of objects with symmetries well and cannot represent uncertainty. Alternatively, some works predict a distribution over orientations in $\mathrm{SO}(3)$. However, training such models can be computation- and sample-inefficien… ▽ More Predicting the pose of objects from a single image is an important but difficult computer vision problem. Methods that predict a single point estimate do not predict the pose of objects with symmetries well and cannot represent uncertainty. Alternatively, some works predict a distribution over orientations in $\mathrm{SO}(3)$. However, training such models can be computation- and sample-inefficient. Instead, we propose a novel mapping of features from the image domain to the 3D rotation manifold. Our method then leverages $\mathrm{SO}(3)$ equivariant layers, which are more sample efficient, and outputs a distribution over rotations that can be sampled at arbitrary resolution. We demonstrate the effectiveness of our method at object orientation prediction, and achieve state-of-the-art performance on the popular PASCAL3D+ dataset. Moreover, we show that our method can model complex object symmetries, without any modifications to the parameters or loss function. Code is available at https://dmklee.github.io/image2sphere. △ Less

Submitted 27 February, 2023; originally announced February 2023.

Generative Adversarial Symmetry Discovery

Authors: Jianke Yang, Robin Walters, Nima Dehmamy, Rose Yu

Abstract: Despite the success of equivariant neural networks in scientific applications, they require knowing the symmetry group a priori. However, it may be difficult to know which symmetry to use as an inductive bias in practice. Enforcing the wrong symmetry could even hurt the performance. In this paper, we propose a framework, LieGAN, to automatically discover equivariances from a dataset using a paradi… ▽ More Despite the success of equivariant neural networks in scientific applications, they require knowing the symmetry group a priori. However, it may be difficult to know which symmetry to use as an inductive bias in practice. Enforcing the wrong symmetry could even hurt the performance. In this paper, we propose a framework, LieGAN, to automatically discover equivariances from a dataset using a paradigm akin to generative adversarial training. Specifically, a generator learns a group of transformations applied to the data, which preserve the original distribution and fool the discriminator. LieGAN represents symmetry as interpretable Lie algebra basis and can discover various symmetries such as the rotation group $\mathrm{SO}(n)$, restricted Lorentz group $\mathrm{SO}(1,3)^+$ in trajectory prediction and top-quark tagging tasks. The learned symmetry can also be readily used in several existing equivariant neural networks to improve accuracy and generalization in prediction. △ Less

Submitted 18 June, 2023; v1 submitted 31 January, 2023; originally announced February 2023.

On the Finkelberg-Ginzburg Mirabolic Monodromy Conjecture

Authors: Valerio Toledano-Laredo, Robin Walters

Abstract: We compute the monodromy of the mirabolic Harish-Chandra D-module for all values of the parameters (theta,c) in rank 1, and outside an explicit codimension 2 set of values in ranks 2 and higher. This shows in particular that the Finkelberg-Ginzburg conjecture, which is known to hold for generic values of (theta,c), fails at special values even in rank 1. Our main tools are Opdam's shift operators… ▽ More We compute the monodromy of the mirabolic Harish-Chandra D-module for all values of the parameters (theta,c) in rank 1, and outside an explicit codimension 2 set of values in ranks 2 and higher. This shows in particular that the Finkelberg-Ginzburg conjecture, which is known to hold for generic values of (theta,c), fails at special values even in rank 1. Our main tools are Opdam's shift operators and intertwiners for the extended affine Weyl group, which allow for the resolution of resonances outside the codimension two set. △ Less

Submitted 15 June, 2024; v1 submitted 27 December, 2022; originally announced December 2022.

Comments: Substantial revision. 36 pages, 7 figures

The prevalence and influence of circumstellar material around hydrogen-rich supernova progenitors

Authors: Rachel J. Bruch, Avishay Gal-Yam, Ofer Yaron, Ping Chen, Nora L. Strotjohann, Ido Irani, Erez Zimmerman, Steve Schulze, Yi Yang, Young-Lo Kim, Mattia Bulla, Jesper Sollerman, Mickael Rigault, Eran Ofek, Maayane Soumagnac, Frank J. Masci, Christoffer Fremling, Daniel Perley, Jakob Nordin, S. Bradley Cenko, Anna Y. Q. Ho, S. Adams, Igor Adreoni, Eric C. Bellm, Nadia Blagorodnova , et al. (22 additional authors not shown)

Abstract: Narrow transient emission lines (flash-ionization features) in early supernova (SN) spectra trace the presence of circumstellar material (CSM) around the massive progenitor stars of core-collapse SNe. The lines disappear within days after the SN explosion, suggesting that this material is spatially confined, and originates from enhanced mass loss shortly (months to a few years) prior to explosion.… ▽ More Narrow transient emission lines (flash-ionization features) in early supernova (SN) spectra trace the presence of circumstellar material (CSM) around the massive progenitor stars of core-collapse SNe. The lines disappear within days after the SN explosion, suggesting that this material is spatially confined, and originates from enhanced mass loss shortly (months to a few years) prior to explosion. We performed a systematic survey of H-rich (Type II) SNe discovered within less than two days from explosion during the first phase of the Zwicky Transient Facility (ZTF) survey (2018-2020), finding thirty events for which a first spectrum was obtained within $< 2$ days from explosion. The measured fraction of events showing flash ionisation features ($>36\%$ at $95\%$ confidence level) confirms that elevated mass loss in massive stars prior to SN explosion is common. We find that SNe II showing flash ionisation features are not significantly brighter, nor bluer, nor more slowly rising than those without. This implies that CSM interaction does not contribute significantly to their early continuum emission, and that the CSM is likely optically thin. We measured the persistence duration of flash ionisation emission and find that most SNe show flash features for $\approx 5 $ days. Rarer events, with persistence timescales $>10$ days, are brighter and rise longer, suggesting these may be intermediate between regular SNe II and strongly-interacting SNe IIn. △ Less

Submitted 13 December, 2022; v1 submitted 6 December, 2022; originally announced December 2022.

The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry

Authors: Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L. S. Wong, Robin Walters, Robert Platt

Abstract: Extensive work has demonstrated that equivariant neural networks can significantly improve sample efficiency and generalization by enforcing an inductive bias in the network architecture. These applications typically assume that the domain symmetry is fully described by explicit transformations of the model inputs and outputs. However, many real-life applications contain only latent or partial sym… ▽ More Extensive work has demonstrated that equivariant neural networks can significantly improve sample efficiency and generalization by enforcing an inductive bias in the network architecture. These applications typically assume that the domain symmetry is fully described by explicit transformations of the model inputs and outputs. However, many real-life applications contain only latent or partial symmetries which cannot be easily described by simple transformations of the input. In these cases, it is necessary to learn symmetry in the environment instead of imposing it mathematically on the network architecture. We discover, surprisingly, that imposing equivariance constraints that do not exactly match the domain symmetry is very helpful in learning the true symmetry in the environment. We differentiate between extrinsic and incorrect symmetry constraints and show that while imposing incorrect symmetry can impede the model's performance, imposing extrinsic symmetry can actually improve performance. We demonstrate that an equivariant model can significantly outperform non-equivariant methods on domains with latent symmetries both in supervised learning and in reinforcement learning for robotic manipulation and control problems. △ Less

Submitted 10 February, 2023; v1 submitted 16 November, 2022; originally announced November 2022.

Comments: Published at ICLR 2023, notable top 25% (Spotlight)

SEIL: Simulation-augmented Equivariant Imitation Learning

Authors: Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, Robert Platt

Abstract: In robotic manipulation, acquiring samples is extremely expensive because it often requires interacting with the real world. Traditional image-level data augmentation has shown the potential to improve sample efficiency in various machine learning tasks. However, image-level data augmentation is insufficient for an imitation learning agent to learn good manipulation policies in a reasonable amount… ▽ More In robotic manipulation, acquiring samples is extremely expensive because it often requires interacting with the real world. Traditional image-level data augmentation has shown the potential to improve sample efficiency in various machine learning tasks. However, image-level data augmentation is insufficient for an imitation learning agent to learn good manipulation policies in a reasonable amount of demonstrations. We propose Simulation-augmented Equivariant Imitation Learning (SEIL), a method that combines a novel data augmentation strategy of supplementing expert trajectories with simulated transitions and an equivariant model that exploits the $\mathrm{O}(2)$ symmetry in robotic manipulation. Experimental evaluations demonstrate that our method can learn non-trivial manipulation tasks within ten demonstrations and outperforms the baselines with a significant margin. △ Less

Submitted 31 October, 2022; originally announced November 2022.

Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detection

Authors: Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, Robert Platt

Abstract: Given point cloud input, the problem of 6-DoF grasp pose detection is to identify a set of hand poses in SE(3) from which an object can be successfully grasped. This important problem has many practical applications. Here we propose a novel method and neural network model that enables better grasp success rates relative to what is available in the literature. The method takes standard point cloud… ▽ More Given point cloud input, the problem of 6-DoF grasp pose detection is to identify a set of hand poses in SE(3) from which an object can be successfully grasped. This important problem has many practical applications. Here we propose a novel method and neural network model that enables better grasp success rates relative to what is available in the literature. The method takes standard point cloud data as input and works well with single-view point clouds observed from arbitrary viewing directions. △ Less

Submitted 31 October, 2022; originally announced November 2022.

Comments: https://haojhuang.github.io/edge_grasp_page/

Symmetries, flat minima, and the conserved quantities of gradient flow

Authors: Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, Nima Dehmamy

Abstract: Empirical studies of the loss landscape of deep networks have revealed that many local minima are connected through low-loss valleys. Yet, little is known about the theoretical origin of such valleys. We present a general framework for finding continuous symmetries in the parameter space, which carve out low-loss valleys. Our framework uses equivariances of the activation functions and can be appl… ▽ More Empirical studies of the loss landscape of deep networks have revealed that many local minima are connected through low-loss valleys. Yet, little is known about the theoretical origin of such valleys. We present a general framework for finding continuous symmetries in the parameter space, which carve out low-loss valleys. Our framework uses equivariances of the activation functions and can be applied to different layer architectures. To generalize this framework to nonlinear neural networks, we introduce a novel set of nonlinear, data-dependent symmetries. These symmetries can transform a trained model such that it performs similarly on new samples, which allows ensemble building that improves robustness under certain adversarial attacks. We then show that conserved quantities associated with linear symmetries can be used to define coordinates along low-loss valleys. The conserved quantities help reveal that using common initialization methods, gradient flow only explores a small part of the global minimum. By relating conserved quantities to convergence rate and sharpness of the minimum, we provide insights on how initialization impacts convergence and generalizability. △ Less

Submitted 23 March, 2023; v1 submitted 31 October, 2022; originally announced October 2022.

Comments: To appear at ICLR 2023

Secondary Protein Structure Prediction Using Neural Networks

Authors: Sidharth Malhotra, Robin Walters

Abstract: In this paper we experiment with using neural network structures to predict a protein's secondary structure (α helix positions) from only its primary structure (amino acid sequence). We implement a fully connected neural network (FCNN) and preform three experiments using that FCNN. Firstly, we do a cross-species comparison of models trained and tested on mouse and human datasets. Secondly, we test… ▽ More In this paper we experiment with using neural network structures to predict a protein's secondary structure (α helix positions) from only its primary structure (amino acid sequence). We implement a fully connected neural network (FCNN) and preform three experiments using that FCNN. Firstly, we do a cross-species comparison of models trained and tested on mouse and human datasets. Secondly, we test the impact of varying the length of protein sequence we input into the model. Thirdly, we compare custom error functions designed to focus on the center of the input window. At the end of paper we propose a alternative, recurrent neural network model which can be applied to the problem. △ Less

Submitted 23 August, 2022; originally announced August 2022.

Quiver neural networks

Authors: Iordan Ganev, Robin Walters

Abstract: We develop a uniform theoretical approach towards the analysis of various neural network connectivity architectures by introducing the notion of a quiver neural network. Inspired by quiver representation theory in mathematics, this approach gives a compact way to capture elaborate data flows in complex network architectures. As an application, we use parameter space symmetries to prove a lossless… ▽ More We develop a uniform theoretical approach towards the analysis of various neural network connectivity architectures by introducing the notion of a quiver neural network. Inspired by quiver representation theory in mathematics, this approach gives a compact way to capture elaborate data flows in complex network architectures. As an application, we use parameter space symmetries to prove a lossless model compression algorithm for quiver neural networks with certain non-pointwise activations known as rescaling activations. In the case of radial rescaling activations, we prove that training the compressed model with gradient descent is equivalent to training the original model with projected gradient descent. △ Less

Submitted 26 July, 2022; originally announced July 2022.

Comments: Preliminary version, comments welcome

Image to Icosahedral Projection for $\mathrm{SO}(3)$ Object Reasoning from Single-View Images

Authors: David Klee, Ondrej Biza, Robert Platt, Robin Walters

Abstract: Reasoning about 3D objects based on 2D images is challenging due to variations in appearance caused by viewing the object from different orientations. Tasks such as object classification are invariant to 3D rotations and other such as pose estimation are equivariant. However, imposing equivariance as a model constraint is typically not possible with 2D image input because we do not have an a prior… ▽ More Reasoning about 3D objects based on 2D images is challenging due to variations in appearance caused by viewing the object from different orientations. Tasks such as object classification are invariant to 3D rotations and other such as pose estimation are equivariant. However, imposing equivariance as a model constraint is typically not possible with 2D image input because we do not have an a priori model of how the image changes under out-of-plane object rotations. The only $\mathrm{SO}(3)$-equivariant models that currently exist require point cloud or voxel input rather than 2D images. In this paper, we propose a novel architecture based on icosahedral group convolutions that reasons in $\mathrm{SO(3)}$ by learning a projection of the input image onto an icosahedron. The resulting model is approximately equivariant to rotation in $\mathrm{SO}(3)$. We apply this model to object pose estimation and shape classification tasks and find that it outperforms reasonable baselines. Project website: \url{https://dmklee.github.io/image2icosahedral} △ Less

Submitted 15 November, 2022; v1 submitted 18 July, 2022; originally announced July 2022.

Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting

Authors: Rui Wang, Robin Walters, Rose Yu

Abstract: Exploiting symmetry in dynamical systems is a powerful way to improve the generalization of deep learning. The model learns to be invariant to transformation and hence is more robust to distribution shift. Data augmentation and equivariant networks are two major approaches to injecting symmetry into learning. However, their exact role in improving generalization is not well understood. In this wor… ▽ More Exploiting symmetry in dynamical systems is a powerful way to improve the generalization of deep learning. The model learns to be invariant to transformation and hence is more robust to distribution shift. Data augmentation and equivariant networks are two major approaches to injecting symmetry into learning. However, their exact role in improving generalization is not well understood. In this work, we derive the generalization bounds for data augmentation and equivariant networks, characterizing their effect on learning in a unified framework. Unlike most prior theories for the i.i.d. setting, we focus on non-stationary dynamics forecasting with complex temporal dependencies. △ Less

Submitted 19 June, 2022; originally announced June 2022.

Integrating Symmetry into Differentiable Planning with Steerable Convolutions

Authors: Linfeng Zhao, Xupeng Zhu, Lingzhi Kong, Robin Walters, Lawson L. S. Wong

Abstract: We study how group symmetry helps improve data efficiency and generalization for end-to-end differentiable planning algorithms when symmetry appears in decision-making tasks. Motivated by equivariant convolution networks, we treat the path planning problem as \textit{signals} over grids. We show that value iteration in this case is a linear equivariant operator, which is a (steerable) convolution.… ▽ More We study how group symmetry helps improve data efficiency and generalization for end-to-end differentiable planning algorithms when symmetry appears in decision-making tasks. Motivated by equivariant convolution networks, we treat the path planning problem as \textit{signals} over grids. We show that value iteration in this case is a linear equivariant operator, which is a (steerable) convolution. This extends Value Iteration Networks (VINs) on using convolutional networks for path planning with additional rotation and reflection symmetry. Our implementation is based on VINs and uses steerable convolution networks to incorporate symmetry. The experiments are performed on four tasks: 2D navigation, visual navigation, and 2 degrees of freedom (2DOFs) configuration space and workspace manipulation. Our symmetric planning algorithms improve training efficiency and generalization by large margins compared to non-equivariant counterparts, VIN and GPPN. △ Less

Submitted 1 May, 2023; v1 submitted 8 June, 2022; originally announced June 2022.

Comments: ICLR 2023 camera-ready version. Original name = "Integrating Symmetry into Differentiable Planning". Website: http://lfzhao.com/SymPlan

Topological Deep Learning: Going Beyond Graph Data

Authors: Mustafa Hajij, Ghada Zamzmi, Theodore Papamarkou, Nina Miolane, Aldo Guzmán-Sáenz, Karthikeyan Natesan Ramamurthy, Tolga Birdal, Tamal K. Dey, Soham Mukherjee, Shreyas N. Samaga, Neal Livesay, Robin Walters, Paul Rosen, Michael T. Schaub

Abstract: Topological deep learning is a rapidly growing field that pertains to the development of deep learning models for data supported on topological domains such as simplicial complexes, cell complexes, and hypergraphs, which generalize many domains encountered in scientific computations. In this paper, we present a unifying deep learning framework built upon a richer data structure that includes widel… ▽ More Topological deep learning is a rapidly growing field that pertains to the development of deep learning models for data supported on topological domains such as simplicial complexes, cell complexes, and hypergraphs, which generalize many domains encountered in scientific computations. In this paper, we present a unifying deep learning framework built upon a richer data structure that includes widely adopted topological domains. Specifically, we first introduce combinatorial complexes, a novel type of topological domain. Combinatorial complexes can be seen as generalizations of graphs that maintain certain desirable properties. Similar to hypergraphs, combinatorial complexes impose no constraints on the set of relations. In addition, combinatorial complexes permit the construction of hierarchical higher-order relations, analogous to those found in simplicial and cell complexes. Thus, combinatorial complexes generalize and combine useful traits of both hypergraphs and cell complexes, which have emerged as two promising abstractions that facilitate the generalization of graph neural networks to topological spaces. Second, building upon combinatorial complexes and their rich combinatorial and algebraic structure, we develop a general class of message-passing combinatorial complex neural networks (CCNNs), focusing primarily on attention-based CCNNs. We characterize permutation and orientation equivariances of CCNNs, and discuss pooling and unpooling operations within CCNNs in detail. Third, we evaluate the performance of CCNNs on tasks related to mesh shape analysis and graph learning. Our experiments demonstrate that CCNNs have competitive performance as compared to state-of-the-art deep learning models specifically tailored to the same tasks. Our findings demonstrate the advantages of incorporating higher-order relations into deep learning models in different applications. △ Less

Submitted 19 May, 2023; v1 submitted 1 June, 2022; originally announced June 2022.

Symmetry Teleportation for Accelerated Optimization

Authors: Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu

Abstract: Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We der… ▽ More Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove a necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad for several optimization problems including test functions, multi-layer regressions, and MNIST classification. △ Less

Submitted 4 January, 2023; v1 submitted 21 May, 2022; originally announced May 2022.

Comments: 20 pages, 8 figures, NeurIPS 2022

Probabilistic Symmetry for Multi-Agent Dynamics

Authors: Sophia Sun, Robin Walters, Jinxi Li, Rose Yu

Abstract: Learning multi-agent dynamics is a core AI problem with broad applications in robotics and autonomous driving. While most existing works focus on deterministic prediction, producing probabilistic forecasts to quantify uncertainty and assess risks is critical for downstream decision-making tasks such as motion planning and collision avoidance. Multi-agent dynamics often contains internal symmetry.… ▽ More Learning multi-agent dynamics is a core AI problem with broad applications in robotics and autonomous driving. While most existing works focus on deterministic prediction, producing probabilistic forecasts to quantify uncertainty and assess risks is critical for downstream decision-making tasks such as motion planning and collision avoidance. Multi-agent dynamics often contains internal symmetry. By leveraging symmetry, specifically rotation equivariance, we can improve not only the prediction accuracy but also uncertainty calibration. We introduce Energy Score, a proper scoring rule, to evaluate probabilistic predictions. We propose a novel deep dynamics model, Probabilistic Equivariant Continuous COnvolution (PECCO) for probabilistic prediction of multi-agent trajectories. PECCO extends equivariant continuous convolution to model the joint velocity distribution of multiple agents. It uses dynamics integration to propagate the uncertainty from velocity to position. On both synthetic and real-world datasets, PECCO shows significant improvements in accuracy and calibration compared to non-equivariant baselines. △ Less

Submitted 18 May, 2023; v1 submitted 4 May, 2022; originally announced May 2022.

Toward Compositional Generalization in Object-Oriented World Modeling

Authors: Linfeng Zhao, Lingzhi Kong, Robin Walters, Lawson L. S. Wong

Abstract: Compositional generalization is a critical ability in learning and decision-making. We focus on the setting of reinforcement learning in object-oriented environments to study compositional generalization in world modeling. We (1) formalize the compositional generalization problem with an algebraic approach and (2) study how a world model can achieve that. We introduce a conceptual environment, Obj… ▽ More Compositional generalization is a critical ability in learning and decision-making. We focus on the setting of reinforcement learning in object-oriented environments to study compositional generalization in world modeling. We (1) formalize the compositional generalization problem with an algebraic approach and (2) study how a world model can achieve that. We introduce a conceptual environment, Object Library, and two instances, and deploy a principled pipeline to measure the generalization ability. Motivated by the formulation, we analyze several methods with exact or no compositional generalization ability using our framework, and design a differentiable approach, Homomorphic Object-oriented World Model (HOWM), that achieves soft but more efficient compositional generalization. △ Less

Submitted 17 June, 2022; v1 submitted 28 April, 2022; originally announced April 2022.

Comments: ICML 2022 Long Presentation. Website: http://lfzhao.com/oowm/

Learning Symmetric Embeddings for Equivariant World Models

Authors: Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, Robin Walters

Abstract: Incorporating symmetries can lead to highly data-efficient and generalizable models by defining equivalence classes of data samples related by transformations. However, characterizing how transformations act on input data is often difficult, limiting the applicability of equivariant models. We propose learning symmetric embedding networks (SENs) that encode an input space (e.g. images), where we d… ▽ More Incorporating symmetries can lead to highly data-efficient and generalizable models by defining equivalence classes of data samples related by transformations. However, characterizing how transformations act on input data is often difficult, limiting the applicability of equivariant models. We propose learning symmetric embedding networks (SENs) that encode an input space (e.g. images), where we do not know the effect of transformations (e.g. rotations), to a feature space that transforms in a known manner under these operations. This network can be trained end-to-end with an equivariant task network to learn an explicitly symmetric representation. We validate this approach in the context of equivariant transition models with 3 distinct forms of symmetry. Our experiments demonstrate that SENs facilitate the application of equivariant networks to data with complex symmetry representations. Moreover, doing so can yield improvements in accuracy and generalization relative to both fully-equivariant and non-equivariant baselines. △ Less

Submitted 30 June, 2022; v1 submitted 24 April, 2022; originally announced April 2022.

Comments: ICML 2022

On-Robot Learning With Equivariant Models

Authors: Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, Robert Platt

Abstract: Recently, equivariant neural network models have been shown to improve sample efficiency for tasks in computer vision and reinforcement learning. This paper explores this idea in the context of on-robot policy learning in which a policy must be learned entirely on a physical robotic system without reference to a model, a simulator, or an offline dataset. We focus on applications of Equivariant SAC… ▽ More Recently, equivariant neural network models have been shown to improve sample efficiency for tasks in computer vision and reinforcement learning. This paper explores this idea in the context of on-robot policy learning in which a policy must be learned entirely on a physical robotic system without reference to a model, a simulator, or an offline dataset. We focus on applications of Equivariant SAC to robotic manipulation and explore a number of variations of the algorithm. Ultimately, we demonstrate the ability to learn several non-trivial manipulation tasks completely through on-robot experiences in less than an hour or two of wall clock time. △ Less

Submitted 17 October, 2022; v1 submitted 9 March, 2022; originally announced March 2022.

Comments: Published at CoRL 2022

$\mathrm{SO}(2)$-Equivariant Reinforcement Learning

Authors: Dian Wang, Robin Walters, Robert Platt

Abstract: Equivariant neural networks enforce symmetry within the structure of their convolutional layers, resulting in a substantial improvement in sample efficiency when learning an equivariant or invariant function. Such models are applicable to robotic manipulation learning which can often be formulated as a rotationally symmetric problem. This paper studies equivariant model architectures in the contex… ▽ More Equivariant neural networks enforce symmetry within the structure of their convolutional layers, resulting in a substantial improvement in sample efficiency when learning an equivariant or invariant function. Such models are applicable to robotic manipulation learning which can often be formulated as a rotationally symmetric problem. This paper studies equivariant model architectures in the context of $Q$-learning and actor-critic reinforcement learning. We identify equivariant and invariant characteristics of the optimal $Q$-function and the optimal policy and propose equivariant DQN and SAC algorithms that leverage this structure. We present experiments that demonstrate that our equivariant versions of DQN and SAC can be significantly more sample efficient than competing algorithms on an important class of robotic manipulation problems. △ Less

Submitted 8 March, 2022; originally announced March 2022.

Comments: Published at ICLR 2022

New Modules for the SEDMachine to Remove Contaminations from Cosmic Rays and Non-target Light: BYECR and CONTSEP

Authors: Y. -L. Kim, M. Rigault, J. D. Neill, M. Briday, Y. Copin, J. Lezmy, N. Nicolas, R. Riddle, Y. Sharma, M. Smith, J. Sollerman, R. Walters

Abstract: Currently time-domain astronomy can scan the entire sky on a daily basis, discovering thousands of interesting transients every night. Classifying the ever-increasing number of new transients is one of the main challenges for the astronomical community. One solution that addresses this issue is the robotically controlled Spectral Energy Distribution Machine (SEDM) which supports the Zwicky Transie… ▽ More Currently time-domain astronomy can scan the entire sky on a daily basis, discovering thousands of interesting transients every night. Classifying the ever-increasing number of new transients is one of the main challenges for the astronomical community. One solution that addresses this issue is the robotically controlled Spectral Energy Distribution Machine (SEDM) which supports the Zwicky Transient Facility (ZTF). SEDM with its pipeline PYSEDM demonstrates that real-time robotic spectroscopic classification is feasible. In an effort to improve the quality of the current SEDM data, we present here two new modules, BYECR and CONTSEP. The first removes contamination from cosmic rays, and the second removes contamination from non-target light. These new modules are part of the automated PYSEDM pipeline and fully integrated with the whole process. Employing BYECR and CONTSEP modules together automatically extracts more spectra than the current PYSEDM pipeline. Using SNID classification results, the new modules show an improvement in the classification rate and accuracy of 2.8% and 1.7%, respectively, while the strength of the cross-correlation remains the same. Improvements to the SEDM astrometry would further boost the improvement of the CONTSEP module. This kind of robotic follow-up with a fully automated pipeline has the potential to provide the spectroscopic classifications for the transients discovered by ZTF and also by the Rubin Observatory's Legacy Survey of Space and Time. △ Less

Submitted 2 March, 2022; originally announced March 2022.

Comments: 9 pages, 6 figures, 2 tables, accepted for publication in PASP

Journal ref: PASP, 134:024505 (9pp), 2022 February

Constraining Type Ia supernova explosions and early flux excesses with the Zwicky Transient Factory

Authors: M. Deckers, K. Maguire, M. R. Magee, G. Dimitriadis, M. Smith, A. Sainz de Murieta, A. A. Miller, A. Goobar, J. Nordin, M. Rigault, E. Bellm, M. W. Coughlin, R. R. Laher, D. Shupe, M. J. Graham, M. M. Kasliwal, R. Walters

Abstract: In the new era of time-domain surveys Type Ia supernovae are being caught sooner after explosion, which has exposed significant variation in their early light curves. Two driving factors for early time evolution are the distribution of nickel in the ejecta and the presence of flux excesses of various causes. We perform an analysis of the largest young SN Ia sample to date. We compare 115 SN Ia lig… ▽ More In the new era of time-domain surveys Type Ia supernovae are being caught sooner after explosion, which has exposed significant variation in their early light curves. Two driving factors for early time evolution are the distribution of nickel in the ejecta and the presence of flux excesses of various causes. We perform an analysis of the largest young SN Ia sample to date. We compare 115 SN Ia light curves from the Zwicky Transient Facility to the turtls model grid containing light curves of Chandrasekhar-mass explosions with a range of nickel masses, nickel distributions and explosion energies. We find that the majority of our observed light curves are well reproduced by Chandrasekhar-mass explosion models with a preference for highly extended nickel distributions. We identify six SNe Ia with an early-time flux excess in our g- and r-band data (four `blue' and two `red' flux excesses). We find an intrinsic rate of 18+/-11 per cent of early flux excesses in SNe Ia at z < 0.07, based on three detected flux excesses out of 30 (10 per cent) observed SNe Ia with a simulated efficiency of 57 per cent. This is comparable to rates of flux excesses in the literature but also accounts for detection efficiencies. Two of these events are mostly consistent with CSM interaction, while the other four have longer lifetimes in agreement with companion interaction and nickel-clump models. We find a higher frequency of flux excesses in 91T/99aa-like events (44+/-13 per cent). △ Less

Submitted 25 February, 2022; originally announced February 2022.

Compact Michelson interferometers with subpicometer sensitivity

Authors: Jiri Smetana, Rebecca Walters, Sophie Bauchinger, Amit Singh Ubhi, Sam Cooper, David Hoyland, Richard Abbott, Christoph Baune, Peter Fritchel, Oliver Gerberding, Semjon Köhnke, Haixing Miao, Sebastian Rode, Denis Martynov

Abstract: The network of interferometric gravitational-wave observatories has successfully detected tens of astrophysical signals since 2015. In this paper, we experimentally investigate compact sensors that have the potential to improve the sensitivity of gravitational-wave detectors to intermediate-mass black holes. We use only commercial components, such as sensing heads and lasers, to assemble the setup… ▽ More The network of interferometric gravitational-wave observatories has successfully detected tens of astrophysical signals since 2015. In this paper, we experimentally investigate compact sensors that have the potential to improve the sensitivity of gravitational-wave detectors to intermediate-mass black holes. We use only commercial components, such as sensing heads and lasers, to assemble the setup and demonstrate its subpicometer precision. The setup consists of a pair of Michelson interferferometers that use deep frequency modulation techniques to obtain a linear, relative displacement readout over multiple interference fringes. We implement a laser-frequency stabilisation scheme to achieve a sensitivity of 0.3\,$\text{pm} / \sqrt{\text{Hz}}$ above 0.1\,Hz. The device has also the potential to improve other experiments, such as torsion balances and commercial seismometers. △ Less

Submitted 4 October, 2022; v1 submitted 21 February, 2022; originally announced February 2022.

Comments: 7 pages, 3 figures

Sample Efficient Grasp Learning Using Equivariant Models

Authors: Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, Robert Platt

Abstract: In planar grasp detection, the goal is to learn a function from an image of a scene onto a set of feasible grasp poses in $\mathrm{SE}(2)$. In this paper, we recognize that the optimal grasp function is $\mathrm{SE}(2)$-equivariant and can be modeled using an equivariant convolutional neural network. As a result, we are able to significantly improve the sample efficiency of grasp learning, obtaini… ▽ More In planar grasp detection, the goal is to learn a function from an image of a scene onto a set of feasible grasp poses in $\mathrm{SE}(2)$. In this paper, we recognize that the optimal grasp function is $\mathrm{SE}(2)$-equivariant and can be modeled using an equivariant convolutional neural network. As a result, we are able to significantly improve the sample efficiency of grasp learning, obtaining a good approximation of the grasp function after only 600 grasp attempts. This is few enough that we can learn to grasp completely on a physical robot in about 1.5 hours. △ Less

Submitted 18 February, 2022; originally announced February 2022.

Equivariant Transporter Network

Authors: Haojie Huang, Dian Wang, Robin Walters, Robert Platt

Abstract: Transporter Net is a recently proposed framework for pick and place that is able to learn good manipulation policies from a very few expert demonstrations. A key reason why Transporter Net is so sample efficient is that the model incorporates rotational equivariance into the pick module, i.e. the model immediately generalizes learned pick knowledge to objects presented in different orientations. T… ▽ More Transporter Net is a recently proposed framework for pick and place that is able to learn good manipulation policies from a very few expert demonstrations. A key reason why Transporter Net is so sample efficient is that the model incorporates rotational equivariance into the pick module, i.e. the model immediately generalizes learned pick knowledge to objects presented in different orientations. This paper proposes a novel version of Transporter Net that is equivariant to both pick and place orientation. As a result, our model immediately generalizes place knowledge to different place orientations in addition to generalizing pick knowledge as before. Ultimately, our new model is more sample efficient and achieves better pick and place success rates than the baseline Transporter Net model. △ Less

Submitted 20 September, 2022; v1 submitted 18 February, 2022; originally announced February 2022.

Comments: Project Website: https://haojhuang.github.io/etp_page/

Journal ref: RSS 2022

Interactivity: the missing link between virtual reality technology and drug discovery pipelines

Authors: Rebecca K. Walters, Ella M. Gale, Jonathan Barnoud, David R. Glowacki, Adrian J. Mulholland

Abstract: The potential of virtual reality (VR) to contribute to drug design and development has been recognised for many years. Hardware and software developments now mean that this potential is beginning to be realised, and VR methods are being actively used in this sphere. A recent advance is to use VR not only to visualise and interact with molecular structures, but also to interact with molecular dynam… ▽ More The potential of virtual reality (VR) to contribute to drug design and development has been recognised for many years. Hardware and software developments now mean that this potential is beginning to be realised, and VR methods are being actively used in this sphere. A recent advance is to use VR not only to visualise and interact with molecular structures, but also to interact with molecular dynamics simulations of 'on the fly' (interactive molecular dynamics in VR, IMD-VR), which is useful not only for flexible docking but also to examine binding processes and conformational changes. iMD-VR has been shown to be useful for creating complexes of ligands bound to target proteins, e.g., recently applied to peptide inhibitors of the SARS-CoV-2 main protease. In this review, we use the term 'interactive VR' to refer to software where interactivity is an inherent part of the user VR experience e.g., in making structural modifications or interacting with a physically rigorous molecular dynamics (MD) simulation, as opposed to simply using VR controllers to rotate and translate the molecule for enhanced visualisation. Here, we describe these methods and their application to problems relevant to drug discovery, highlighting the possibilities that they offer in this arena. We suggest that the ease of viewing and manipulating molecular structures and dynamics, and the ability to modify structures on the fly (e.g., adding or deleting atoms) makes modern interactive VR a valuable tool to add to the armoury of drug development methods. △ Less

Submitted 8 February, 2022; originally announced February 2022.

Comments: 19 pages, 3 figures