Jacky Liang

Jacky Liang, Oliver Kroemer

Detecting and localizing contacts is essential for robot manipulators to perform contact-rich tasks in unstructured environments. While robot skins can localize contacts on the surface of robot arms, these sensors are not yet robust or easily accessible. As such, prior works have explored using proprioceptive observations, such as joint velocities and torques, to perform contact localization. Many past approaches assume the robot is static during contact… Show more

Jacky Liang, Oliver Kroemer

Detecting and localizing contacts is essential for robot manipulators to perform contact-rich tasks in unstructured environments. While robot skins can localize contacts on the surface of robot arms, these sensors are not yet robust or easily accessible. As such, prior works have explored using proprioceptive observations, such as joint velocities and torques, to perform contact localization. Many past approaches assume the robot is static during contact incident, a single contact is made at a time, or having access to accurate dynamics models and joint torque sensing. In this work, we relax these assumptions and propose using Domain Randomization to train a neural network to localize contacts of robot arms in motion without joint torque observations. Our method uses a novel cylindrical projection encoding of the robot arm surface, which allows the network to use convolution layers to process input features and transposed convolution layers to predict contacts. The trained network achieves a contact detection accuracy of 91.5% and a mean contact localization error of 3.0cm. We further demonstrate an application of the contact localization model in an obstacle mapping task, evaluated in both simulation and the real world. Show less

Kevin Zhang, Mohit Sharma, Jacky Liang, Oliver Kroemer

We designed a modular robotic control stack that provides a customizable and accessible interface to the Franka Emika Panda Research robot. This framework abstracts high-level robot control commands as skills, which are decomposed into combinations of trajectory generators, feedback controllers, and termination handlers. Low-level control is implemented in C++ and runs at 1kHz, and high-level commands are exposed in Python. In… Show more

Kevin Zhang, Mohit Sharma, Jacky Liang, Oliver Kroemer

We designed a modular robotic control stack that provides a customizable and accessible interface to the Franka Emika Panda Research robot. This framework abstracts high-level robot control commands as skills, which are decomposed into combinations of trajectory generators, feedback controllers, and termination handlers. Low-level control is implemented in C++ and runs at 1kHz, and high-level commands are exposed in Python. In addition, external sensor feedback, like estimated object poses, can be streamed to the low-level controllers in real time. This modular approach allows us to quickly prototype new control methods, which is essential for research applications. We have applied this framework across a variety of real-world robot tasks in more than 5 published research papers. The framework is currently shared internally with other robotics labs at Carnegie Mellon University, and we plan for a public release in the near future. Show less

Mohit Sharma*, Jacky Liang*, Jialiang (Alan) Zhao, Alex LaGrassa, Oliver Kroemer

*Equal Contribution

Manipulation tasks can often be decomposed into multiple subtasks performed in parallel, e.g., sliding an object to a goal pose while maintaining contact with a table. Individual subtasks can be achieved by task-axis controllers defined relative to the objects being manipulated, and a set of object-centric controllers can be combined in an hierarchy. In prior works, such… Show more

Mohit Sharma*, Jacky Liang*, Jialiang (Alan) Zhao, Alex LaGrassa, Oliver Kroemer

*Equal Contribution

Manipulation tasks can often be decomposed into multiple subtasks performed in parallel, e.g., sliding an object to a goal pose while maintaining contact with a table. Individual subtasks can be achieved by task-axis controllers defined relative to the objects being manipulated, and a set of object-centric controllers can be combined in an hierarchy. In prior works, such combinations are defined manually or learned from demonstrations. By contrast, we propose using reinforcement learning to dynamically compose hierarchical object-centric controllers for manipulation tasks. Experiments in both simulation and real world show how the proposed approach leads to improved sample efficiency, zero-shot generalization to novel test environments, and simulation-to-reality transfer without fine-tuning. Show less

Jacky Liang, Saumya Saxena, Oliver Kroemer

Training robotic policies in simulation suffers from the sim-to-real gap, as simulated dynamics can be different from real-world dynamics. Past works tackled this problem through domain randomization and online system-identification. The former is sensitive to the manually-specified training distribution of dynamics parameters and can result in behaviors that are overly conservative. The latter requires learning policies that concurrently… Show more

Jacky Liang, Saumya Saxena, Oliver Kroemer

Training robotic policies in simulation suffers from the sim-to-real gap, as simulated dynamics can be different from real-world dynamics. Past works tackled this problem through domain randomization and online system-identification. The former is sensitive to the manually-specified training distribution of dynamics parameters and can result in behaviors that are overly conservative. The latter requires learning policies that concurrently perform the task and generate useful trajectories for system identification. In this work, we propose and analyze a framework for learning exploration policies that explicitly perform task-oriented exploration actions to identify task-relevant system parameters. These parameters are then used by model-based trajectory optimization algorithms to perform the task in the real world. We instantiate the framework in simulation with the Linear Quadratic Regulator as well as in the real world with pouring and object dragging tasks. Experiments show that task-oriented exploration helps model-based policies adapt to systems with initially unknown parameters, and it leads to better task performance than task-agnostic exploration. Show less

Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox

Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision… Show more

Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox

Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex manipulation tasks that go beyond simple pick-and-place operations. This allows for collection of high dimensional, multi-modality, state-action data that can be leveraged in the future to learn sensorimotor policies for challenging manipulation tasks. The system performance was measured through speed and reliability metrics across two human demonstrators on a variety of tasks. Show less

Jacky Liang, Ankur Handa, Karl Van Wyk, Viktor Makoviychuk, Oliver Kroemer, Dieter Fox

Tracking the pose of an object while it is being held and manipulated by a robot hand is difficult for vision-based methods due to significant occlusions. Prior works have explored using contact feedback and particle filters to localize in-hand objects. However, they have mostly focused on the static grasp setting and not when the object is in motion, as doing so requires modeling of complex contact… Show more

Jacky Liang, Ankur Handa, Karl Van Wyk, Viktor Makoviychuk, Oliver Kroemer, Dieter Fox

Tracking the pose of an object while it is being held and manipulated by a robot hand is difficult for vision-based methods due to significant occlusions. Prior works have explored using contact feedback and particle filters to localize in-hand objects. However, they have mostly focused on the static grasp setting and not when the object is in motion, as doing so requires modeling of complex contact dynamics. In this work, we propose using GPU-accelerated parallel robot simulations and derivative-free, sample-based optimizers to track in-hand object poses with contact feedback during manipulation. We use physics simulation as the forward model for robot-object interactions, and the algorithm jointly optimizes for the state and the parameters of the simulations, so they better match with those of the real world. Our method runs in real-time (30Hz) on a single GPU, and it achieves an average point cloud distance error of 6mm in simulation experiments and 13mm in the real-world ones. Show less

Nathan Zhang*, Jacky Liang*, Amanda Tomlinson*, Frank Boensch, Anant Sahai

Many first-year undergraduate students do not have sufficient breadth of technical knowledge about subjects in Electrical Engineering (EE) and Computer Science (CS) to make informed choices toward their education. By the time students are exposed to subjects they may be interested in, the cost of switching areas of focus may be too high. With undergraduate enrollment in CS more than doubling in the past decade… Show more

Nathan Zhang*, Jacky Liang*, Amanda Tomlinson*, Frank Boensch, Anant Sahai

Many first-year undergraduate students do not have sufficient breadth of technical knowledge about subjects in Electrical Engineering (EE) and Computer Science (CS) to make informed choices toward their education. By the time students are exposed to subjects they may be interested in, the cost of switching areas of focus may be too high. With undergraduate enrollment in CS more than doubling in the past decade, many institutions lack adequate staff and infrastructure to address students' needs. To help newly enrolled students make better decisions with limited departmental resources, we present a first-semester, low-overhead survey course that covers a wide variety of topics. The class has been offered for five consecutive semesters by upper-class undergraduate volunteers with minimal faculty involvement. We report the format, content, and student feedback for the course. Our results suggest that such a class can provide new students with valuable guidance and better prepare them for an education in CS and EE. Show less

Austin S. Wang, Wuming Zhang, Daniel Troniak, Jacky Liang, Oliver Kroemer

We propose a visual servoing framework for learn- ing to improve grasps of objects. RGB and depth images from grasp attempts are collected using an automated data collection process. The data is then used to train a Grasp Quality Network (GQN) that predicts the outcome of grasps from visual information. A grasp optimization pipeline uses homography models with the trained network to optimize the grasp success rate.… Show more

Austin S. Wang, Wuming Zhang, Daniel Troniak, Jacky Liang, Oliver Kroemer

We propose a visual servoing framework for learn- ing to improve grasps of objects. RGB and depth images from grasp attempts are collected using an automated data collection process. The data is then used to train a Grasp Quality Network (GQN) that predicts the outcome of grasps from visual information. A grasp optimization pipeline uses homography models with the trained network to optimize the grasp success rate. We evaluate and compare several algorithms for adjusting the current gripper pose based on the current observation from a gripper-mounted camera to perform visual servoing. Evaluations in both simulated and hardware environments show considerable improvement in grasp robustness with models trained using less than 30K grasp trials. Success rates for grasping novel objects unseen during training increased from 18.5% to 81.0% in simulation, and from 17.8% to 78.0% in the real world. Show less

Jialiang Zhao, Jacky Liang, Oliver Kroemer

Precise robotic grasping is important for many industrial applications, such as assembly and palletizing, where the location of the object needs to be controlled and known. However, achieving precise grasps is challenging due to noise in sensing and control, as well as unknown object properties. We propose a method to plan robotic grasps that are both robust and precise by training two convolutional neural networks – one to predict the… Show more

Jialiang Zhao, Jacky Liang, Oliver Kroemer

Precise robotic grasping is important for many industrial applications, such as assembly and palletizing, where the location of the object needs to be controlled and known. However, achieving precise grasps is challenging due to noise in sensing and control, as well as unknown object properties. We propose a method to plan robotic grasps that are both robust and precise by training two convolutional neural networks – one to predict the robustness of a grasp and another to predict a distribution of post-grasp object displacements. Our networks are trained with depth images in simulation on a dataset of over 1000 industrial parts and were successfully deployed on a real robot without having to be further fine-tuned. The proposed displacement estimator achieves a mean prediction errors of 0.68cm and 3.42deg on novel objects in real world experiments. Show less

Viktor Makoviychuk*, Ankur Handa*, Nuttapong Chentanez, Miles Macklin, Dieter Fox

Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA… Show more

Viktor Makoviychuk*, Ankur Handa*, Nuttapong Chentanez, Miles Macklin, Dieter Fox

Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train for more challenging locomotion tasks.

* Equal Contribution Show less

Jacky Liang, Jeffrey Mahler, Michael Laskey, Pusong Li, Ken Goldberg

Deep Learning from Demonstrations (Deep LfD) is a promising approach for robots to perform bilateral automation tasks, such as tasks involving dynamic contact and deformation, where dynamics are difficult to model explicitly. Deep LfD methods typically require substantial datasets of 1) videos of humans which do not match robot kinematics and capabilities or 2) waypoints collected with tedious move-and-record interfaces… Show more

Jacky Liang, Jeffrey Mahler, Michael Laskey, Pusong Li, Ken Goldberg

Deep Learning from Demonstrations (Deep LfD) is a promising approach for robots to perform bilateral automation tasks, such as tasks involving dynamic contact and deformation, where dynamics are difficult to model explicitly. Deep LfD methods typically require substantial datasets of 1) videos of humans which do not match robot kinematics and capabilities or 2) waypoints collected with tedious move-and-record interfaces such as teaching pendants or kinesthetic teaching. We explore an alternative using the Intuitive Surgical da Vinci, where a pair of gravity-balanced high-precision passive 6-DOF master arms is combined with stereo vision allowing humans to perform precise surgical automation tasks with slave arms. We present DY-Teleop, an interface between the da Vinci master manipulators and an ABB YuMi industrial robot to facilitate the collection of time-synchronized images and robot states for deep learning of automation tasks involving deformation and dynamic contact. We also present YuMiPy, an open source library and ROS package for controlling an ABB YuMi over Ethernet. Experiments with scooping a ball into a cup, pipetting liquid between two containers, and untying a knot in a rope suggest that demonstrations obtained using DY-Teleop are 1.8X as effective for LfD than those obtained using kinesthetic teaching. Show less

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg

To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table.We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network… Show more

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg

To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table.We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8s with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive
out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net.
Show less

Menglong Guo, David V. Gealy, Jacky Liang, Jeffrey Mahler, Aimee Goncalves, Stephen McKinley, Juan Aparicio Ojea, Ken Goldberg

Parallel-jaw robot grippers can grasp almost any object and are ubiquitous in industry. Although the shape, texture, and compliance of gripper jaw surfaces affect grasp robustness, almost all commercially available grippers provide a pair of rectangular, planar, rigid jaw surfaces. Practitioners often modify these surfaces with a variety of ad-hoc methods such as… Show more

Menglong Guo, David V. Gealy, Jacky Liang, Jeffrey Mahler, Aimee Goncalves, Stephen McKinley, Juan Aparicio Ojea, Ken Goldberg

Parallel-jaw robot grippers can grasp almost any object and are ubiquitous in industry. Although the shape, texture, and compliance of gripper jaw surfaces affect grasp robustness, almost all commercially available grippers provide a pair of rectangular, planar, rigid jaw surfaces. Practitioners often modify these surfaces with a variety of ad-hoc methods such as adding rubber caps and/or wrapping with textured tape. This paper explores data-driven optimization of gripper jaw surfaces over a design space based on shape, texture, and compliance using rapid prototyping. In total, 37 jaw surface design variations were created using 3D printed casting molds and silicon rubber. The designs were evaluated with 1377 physical grasp experiments using a 4-axis robot (with automated reset). These tests evaluate grasp robustness as the probability that the jaws will acquire, lift, and hold a training set of objects at nominal grasp configurations computed by Dex-Net 1.0. Hill-climbing in parameter space yielded a grid
pattern of 0.03 inch void depth and 0.0375 inch void width on a silicone polymer with durometer of A30. We then evaluated performance of this design using an ABB YuMi robot grasping a set of eight difficult-to-grasp 3D printed objects in 80 grasps with four gripper surfaces. The factory-provided gripper tips
succeeded in 28.7% of the 80 trials, increasing to 68.7% when the tips were wrapped with tape. Gripper tips with geckoinspired surfaces succeeded in 80.0% of trials, and gripper tips with the designed silicone surfaces succeeded in 93.7% of trials. Show less