Smart Factory: Advanced Industrial Robots from

Perception to Motion
YJ Lim Ronal George

0
Autonomous Robot Development with MATLAB & Simulink

Platform design tools for industrial

robots

Tools for autonomous industrial

robot applications

Deployment and hardware interface

with MATLAB and Simulink

Integrated workflows enabled

by MATLAB and Simulink

1
Autonomous Robot Development with MATLAB & Simulink

Platform design tools for industrial

robots

Tools for autonomous industrial

robot applications

Deployment and hardware interface

with MATLAB and Simulink

Integrated workflows enabled

by MATLAB and Simulink

2
Industry Trends – Advanced Robotics

Factories are:
• Incorporating more advanced systems
• Taking advantage of new technology
• Increasing connectivity between components
• Enhancing infrastructure to reach Industrial
IoT

Smart factories integrate autonomy and big data for

actionable insights

Connected, optimized, proactive, and

agile

3
 Smart Factory Concept: Asset, Operation, & Workforce Optimization

IoT
Factory Cloud + AI

Local 5G Operation
Big Data
Process Management・Quality Increase Management

Robot Optimization
✔?
✘?
OT
Worker AMR
Optimization

Autonomy Monitor
Update
4
 Smart Factory Concept: Asset, Operation, & Workforce Optimization
What we are talking about today

IoT
Factory Cloud + AI

Local 5G Operation
Big Data
Process Management・Quality Increase Management

Robot Optimization
✔?
✘?
OT
Worker AMR
Optimization

Autonomy Monitor
Update
5
Robotics in Smart Factory: Trends and Directions

Conventional

▪ Automated Systems
▪ Repetitive tasks
▪ Manual programming
▪ Safety fence needed

6
Robotics in Smart Factory: Trends and Directions

Conventional

▪ Automated Systems
▪ Repetitive tasks
▪ Manual programming
▪ Safety fence needed

7
Robotics in Smart Factory: Trends and Directions

Conventional Current
✔?
✘?

▪ Automated Systems ▪ Flexible Automation

▪ Repetitive tasks ▪ Collaborative robots
▪ Manual programming ▪ Advanced algorithm
▪ Safety fence needed ▪ Safety

8
Robotics in Smart Factory: Trends and Directions
✔?
Conventional
✘? Current
✔?
✘?

▪ Automated Systems ▪ Flexible Automation

▪ Repetitive tasks ▪ Collaborative robots
▪ Manual programming ▪ Advanced algorithm
▪ Safety fence needed ▪ Safety
MATLAB EXPO 2019 in Japan
Customer Presentation
9
Robotics in Smart Factory: Trends and Directions

Conventional Current Future

✔?
✘?

▪ Automated Systems ▪ Flexible Automation ▪ Autonomous Systems

▪ Repetitive tasks ▪ Collaborative robot ▪ Task cooperation
▪ Manual programming ▪ Advanced algorithm ▪ Intelligence sharing
▪ Safety fence needed ▪ Safety ▪ Adapting to the environment

10
Robotics in Smart Factory: Trends and Directions

Conventional Current Future

✔?
✘?

▪ Automated Systems ▪ Flexible Automation ▪ Autonomous Systems

▪ Repetitive tasks ▪ Collaborative robot ▪ Task cooperation
▪ Manual programming ▪ Advanced algorithm ▪ Intelligence sharing
▪ Safety fence needed ▪ Safety ▪ Adapting to the environment

11
Challenges: Growing Complexity

Design Complexity is the most commonly Complexity for software features is

challenge in the Aberdeen survey. Cited by 51% currently growing at double to triple the
in 2014, up from 27% in 2009 speed of software-development
productivity
Source: Aberdeen Group, April 2014 Source: McKinsey & Company, Feb 2020
12
Autonomous Robot Development with MATLAB & Simulink

Environment

Robot Model

Development Environment
Design Simulate Analyze Implement Test
13
Autonomous Robot Development with MATLAB & Simulink

Environment

Robot Model

Development Environment
Design Simulate Analyze Implement Test
14
 Physical & Kinematic Modeling

Creating a physical model

Simscape Multibody, Robotics System Toolbox 15

 Physical & Kinematic Modeling

Creating a physical model Automatic import from CAD Tools

CAD Model

Multibody
Model

Simscape Multibody, Robotics System Toolbox 16

 Physical & Kinematic Modeling

Creating a physical model Automatic import from CAD Tools URDF / Robot Library
%% Import robot from URDF
smimport('GEN3_FOR_URDF_ARM_V11_MOD.urdf');
CAD Model

Multibody
Model
%% Use robot library
Robot = loadrobot('abbYuMi’);
Show(robot);

Simscape Multibody, Robotics System Toolbox 17

Robot Simulation
Simulate First and Simulate Often!

Environment

Robot Model

Rigid Body
Low-Fidelity Multibody Co-Sim with Co-Sim with
Kinematics &
Model Modeling Gazebo Unreal
Dynamics
18
 Rigid Body
Fast & Low-Fidelity Simulation Low-Fidelity
Model
Kinematics
& Dynamics
Multibody
Modeling
Co-Sim with
Gazebo
Co-Sim with
Unreal

Simplified system model with a joint space motion model

System Model

Motion Models

Simscape Multibody, Robotics System Toolbox 19

 Rigid Body
Dynamic Simulation Low-Fidelity
Model
Kinematics
& Dynamics
Multibody
Modeling
Co-Sim with
Gazebo
Co-Sim with
Unreal

Manipulator dynamics model that accepts joint torques and gripper commands
System Model

Upgrade

Robotics System
Toolbox

Simscape Multibody, Robotics System Toolbox 20

 Rigid Body

Multibody Simulation Low-Fidelity

Model
Kinematics
& Dynamics
Multibody
Modeling
Co-Sim with
Gazebo
Co-Sim with
Unreal

Physical dynamics system with built-in joint limits and contact modeling
System Model

Upgrade

Simscape Multibody

Simscape Multibody, Robotics System Toolbox 21

 Rigid Body
Environment Modeling Low-Fidelity
Model
Kinematics
& Dynamics
Multibody
Modeling
Co-Sim with
Gazebo
Co-Sim with
Unreal

Co-simulation with third-party simulators for sensors and environment models

Simscape Multibody, Robotics System Toolbox, ROS Toolbox 22

 Rigid Body
Environment Modeling Low-Fidelity
Model
Kinematics
& Dynamics
Multibody
Modeling
Co-Sim with
Gazebo
Co-Sim with
Unreal

Co-simulation with third-party simulators for sensors and environment models

Gazebo Co-simulation with Time Synchronization

Simscape Multibody, Robotics System Toolbox, ROS Toolbox 23

 Rigid Body
Environment Modeling Low-Fidelity
Model
Kinematics
& Dynamics
Multibody
Modeling
Co-Sim with
Gazebo
Co-Sim with
Unreal

Co-simulation with third-party simulators for sensors and environment models

Gazebo Co-simulation with Time Synchronization

Simscape Multibody, Robotics System Toolbox, ROS Toolbox 24

Autonomous Robot Development with MATLAB & Simulink

Environment

Robot Model

Development Environment
Design Simulate Analyze Implement Test
25
How is deep learning used for robotics?

❑Speech Recognition
Audio Deep Learning Text

Voice Command to
Robots

26
How is deep learning used for robotics?

❑Speech Recognition
Audio Deep Learning Text Voice Command to
Robots

❑Computer Vision
Pixels Deep Learning Labels

Object Identification for

Robots

27
How is deep learning used for robotics?

❑Speech Recognition
Audio Deep Learning Text Voice Command to
Robots

❑Computer Vision
Labels Object Identification for
Pixels Deep Learning
Robots

Image
Pixels Deep Learning
Abnormal

Industrial Inspection
with Robots / UAVs

28
How is deep learning used for robotics?

❑Speech Recognition
Audio Deep Learning Text Voice Command to
Robots

❑Computer Vision
Labels Object Identification for
Pixels Deep Learning
Robots

Image Industrial Inspection

Pixels Deep Learning
Abnormal with Robots / UAVs

Pose Estimation
Localization
for Robots
❑Point Cloud Processing
Point
Deep Learning Structure
Clouds
29
How is deep learning used for robotics?

❑Speech Recognition
Audio Deep Learning Text Voice Command to
Robots

❑Computer Vision
Labels Object Identification for
Pixels Deep Learning
Robots

Image Industrial Inspection

Pixels Deep Learning
Abnormal with Robots / UAVs

❑Point Cloud Processing

Point Pose Estimation
Deep Learning Structure Localization
Clouds for Robots
30
 Perception
Deep learning for object classification

•
•
•

Computer Vision Toolbox, Deep Learning Toolbox 31

 Perception
Deep learning for object classification

•
•
•
Automated Labeling &
Iterative Learning

Computer Vision Toolbox, Deep Learning Toolbox 32

 Perception
Deep learning for object classification

•
•
•
Automated Labeling &
Iterative Learning

Object detector using Deep

Learning (YOLO v2)

Computer Vision Toolbox, Deep Learning Toolbox 33

 Perception
Object Classification

34
Computer Vision Toolbox, Deep Learning Toolbox, Robotics System Toolbox, Stateflow 34
 Full Workflow of Pick-and-Place Robots

Robotics System Toolbox, Computer Vision Toolbox, ROS Toolbox, Image Processing Toolbox, Stateflow 35
Full Workflow of Pick-and-Place Robots

• Dynamic environment
• Flexible operations

Scan‒and‒Build Environment

36
Full Workflow of Pick-and-Place Robots

• Dynamic environment
• Flexible operations

Scan‒and‒Build Environment

1 2

37
Full Workflow of Pick-and-Place Robots

Shipping examples

• Dynamic environment
• Flexible operations

Scan‒and‒Build Environment

1 2 3

38
Full Workflow of Pick-and-Place Robots

1 2 3

39
Full Workflow of Pick-and-Place Robots

1 2 3

40
Full Workflow of Pick-and-Place Robots

41
42
 Advanced Control: Reinforcement Learning
Train robot to reach the target object

AGENT
ACTION
STATE
Policy
Train to reach the target object
Policy update
Reinforcement
Learning
Algorithm

REWARD

ENVIRONMENT

Reinforcement Learning Toolbox, Simscape Multibody 43

 Advanced Control: Reinforcement Learning
Train robot to reach the target object

AGENT
ACTION
STATE
Policy
Train to reach the target object Train to grasp the target object
Policy update
Reinforcement
Learning
Algorithm

REWARD

ENVIRONMENT

Reinforcement Learning Toolbox, Simscape Multibody 44

Autonomous Robot Development with MATLAB & Simulink

Environment

Robot Model

Development Environment
Design Simulate Analyze Implement Test
45
 Hardware Connectivity & Deployment

Kinova Gen 3 Franka Emika Panda

MATLAB APIs

Robotics System Toolbox, ROS Toolbox, Coders 46

 Hardware Connectivity & Deployment

Kinova Gen 3 Franka Emika Panda

MATLAB APIs

Robotics System Toolbox, ROS Toolbox, Coders 47

 Hardware Connectivity & Deployment

Kinova Gen 3 Franka Emika Panda

MATLAB APIs

Sense Sense Simula

1 2 tor

Middleware

Act1 HMI1 HMI2

Application
Robotics System Toolbox, ROS Toolbox, Coders 48
 Hardware Connectivity

MATLAB API

Environment

Robot Model
ROS Driver

Robotics System Toolbox, ROS Toolbox 49

Autonomous Robot Development with MATLAB & Simulink

Environment

Robot Model

Development Environment
Simulink
50
Autonomous Robot Development for Smart Factories
✔?
✘?

Simulink

Develop Autonomous Applications from Perception to Motion Planning

and Optimize System-Level Behavior ​
Supported by MATLAB/Simulink Unified Environment
51
Use Cases: Warehouse Pick-and-Place Robot

52
Use Cases: Delta Robot for Automated Parts Sorting
Connect

Sense

Perceive

Plan &
Control
Platform Decide

53
Key Takeaways

Advanced robotics systems in the factory of the future

 Collaborative Robots
 AI-enabled Robots

Three pillars to develop autonomous robotics systems

 Platform design with environment models
 Autonomous application design
 Deployment

MATLAB and Simulink is a unified development environment

 Develop autonomous applications from perception to motion planning
 Simulate and optimize system-level behavior

54
Learn More

MathWorks Robotics Solution Page Awesome-MATLAB-Robotics

Robotics Examples
mathworks.com/robotics GitHub Repo

55
Thank you
Contact:

YJ Lim Ronal George

[email protected] [email protected]

© 2021 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks
for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders.

56