ROS AND UNITY

A COMPREHENSIVE INTRODUCTION
Danial Bagheri
Supervisor: Sebastian Starke

1
Content:
Ø ROS Ø Unity Introduction
§ Introduction
§ What uses ROS at the moment? Ø Unity with ROS
§ Peripheral units
§ What make ROS outstanding? Ø Stand alone Unity
§ Core modules
§ Standard Message Definitions
§ Robot Geometry Library
§ ROS visualizer (RVIZ)
§ Robot Description Language (URDF)
§ Side modules
§ GAZEBO
§ MoveIt
§ OpenCV
§ Arduino
§ How ROS works
§ Publisher- subscriber Topic-
Message
§ Service-client 2
ROS : Introduction[1] [1]

Ø ROS stands for Robot Operating System . Collection of tools, libraries, and conventions
to simplify the task of creating robot across a wide variety of robotic platforms. [1]
Ø Stablishing and controlling communication between peripheral modules of a robot :
sensors, cameras , physical fingers and etc. [1]
Ø ROS started at Stanford Artificial Intelligence Lab then further developed at Willow
Garage. [2]
Ø ROS is fully functional on Ubuntu and partially functional on other OS like Windows or
[3]
Mac[5]
Ø ROS is open source Therefore[5]:
§ It is free
§ There is a large community of contributors. You can be one of them.
http://wiki.ros.org/ [4]

[1 - Powering the world’s Robots- ROS.ORG- http://www.ros.org/] [5]

[2 - Powering the world’s Robots- ROS.ORG - History http://www.ros.org/history]
[3 - Willow Garage http://www.willowgarage.com/pages/software/ros-platform]
[4 - Ubuntu - The Ubuntu stacked logo http://design.ubuntu.com/brand/ubuntu-logo]
[5 – Wiki.ros.org - Introduction- http://wiki.ros.org/ROS/Introduction]

3
What uses ROS at the moment? [1]

Ø Almost all robots you have seen in Academic and to some extend in industry.
Ø Humanoid Robots : Nao®, GeRo®, Robonaut 2, ROBOTIS Thormang3, REEM® , …
Ø Manipulators: Barrett WAM ®, Baxter®, …

[6]
[3] [4] [5]
[2]
[7]
Ø Multi-fingered graspers : BarrettHand® , shadowHand, ..
Ø Intelligent vehicles : quadrotor helicopters, Autonomous cars , …
[1 - Powering the world’s Robots- ROS.ORG – Robots - http://wiki.ros.org/Robots]
[2 – Pullman Academic – Baxter robot- http://www.pullmanacademic.com.au/Products_Robotics_Baxter.html]
[3 – Robotnic – BARRETT WAM - http://www.robotnik.eu/robotics-arms/barrett-wam/] [8]
[4 - ROS Spotlight: Pal Robotics' REEM-C http://www.ros.org/news/2013/12/ros-spotlight-pal-robotics-reem-c.html]
[5 – German robot - Opensource humanoid robot http://www.german-robot.com/]
[6 – Generation Robotics –NAO - https://www.generationrobots.com/en/401617-programmable-humanoid-
nao-evolution-robot-red.html]
[7 –Shadow Robot Company – Shadowhand -https://www.shadowrobot.com/products/dexterous-hand]
[8 – Barrett Technologies - http://www.barrett.com/products-hand.htm] 4
Peripheral units [1]

Ø 1D range finders : TeraRanger, Sharp IR range finder

Ø 2D range finders : SICK LMS2xx lasers, Leuze rotoScan lase [2]

Ø 3D Sensors : DUO3D™ stereo camera, Kinect, PMD Camcube 3.0, …

[3] [4]
Ø Cameras : USB Cameras , Ethernet camera, ….
Ø Force/Torque/Touch Sensors: ATI f/t sensors, Nano17 6-axis, …
Ø Motion Capture: OptiTrack, VICON, LEAP Motion , …
[6]
Ø Pose Estimation (GPS/IMU) : BOSCH® IMU, Razor's® IMU, … [5]
Ø RFID : UHF RFID Reader
[7]
[1 - Powering the world’s Robots- ROS.ORG – Sensors - http://wiki.ros.org/Sensors]
[2 – TeraRanger One - http://www.teraranger.com/product/teraranger-one-distance-sensor-for-drones-and-robotics/]
[3 – Drexel University- SICK LMS200 tutorial - http://www.pages.drexel.edu/~kws23/tutorials/sick/sick.html
[4 - Digital-circuitry : SICK LMS-200 / LMS-291 LIDAR LASER SCANNER RS-232 INTERFACING WITH UBUNTU &
R.O.S. - http://www.digital-circuitry.com/Wordpress/sick-lms-200-lidar-laser-scanner-interfacing-with-ubuntu/]

[5 – Microsoft –Kinect for Xbox 360 - http://www.xbox.com/en-US/xbox 360/accessories/kinect]

[6 – Bosch – Mobility sensors SMI130 SMG130 SMA 130- http://www.bosch-
semiconductors.de/en/automotive_electronics/news_4/ces/ces_1.html]
[7 – 9 Degrees of Freedom - Razor IMU - https://www.sparkfun.com/products/retired/10736]
[8 – ATI Industrial Automation - Multi-Axis Force / Torque Sensors- http://www.ati-ia.com/products/ft/sensors.aspx] [8]
5
What make ROS outstanding?
Ø ROS is completely modular :
Ø Packages : A collection of Nodes, Messages , services.
§ Nodes: a process that uses ROS framework
§ Messages: Standard definition for passing information between nodes.
Ø Stack: Set of multiple package

Ø ROS is multi-language:
Ø C++ : full functionality with ROSCPP library
Ø Python : full functionality with ROSPY library
Ø JAVA, LISP, Octave, LUA : experimental development.

Ø Large set of tools out of box :Standard Robot Messages, Robot Description
Language, pose estimation, localization in a map, building a map, and even mobile
navigation.
Ø Integration with other libraries for: Simulation, Image processing and etc.

6
Powerful ROS libraries
Ø Standard Message Definitions Ø Example of a message structure:
For Each peripheral module or concept § Package : sensor_msgs
§ Message Type : Imu
code compatibility with all other part of the
robotic eco system. std_msgs/Header header
geometry_msgs/Quaternion orientation
categorized by types in different packages. float64[9] orientation_ covariance
geometry_msgs/Vector3 angular_velocity
Package : geometry_msgs float64[9] angular_veloci ty_covariance
§ Message Types available in this package: geometry_msgs/Vector3 linear_acceleration
§ Point float64[9] linear_acceleration_covariance
§ Pose
§ Transform
§ …

7
 Powerful ROS libraries
Ø Robot Geometry Library Ø TF listeners listen to the frames and
This is essential to keep track of position of each part of robot , provides a Tree which describes how
regarding to the other parts. where is the hand, in respect to coordinate systems are related to each
the head ? Where is robot1 regarding to the hand of robot2 ? other.

§ Transform library (TF) is a core library of ROS and provides

a coordinate tracking system.

§ TF is not a centralized library [1]

§ works base on publisher/subscriber messaging system of /map [2]

ROS.

Every node has :

/robot1/odom /robot2/odom
§ Publisher (user needs to write)
§ Listener (user needs to write)
/robot1/base /robot2/base

[1 - ROS.ORG – Robot Geometry Library - http://www.ros.org/core-components/]

[2 – TF ROS tutorial - https://www.youtube.com/watch?v=Xf25dVrG5ks] /robot1/laser /robot1/camera /robot2/laser /robot2/camera

8
Powerful ROS libraries
Ø ROS visualizer (RVIZ)
§ RVIZ is the default 3D visualization tool for.
§ RVIZ is not a ”simulator”.
§ RVIZ can show data that it has a plugin for
displaying (DisplayTypes ) and has been
published by nodes:
§ Axes : Displays a set of Axes
§ Camera: Creates a new rendering window
from the perspective of a camera
§ Map : Displays a map on the ground plane
§ Pose : Draws a pose as an arrow or axes.
§ ….. [Powering the world’s Robots- ROS.ORG - RVIZ Camera type
§ Complete set: http://wiki.ros.org/rviz/DisplayTypes/Camera]

http://wiki.ros.org/rviz/DisplayTypes

§ Each DisplayType uses specific message.

Axes => sensor_msgs/JointStates

9
 Powerful ROS libraries
Ø Robot Description Language (URDF) Ø Example of an URDF file:
Describe a robot in a machine readable format.
URDF is an XML file describing following physical properties: <?xml version="1.0"?>
§ Main parts: cylinder, box, length, radius, … <robot name="multipleshapes">
<link name="base_link">
§ Joints : continuous joints, prismatic joint, planar joint, Joint Dynamics
<visual>
(friction, damping) , Inertia <geometry>
<cylinder length="0.6" radius="0.2"/>
Used by different tools for simulation, visualization and motion planning: </geometry>
§ Rviz </visual>
§ Gazebo </link>
§ Moveit <link name="right_leg">
<visual>
§ Stage <geometry>
<box size="0.6 .1 .2"/>
</geometry>
</visual>
</link>
<joint name="base_to_right_leg" type="fixed">
<parent link="base_link"/>
<child link="right_leg"/>
</joint>
[ROS.ORG - Building a Visual Robot Model with URDF from Scratch
http://wiki.ros.org/urdf/Tutorials/Building a Visual Robot Model with URDF from Scratch </robot>
10
Powerful ROS 3rd party tools
Ø GAZEBO § URDF in Gazebo : URDF describes kinematic
and dynamic properties of a robot.
§ Simulation environment and supports many
robots and sensors. § Not enough information for Gazebo for
§ Developing and test a node without a physical accurate simulation : pose, friction, …
robot.
§ Deployment of after test with minimal change. § Simulation Description Format(SDF)
§ Start with ’gazebo’ command invented for simulation in Gazebo.
§ ’gzserver’ : § Stable, robust, and extensible format for
§ Run the physics describing all aspects of robots, static and
§ Sensor data generation dynamic objects, lighting, friction and even
§ Can be used without any GUI physics.
§ ’gzclient’:
§ Provide a GUI for visualization of § SDF uses XML files like URDF.
simulation

11
Powerful ROS 3rd party tools
Ø Part of an SDF as example
Ø GAZEBO
<camera name="head">
<horizontal_fov>1.3962634</horizontal_fov>
§ Converting URDF to SDF
<image>
§ Add tags and modify the URDF for example:
<width>800</width>
<height>800</height>
§ An <inertia> element within each <link> element
<format>R8G8B8</format>
must be properly specified and configured.
</image>
§ Add a <gazebo> element for every <link>
<clip>
§ Add a <gazebo> element for every <joint>
<near>0.02</near>
§ Add a <gazebo> element for the <robot> element
<far>300</far>
§ …
</clip>
§ The complete instruction in Gazebo website.
<noise>
<type>gaussian</type>
<mean>0.0</mean>
<stddev>0.007</stddev>
</noise>
</camera>
12
Powerful ROS 3rd party tools
Ø Moveit

§ The most widely used open-source

software for manipulation, motion
planning and analyzing of robot
interaction with environment.

§ Capabilities:
§ Collision checking
§ Integrated kinematics
§ Motion planning
§ Integrated perceptions about
environment
§ Execution and monitoring
§ Interactive
MoveIt in Rviz moving the ABB robot around - https://www.youtube.com/watch?v=0hS0XUOgYXk -
Pablo Negrete

13
Powerful ROS 3rd party tools
Ø OpenCV § Conversion by cv_bridge : ready functions

§ The most powerful image processing library cv_ptr = cv_bridge::toCvCopy(msg,

§ Implemented in Python and C++. sensor_msgs::image_encodings::BGR8);
§ Many functionalities out of box : Face
detectio, Object tracking ,motion analysis, cv::circle(cv_ptr->image, cv::Point(50, 50),
Feature detection and … 10, CV_RGB(255,0,0));

§ ROS have drivers for many sort of cameras:

§ openni_kinect for Microsoft kinect
§ gscam for most webcams
§ swissranger_camera
§ …
§ ROS uses sensor_msgs/Image message
and OpenCV need matrices for images.
§ Conversion by cv_bridge stack.
[ROS.ORG- vision_opencv - http://wiki.ros.org/vision_opencv]

14
 ROS and external hardware : Arduino
Ø Arduino § Implementation

§ A microcontroller with powerful interface § ROS side : rosserial stack for serialization of message
library for different hardware. over USB [3]
§ Different I/O ports : Analog and digital § Arduino side: rosserial_arduino to create messages,
§ C-like language and syntax , Easy to publish, subscribe. [3]
program. Many open source projects. #include <ros.h> <std_msgs/String.h>
ros::NodeHandle n; std_msgs::String msg;
ros::Publisher pub("/my_topic", &msg); int count = 0;
char data[100];
[1] [2] void setup(){
n.initNode();
n.advertise(pub);
Arduino ROS }
USB rosserial_python void loop(){
ros_lib UART tty serial_node.py sprintf(data, "Hello world %d", ++count);
msg.data = data;
pub.publish(&msg);
[3]
[1 - Arduino Products - https://www.arduino.cc/en/Main/Products] n.spinOnce();
[2 – German robot - Opensource humanoid robot http://www.german-robot.com/]
delay(1000);
}
[3 – Wiki.ros.org - rosserial_arduinoTutorials - http://wiki.ros.org/rosserial_arduino/Tutorials]
http://wiki.ros.org/rosserial_arduino 15
How ROS works ?
Ø Nodes – Messages – Topics Ø Workflow:
§ Node : a process that uses ROS framework. 1. Node A publish a message to a topic
ROSCORE connects all nodes together and 2. All nodes which are subscribed to that
provide connectivity. topic, will receive the message.

Node B:
Node ROSCORE Node subscriber
§ Message: Standard definitions for Node A : Topic : Node C:
transferring data between nodes. Publisher Odometry subscriber

§ Topic: Mechanism of transferring data Ø Nodes commands: Ø Topiccommands:

between nodes.
§ Publisher: A node which produce message § rosrun package #show list of messages inside
and publish them. executable topic
§ Roslaunch § Rostopic echo /topicName
§ Subscriber: A node which receives the package_name § Rostopic list
messages. file.launch § Rostopic info topicName
16
How ROS works ?
Ø Service-Client
ROS Node
The publish/subscribe model is very flexible but not enough for a Service
distributed system.

§ Service-Client is way to retrieve the data immediately instead Request Response

of waiting for a message to be published.

§ A node provides a service , the client node call the service by Client
sending request message.
ROS Node
§ Service-client => one-to-one
§ Topic- message => one-to-one, one-to-many, many-to-many [Mathwork -
https://de.mathworks.com/help/robotics/examples/
call-and-provide-ros-services.html]

17
Implementation example : Message-Topic
Ø Subscribing to a topic
# Initialize rospy
NODE_NAME = 'localization'
import roslib; roslib.load_manifest(NODE_NAME)
import rospy This is a callback
function.
# Import LaserScan message type This is called whenever a
from nav_msgs.Odometry import * message of type
Odometry is received.
# Scan message handler
def odom_handler(msg):
# this code is executed whenever a scan is published Topic name
[...]
# Main function
def main():
rospy.init_node(NODE_NAME)
rospy.Subscriber("/odom", Odometry, odom_handler) Callback function
rospy.spin()
18
Implementation example : Message-Topic
Ø Publishing to a topic Ø Main benefits of message/topic system
# Initialize rospy
NODE_NAME = 'localization' § capture messages in a file And replay them
import roslib; roslib.load_manifest(NODE_NAME) later independently
import rospy
§ Clear communication structure between side
# Import standard String message type tools and libraries. As pointed out for
from std_msgs.msg import * example in RVIZ
Topic name
# Main function
def main():
pub = rospy.Publisher(“/scout/viewer”, String)
rospy.init_node(NODE_NAME)
msg = “Hello world”
pub.publish( String(msg) )
Constructor call of
message
Publish function

19
 Implementation example : Service-client
Ø Service
# Initialize rospy
Ø client
NODE_NAME = 'localization' # Initialize rospy
NODE_NAME = ‘viewer '
import roslib; roslib.load_manifest(NODE_NAME)
import rospy import roslib; roslib.load_manifest(NODE_NAME)
import rospy
# Import standard String message type
from std_msgs.msg import * # Import standard String message type
from std_msgs.msg import *
# Service handler
def handler(req): Service functionality # Main function
# this code is executed whenever the service is called def main():
return LocalizationSrvResponse() srv = rospy.ServiceProxy("/scout/localization",LocalizationSrv)
rospy.init_node(NODE_NAME)
# Main function Service name response = srv(1, x, y, theta)
def main():
rospy.init_node(NODE_NAME)
rospy.Service("/scout/localization", LocalizationSrv, handler)
rospy.spin()
Service type
20
How nodes find each other : ROS Master
Ø One node is a ROS Master by running Publish on Master
roscore command on it. Image topic

Ø Keep track of publishers, subscribers and Camera

(Publisher) topic
Image Viewer

topics.
Image

Ø After nodes locate each other, they

communicate peer-to-peer. Master
Subscribe
to image topic

Ø Steps: Camera Image

Publisher informs the ROS master about the topic and start (Publisher) Viewer

publishing.
Subscriber informs the ROS master about the interested Master
topics
Camera Image
ROS master inform Publisher that who is interested , and (Publisher) topic Viewer
publisher start sending messages to them.
Image

[ROS.ORG – ROS Master - http://wiki.ros.org/Master ]

21
Unity: Introduction
Ø Unity is game engine used to created high qualified visual [1]
scenes.
Ø Unity is visualization tool not a simulation.
Ø Unity is widely used for virtual reality (VR) tasks because:
§ Multi-platform : OSX, Windows, MAC, Android , ….
§ Powerful physics engine : gravity and collisions
§ Very GUI lets you drag and drop elements Ø Unity interface
§ Programming languages : C# and Javascript

[2]
[1 – Deviant art – Angry birds logo - http://www.deviantart.com/morelikethis/421156366] [3]
[2 – Geforde – Assassin's Creed Unity Graphics & Performance Guide - http://www.geforce.com/whats-new/guides/assassins-creed-unity-graphics-and-performance-guide]
[3- Unity Interface overview - Unity Official Tutorials - https://www.youtube.com/watch?v=5cPYpI6_yLs]

22
Unity with ROS
Ø Unity instead of RVIZ For visualization? ROS Unity
Not a good idea but possible. Ros bridge
Web socket
server API
§ ROS messages => events processed by
JSON
rendering loop in Unity. Data
§ liveliness of visualization is lost because rosserial_python Rendering
rendering should be fast. serial_node.py lib
§ Method : Connection between ROS-Unity by ROS
bridge.
§ Rosbrige : connection to outside world by JSON
API through web sockets Ø JSON Data examples:
§ roslaunch rosbridge_server {"op": "subscribe",
rosbridge_websocket.launch "topic": "/clock",
Creates a web socket server working on port 9090 "type": "rosgraph_msgs/Clock”}.

{"op": "publish",
§ Outside software call the server/port for "topic": "/unity/joy",
communication "msg": msg}.

23
Stand alone Unity
Ø Graphical robot controller : The reverse of previous
project
§ Sending move commands from graphical robot to
physical robot
§ Input from environment by camera, Kinect , etc to
control graphical robot.

Ø Physical Robot => Arduino robotic frame

ware
[1]
Ø Calculation of position, etc => Unity

Ø Unity to Arduino Connection => USB

Ø Benefit : Control robot in Real time with

human interaction

[1 - The Robot Engine - Making The Unity 3D Game Engine Work For HRI
Christoph Bartneck, Marius Soucy, Kevin Fleuret, Eduardo B. Sandoval]
24
Conclusions

Ø Complete OS for Robotics Ø Powerful visualization tool

Ø No equivalent Ø Some equivalents: Unreal, DirectX, …
Ø Suitable for industrial large Ø Suitable for game, design and graphic
scale robotic projects industry
Ø To some extend Human Robot Interaction

Ø Research subject : Combining

Unity3D and ROS for nice
environment simulation.
Ø What about sensor data ?????

25
References:
§ ROS wiki - http://wiki.ros.org/
§ Powering the world’s Robots- ROS.ORG- http://www.ros.org/
§ The Robot Engine - Making The Unity 3D Game Engine Work For HRI - Christoph Bartneck,
Marius Soucy, Kevin Fleuret, Eduardo B. Sandoval
§ From ROS to Unity: leveraging robot and virtual environment middleware for immersive
teleoperation - R. Codd-Downey, P. Mojiri Forooshani, A. Speers, H. Wang and M. Jenkin
§ GAZEBO - Robot simulation made easy - http://gazebosim.org/
§ MoveIt! Motion Planning Framework - http://moveit.ros.org/
§ Unity3D - https://unity3d.com/
§ Mathwork - https://de.mathworks.com/help/robotics/examples/call-and-provide-ros-
services.html

26