Autonomous Mobile Robot Design

Dr. Kostas Alexis (CSE)

Course Goals

 To introduce students into the holistic design of

autonomous robots - from the mechatronic
design to sensors and intelligence.
 Develop the capacity to design and implement
robotics.
 Provide strong algorithm and fundamentals
background.
 Combine theory with intuition and practice.
 Go through the process of robot design and
development based a semester-long project.
Course Goals

 To introduce students into the holistic design of

autonomous robots - from the mechatronic
design to sensors and intelligence.
 Develop the capacity to design and implement
robotics.
 Provide strong algorithm and fundamentals
background.
 Combine theory with intuition and practice.
 Go through the process of robot design and
development based a semester-long project.
Course Goals

 To introduce students into the holistic design of

autonomous robots - from the mechatronic
design to sensors and intelligence.
 Develop the capacity to design and implement
robotics.
 Provide strong algorithm and fundamentals
background.
 Combine theory with intuition and practice.
 Go through the process of robot design and
development based a semester-long project.
Course Goals

 To introduce students into the holistic design of

autonomous robots - from the mechatronic
design to sensors and intelligence.
 Develop the capacity to design and implement
robotics.
 Provide strong algorithm and fundamentals
background.
 Combine theory with intuition and practice.
 Go through the process of robot design and
development based a semester-long project.
Course Approach

 Teaching Modules: Each teaching module will be

as independent as possible. At the same time,
each one of them will end with an overview of
on-going research challenges.
 Coding Examples: Each teaching module is
accompanied with a wide set of coding
examples.
 Project-oriented: Most of your effort will be to
work on your team-based semester project. This is
how you will learn to work on robotics!
 Be Proactive & Autonomous: come and discuss
what you want to understand better or what you
want to know more about. Grow your own ideas.
Course Teaching Modules

 Module 1 – Introduction: Get a broad understanding about robotics.

 Module 2 – Propulsion and Vehicle Dynamics: Understand robot propulsion and
locomotion principles as well as the description of vehicle dynamics through the
relevant equations of motion.
 Module 3 – Perception and State Estimation: Learn how on-board estimation of
the vehicle full pose (position and orientation) takes place, how the robot
perceives the environment, localizes itself and maps its surroundings.
 Module 4 – Guidance and Control: Learn how to design high-performance
robot motion controllers and guidance laws.
 Module 5 – Path Planning: Learn how to develop algorithms for autonomous
path planning for aerial robotics.
 Module 6 – Machine Learning: Use robot data to implement learning
methodologies enabling advanced robot behaviors.
Course Material

 Textbook: Roland Siegwart, Illah Reza Nourbakhsh and Davide Scaramuzza,

"Introduction to Autonomous Mobile Robots", Second Edition, MIT Press.
 Textbook: B. Siciliano, O. Khatib (editors), “Handbook of Robotics”, 2nd Version,
For “Flying Robots” chapter (co-author by Dr. Alexis) send an e-mail
 Lecture Slides: Used for the classroom presentations and also as a way for notes
keeping and direct reference to the course contents.
 Code Examples: several examples in MATLAB, Python, C++ and special focus on
ROS and the Pixhawk autopilot.
 Open-Source Aerial Robots Simulator: a complete simulation environment for
advanced designs.

 Get the course material: The complete set of the relevant materials are
available at: http://www.kostasalexis.com/autonomous-mobile-robot-
design.html
Reference Textbook

 Roland Siegwart, Illah Reza Nourbakhsh and

Davide Scaramuzza, "Introduction to Autonomous
Mobile Robots", Second Edition, MIT Press
 B. Siciliano, O. Khatib (editors), Springer
“Handbook of Robotics”, Second Edition, Springer-
Verlag
Code Examples

 Dedicated course repository:

 https://github.com/unr-arl/autonomous_mobile_robot_design_course/
 MATLAB, Python, C++, ROS, Pixhawk examples and more
 Control, Path Planning, Computer Vision, State Estimation, Dynamics and more
Code Examples

 Dedicated course repository - examples

Code Examples

 Dedicated course repository - examples

Code Examples
 Dedicated course repository - examples
Simulator Tools

 Open-Source simulator for Aerial Robotics:

http://www.autonomousrobotslab.com/rotors-simulator4.html
Literature and Links

 Literature references
 Tutorials
 Further coding examples
 User guides
 …
Video Explanations

 Video explanations for special topics

from selected resources.
Course Grading System:
 Project-based:
 Design Project with intermediate report: 80%
 90% in cases of excellence
 Final Exam: 20% (or 10% in case of project excellence)
 Exam-based:
 Project: 40%
 Mid-term Exam: 20%
 Final Exam: 40% (up to 60% in case of excellence)
 Homework: +10% (as a bonus)
Tentative scale (curve will be applied)
 Grade >= 90: A
 80 <= Grade <= 89: B
 70 <= Grade <= 79: C
 60 <= Grade <= 69: D
 59 >= Grade: F
Autonomous Robot Challenges
Propulsion and Vehicle Dynamics

How do I move?
Autonomous Robot Challenges
Propulsion and Vehicle Dynamics

Aerial Robotics Ground Robotics

Autonomous Robot Challenges
Perception and State Estimation

Where am I?
What is my
environment?
Autonomous Robot Challenges
Perception and State Estimation

Visual-Inertial SLAM Application to Autonomous Driving

Autonomous Robot Challenges
Guidance and Control

How do I control
where to go?
Autonomous Robot Challenges
Guidance and Control

Aerial Robotics Ground Robotics

Autonomous Robot Challenges
Path Planning

How do I plan
my motion and
actions?
Autonomous Robot Challenges
Path Planning

Exploration Path Planning And Robust Navigation

Autonomous Robot Challenges
Machine Learning

How to derive
Behaviors from data?
 Autonomous Robot Challenges
Machine Learning

Deep Reinforcement Learning Semantic Classification

 What are the challenges ahead?
No sensible decision can be made any longer
without taking into account not only
the world as it is, but the world as it will be. I.A.
 a robot may not injure a human being or, through
inaction, allow a human being to come to harm
 a robot must obey orders given it by human beings
except where such orders would conflict with the
first law
 Can we operate robots without having  a robot must protect its own existence as long as
such protection does not conflict with the first or
special skills? second law

 Can robots actively explore and navigate

their environments and act on it?
 Can we assign complex tasks to
autonomous robots?
 Can we ensure collision avoidance?
 Can we trust robots to operate within the
urban landscape? Can we trust them to
operate next to us or work for/with us?
Isaac Asimov
The Basic Robot Loop

Block diagram of the main loops running at every robot

The Basic Robot Loop
 Real-life Robot expressing
its dynamic behavior in
response to the control
inputs and external
disturbances
The Basic Robot Loop
 Real-life Robot expressing
its dynamic behavior in
response to the control
inputs and external
disturbances

 Inertial Navigation
System
 Localization & Mapping
 Sensor Fusion
 Semantic Understanding
The Basic Robot Loop
 Real-life Robot expressing
its dynamic behavior in
response to the control
inputs and external
disturbances

 Control and Guidance  Inertial Navigation

system responsible for System
ensuring vehicle stability
and reference trajectory  Localization & Mapping
tracking as well as
disturbance rejection.  Sensor Fusion
 Semantic Understanding
The Basic Robot Loop
 Path planning in order to  Real-life Robot expressing
compute the path the its dynamic behavior in
robot should follow to
ensure safe navigation and response to the control
execute the desired inputs and external
mission. disturbances

 Control and Guidance  Inertial Navigation

system responsible for System
ensuring vehicle stability
and reference trajectory  Localization & Mapping
tracking as well as
disturbance rejection.  Sensor Fusion
 Semantic Understanding
 The Basic Robot Loop

A hard real-time system with relatively limited computational resources!

Course Projects

 This course is organized around a semester-long project to be handled by a

team of students.
 Each student team will have available funds up to $2,000 to acquire the
hardware required to implement the robotics challenge.
 All projects are significantly involved and set state-of-the-art research
challenges to you.
 All projects will require some serious teamwork.
Project 1: GPS-denied Autonomous Car
Localization in Visually-degraded Conditions
 Task 1: Vision, NIR, LiDAR System Integration
 Task 2: Vision, NIR, LiDAR Sensor Fusion
 Task 3: Dataset collection and groundtruth stamping using GPS
 Task 4: Field experiments and evaluation
Project 2: Change Detection for
Autonomous Driving
 Task 1: Change detection in images
 Task 2: Volumetric mapping
 Task 3: Change detection in volumetric maps
 Task 4: Semantic change classification using convolutional neural nets
 Task 5: Dataset collection and groundtruthing
 Task 6: Field experiments and evaluation
Project 3: Robotic Inspection of Mines

 Task 1: Platform development (aerial or ground)

 Task 2.a: Volumetric and surface mapping or
 Task 2.b: Automated micro-excavator
 Task 3: Aerial - to -ground robot collaboration
 Task 4: Dataset collection and groundtruthing
 Task 5: Field experiments and evaluation
Autopilot Solution: Pixhawk

 Open-source project (PX4) started at ETH Zurich

 Currently supports rotorcrafts, fixed-wing vehicles,
rovers, boats and more.
 Robust autopilot solution with large supporting
community.
 Under extensive redesign at the period. Many new
products are expected to come.
Visual-Inertial-Depth SLAM

 Visual-Inertial Robust localization and mapping

approach: performs much better compared to
camera-only solutions.
 Will be the basis for all projects – further sensing
modalities will be integrated when relevant (e.g.
thermal camera or LiDAR)
Time-of-Flight RGB-Depth Sensors

 Sensing systems that capture RGB (visual) images

along with per-pixel depth information. This can be
achieved either via stereo rigs or the use of time-of-
flight concepts.
 Microsoft Kinect created a new class of sensing
solutions that quickly found great application in
robotics.
 Since then, a wide set of sensors with very low-cost
have been released.
High-level processing and Middleware

 For the most advanced functionalities such as

mapping or path planning, a second processing
level is typically employed.
 We look for a system that can support Linux
installation and ability to run the Robot Operating
System (ROS).
 Robot Operating System (ROS) is a collection of
software frameworks for robot software
development providing operating system-like
functionality on a heterogeneous computer cluster.
Course Projects

 Team projects involving approximately 6 students

 Holistic experience. Student team responsible to assign internal role and
split the project into subtasks.
 Project with real-hardware provided by the lab.
 Place within the university to work on the project
 Work at the Facilities of the Autonomous Robots Lab and the Nevada Center for
Applied Research
 Weekly supervision and guidance meeting
 Code examples available for all steps of your student project
 Potential of scientific publication from all of the proposed projects!
Course Projects

 Each team should have a combination of graduate and undergrad

students.
 Graduate students are expected to be able to lead the team.
 Every team will have a weekly meeting at a fixed time to discuss progress
and coordinate the next moves.
 Testing should happen first in the Autonomous Robots Arena when possible.
Course Projects

 Indicate your project preference at [email protected]

 Work on developing your team! Be autonomous and proactive!
 Create Github account if you don’t have one. At your e-mail, also share
with me your github account username.
Testing in the Autonomous Robots Arena

 Indicate your project preference at [email protected]

 Work on developing your team! Be autonomous and proactive!
 Create Github account if you don’t have one. At your e-mail, also share
with me your github account username.
 Ongoing set-up

 Motion Capture-enabled volume of 15x7x5m

 Sub-mm, Sub-degree accuracy of pose estimation
 >$100,000 infrastructure
 Scheduling your experiment
 Once ready to test a functionality in your robot,
appointments can be arranged via e-mail at
[email protected]. An online system will be available soon.
 How do I start?
 Go to:
http://www.autonomousrobotslab.com/autonomous-
mobile-robot-design.html
 Lectures to see what is next (each lecture will be
available worst-case before the corresponding class)
 Code Repository to start working on stuff
 Literature and Links to find good tutorials and relevant
courses from other universities
 Make sure you get to know about ROS
 Make sure you know probability theory and linear algebra. If
any of you feels weak on that let me know.
 2016 Projects
 In the following slides, the project topics provided during
2016 are summarized
Project 1: Autonomous Cars Navigation
 Task 1: Sensing modules and Processing Unit Integration
 Task 2: Autopilot integration and verification
 Task 3: Robot Localization and mapping through fusion of RGBD/Visual-SLAM
 Task 4: Static/Dynamic Obstacle Detection
 Task 5: Robot car motion collision-free planning
 Task 6: Robot Evaluation and Demonstration in the Autonomous Robot Arena and the UNR campus
Project 2: Robots to Study Lake Tahoe!

 Water is a nexus of global struggle, and increasing pressure on water resources is driven by large-
scale perturbations such as climate change, invasive species, dam development and diversions,
pathogen occurrence, nutrient deposition, pollution, toxic chemicals, and increasing and
competing human demands. The goal of this project is to design and develop a platform that
can be used on the surface of a lake to quantify the water quality changes in the nearshore
environment (1-10 m deep). The platform would be autonomous, used to monitor the
environment for water quality (temperature, turbidity, oxygen, chl a) at a given depth.
 Collaborators: Aquatic Ecosystems Analysis Lab: - http://aquaticecosystemslab.org/
Project 2: Robots to Study Lake Tahoe!
 Task 1: Autopilot integration and verification
 Task 2: Sensing modules and Processing unit improvements
 Task 3: Robot Localization and Mapping using Visual-Inertial solution
 Task 4: Fused visible light/thermal fusion for unified 3D reconstruction
 Task 5: Robot boat autonomous navigation for shoreline tracking
 Task 6: Robot Evaluation and Demonstration
Project 3: Aerial Robotics for Climate
Monitoring and Control
 Within this project you are requested to develop an aerial robot capable of environmental
monitoring. In particular, an “environmental sensing pod” that integrates visible light/multispectral
cameras, GPS receiver, and inertial, atmospheric quality, as well as temperature sensors. Through
appropriate sensor fusion, the aerial robot should be able to estimate a consistent 3D
terrain/atmospheric map of its environment according to which every spatial point is annotated
with atmospheric measurements and the altitude that those took place (or ideally their spatial
distribution). To enable advanced operational capacity, a fixed-wing aerial robot should be
employed and GPS-based navigation should be automated.
 Collaborators: Desert Research Institute - https://www.dri.edu/
Project 3: Aerial Robotics for Climate
Monitoring and Control
 Task 1: Autopilot integration and verification
 Task 2: Sensing modules and Processing unit integration
 Task 3: Integration of Visual-Inertial SLAM solution
 Task 4: Development and integration of atmospheric sensors (CO/CO2, aerosol)
 Task 5: Environmental-data trajectory annotation and estimation of spatial distributions
 Task 6: Real-time plane extraction for landing
 Task 7: Robot Evaluation and Demonstration
Project 4: Aerial Robotics for Nuclear Site
Characterization
 A century of nuclear research, war and accidents created a worldwide legacy of contaminated
sites. Massive cleanup of that nuclear complex is underway. Within this project in particular, the
goal is to develop multi-modal sensing and mapping capabilities by fusing visual cues with
thermal and radiation camera data alongside with inertial sensor readings. Ultimately, the aerial
robot should be able to derive 3D maps of its environment that are further annotated with the
spatial thermal and radiation distribution. Technically, this will be achieved via the development
of a multi-modal localization and mapping pipeline that exploits the different sensing modalities
(inertial, visible-light, thermal and radiation camera). Finally, within the project you are expected
to demonstrate the autonomous multi-modal mapping capabilities via relevant experiments
using a multirotor aerial robot.
Project 4: Aerial Robotics for Nuclear Site
Characterization
 Task 1: Thermal, LiDAR, Radiation Sensing modules integration
 Task 2: Thermal camera-SLAM
 Task 3: Multi-modal 3D maps
 Task 4: Estimation of spatial distribution of heat and radiation
 Task 5: Heat/Radiation source seek planning
 Task 6: Robot Evaluation and Demonstration in the Autonomous Robots Arena and a tunnel-like
environment.
Project 5: Smartphone-assisted Delivery
Drone Landing
 The goal of this project is to develop a system that exploits direct/indirect communication
between a smartphone and the aerial robot such that delivery landing "on top" of the
smartphone becomes possible. Such an approach will enable commercial parcel delivery within
challenging and cluttered urban environments. Within the framework of the project, we seek for
the most reliable, novel but also technologically feasible solution for the problem at hand. The
aerial robot will be able of visual processing and may implement different communication
protocols, while the smartphone should be considered "as available" on the market.
 Collaborators: Flirtey - http://flirtey.com/
Project 5: Smartphone-assisted Delivery
Drone Landing
 Task 1: Autopilot integration
 Task 2: Camera systems integration
 Task 3: Robot-to-Phone and Phone-to-Robot cooperative localization
 Task 4: Visual-servoying phone tracking
 Task 5: Autonomous Landing on phone
 Task 6: Robot Evaluation and Demonstration
Thank you!
Please ask your question!