Institute of Automotive Technology

Faculty for Mechanical Engineering

Technical University of Munich

Autonomous Driving Software Engineering

Prof. Dr.-Ing. Markus Lienkamp
Phillip Karle, M. Sc.
 Lecture – 90min Practice – 45min
1 Introduction: Autonomous Driving 1 Practice
Karle Karle
2 Perception I: Mapping 2 Practice
Sauerbeck Sauerbeck
3 Perception II: Localization 3 Practice
Sauerbeck Sauerbeck
4 Perception III: Detection 4 Practice
Huch Huch
5 Prediction 5 Practice
Karle Karle
6 Planning I: Global Planning 6 Practice
Trauth Trauth
7 Planning II: Local Planning 7 Practice
Ögretmen Ögretmen
8 Control 8 Practice
Wischnewski Wischnewski
9 Safety Assessment 9 Practice
Stahl Stahl
10 Teleoperated Driving 10 Practice
Feiler Feiler
11 End-to-End 11 Practice
Betz Betz
12 From Driver to Passenger 12 Practice
Fank Karle

8-2
 Objectives for Lecture 8: Control
Depth of understanding
After the lecture you are able to…
Remember Understand Apply Analyze Evaluate Develop
.. understand the necessity of stabilizing control actions
and the responsibilities of the control algorithm within the
autonomous driving software stack
.. understand the most important control concepts for
autonomous driving (geometric path following controllers,
state-space controllers and model predictive control)
.. design and tune geometric path following controllers and
state-space lateral path tracking controllers
.. design and tune a velocity controller for basic cruise
control in highway scenarios

8-3
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp

Alexander Wischnewski, M. Sc.

Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
 Lecture Overview
12
Human-Machine-Interface

11
End-to-End

Modular Software Stack

5 8
Sensors Perception Prediction Planning Control Actuators
2
Mapping Global
3 6
Localization Behavior
4
Detection 7 Local

9 10
Safety Teleoperated Driving
X = Lectures
8-5
Role of control in the AV software stack

Target trajectory Steering

Vehicle position Throttle &

Brakes
IMU,
Wheelspeeds…

Control module

▪ Control is focussed on the short-term behavior (roughly 2 seconds)

and updates frequently (at minimum with 20 Hertz)
▪ Real-time operation is crucial
▪ Control needs maximum knowledge about the vehicle
8-6
Why do we actually need feedback control?

How engineers like to think What it‘s actually like:

about the world:

▪ The real world is full of uncertainties and disturbances

▪ Control is about taking into account those uncertainties and
manage them in a structured way to achieve the desired task

8-7
The evolution of autonomous vehicle control

Geometric controllers

Stanley – Stanford (2005) Talos – MIT (2007)

Model-based
feedforward and
feedback controllers
Shelley – Stanford (2015) Roborace – TUM (2018)

Model predictive
controllers

1:43 Racing - ETH (2015) Indy Challenge – TUM (2021)

8-8
Additional Slides

The evolution of autonomous vehicle control

The evoluation of autonomous vehicle control can be divided into three major development steps:
• Inspired from the robotics community, the first control algorithms have been inspired by geometric
concepts such as planning a circular movement towards the target path or aligning the steering
wheels with the path. These approaches show good performance for low to medium speeds.
• When increasing the speed and accelerations of the autonomous driving prototypes, more
thorough analysis methods from control theory and system dynamics became popular within the
research community. They allow to design explicit state-feedback controllers and consider more
detailed effects such as vehicle yaw dynamics, steering actuator dynamics and others.
• In recent years, the interest gathers around model predictive control algorithms. These use a
dynamic model of the system to predict the future behavior for a few seconds and determine
optimal control actions. In contrast to explicit state-feedback controllers, they can use and exploit
knowledge about constraints and uncertainties. Furthermore, they can be extended to more
complex (nonlinear) vehicle dynamics.
Image sources:
https://stanford.edu/~cpiech/cs221/img/stanley1.jpg
https://www.researchgate.net/profile/Emilio_Frazzoli/publication/228794498/figure/fig1/AS:30069332
9072128@1448702284145/Talos-Team-MITs-race-vehicle-at-the-DARPA-Urban-Challenge.png
https://news-media.stanford.edu/wp-content/uploads/2019/03/25110734/friction_shelly9-555x370.jpg
https://control.ee.ethz.ch/research/team-projects/autonomous-rc-car-
racing/_jcr_content/par/textimage_1984079938/image.imageformat.lightbox.1404291081.jpg

1- 9
 Control theory basics – System dynamics I

Input Output
System

Examples:
▪ Spring-mass-damper
systems
▪ Inverse pendulum
▪ Autonomous vehicles

8 - 10
Control theory basics – System dynamics II

System matrix State Input matrix

Input

Output

Output matrix Feedthrough matrix

Imaginary axis
Poles of the
System dynamics are given system
from the spectrum of A: Real axis

8 - 11
Additional Slides

Control theory basics – System dynamics I and II

• A dynamic system is a mathematical description of the relation between an input u and an output
y signal. This description is usually given in form of an ODE (Ordinary Differential Equation).
• The system has so-called states x. These are variables which allow us to formulate the system
behavior in form of a set of first-order ODE for each of this variables.
• The standard system description consists of the system dynamics and the output equation. The
former describes the timely behavior of the states as a reaction to the inputs and the initial state.
The latter describes the relation between the state and the output.

• An important special case are linear time-invariant systems (LTI-Systems). They can be
described using matrix algebra only and can be analyzed using Laplace-Transformation theory.
• The system dynamics are specified via the so-called characteristic equation

• The roots of this equation are called the „poles“ and they are related to the stability and
convergence properties of the dynamic system
• If all poles have negative real parts the system is stable
• The absolute value of the negative real part of the poles determines the convergence speed
• The imaginary parts of the poles is related to it‘s oscillation characteristics

1- 12
Control theory basics – Standard control loop I

Control error Disturbances

Setpoint Input Output

Controller System

Examples for control loops in autonomous vehicles:

Vehicle motion Steering angle Brake pressure

8 - 13
Additional Slides

Control theory basics – Standard control loop I

The basic control loop is divided into the main components: sensor, dynamic system under control
(plant), actuator (can also be considered part of the plant) and the controller.
• All parts of this pipeline are packed with uncertainties:
• Sensor calibration errors
• Sensor noise and faulty measurements
• Complex (and unmodelled) actuator dynamics
• External disturbances such as wind gusts, inclination and rolling resistance
• Parametric uncertainties in the plant (e.g. number and weight of passengers)
• In an idealized world a pure feedforward controller would be sufficient → plan the sequence of
actuation commands based on the model of the plant and execute this plan until the task is done
• Uncertainties require corrective actions and a stable feedback loop

1- 14
 Control theory basics – PID controller design I

Block diagram with transfer functions:

Control
error

Control Control
value gains

▪ Three main parts: Proportional / Integral / Derivative

▪ Standard approach for feedback control in industrial applications

8 - 15
Control theory basics – PID controller design II

▪ Tuning often done empirically during system operation

▪ Works well for systems with low complexity (e.g. mass-spring-
damper system)
8 - 16
Control theory basics – PID controller design III

▪ Zero steady-state in the presence of uncertainty or external

disturbances requires integral action
▪ Alternative: high feedback gains to suppress disturbances but
this might lead to oscillations in real world scenarios 8 - 17
Control theory basics – State-space design

• Algorithmic method based on

designing a state-feedback matrix

• Analyze characteristic equation of

closed-loop dynamics

• Possibilities to determine controller

• Pole-placement via coefficient
comparison
• LQR controller design

8 - 18
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp

Alexander Wischnewski, M. Sc.

Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Decoupling of longitudinal and lateral dynamics

Joint trajectory Decoupled longitudinal

tracking control and lateral control

Real vehicles have

✓ coupled dynamics ✓ Controllers are easy to
design and tune

✗ Difficult to handle due to

nonlinearities
✗ Less accurate for high
acceleration and dynamic
maneuvers
✗
Advanced control
algorithms necessary
8 - 20
Stanley controller

Lane
▪ Determine cross-track error and centerline
heading error at the front axle
▪ Align front wheels with path and
add a correction term

▪ The control gain adjusts the

convergence speed

8 - 21
Additional Slides

The stanley controller is a geometric controller. This name originates from the geometric interpration for
the control laws. The PID control concept is more sophisticated but can still be designed without any
knowledge about the vehicle dynamics. In contrast to the geometric controllers, it adds an integral term
which helps to mitigate external disturbances such as cross-wind or banking, but can also lead to
difficulties such as integrator wind-up.

All of the presented concepts are suitable for low-medium speed and acceleration driving (urban
scenarios and easy high-way situations), depending on the tuning efforts spent and the quality of the
vehicle hardware.

Literature:
• S. F. Campbell, “Steering control of an autonomous ground vehicle with application to the DARPA
urban challenge,” University of Notre Dame, 2005
• Snider, Automatic Steering Methods for Autonomous Automobile Path Tracking

1- 22
Velocity PI controller
Disturbances such as
wind, inclination, …

Setpoint Speed Longitudinal Force Speed

PI- Longitudinal
Controller Dynamics

▪ Longitudinal dynamics are a single integrator dynamic, therefore

no derivative part is needed
▪ Integrator required to achieve zero steady-state error due to
external disturbance
8 - 23
Additional Slides

The velocity control task receives a lot less interest within the control engineering community, however,
its a great example for the success of easy to design and tune control concepts.
• The system dynamics consists of a single integrator (the vehicle mass) and a low-pass modelling the
actuation delay within the brake system and the engine. The latter can often be neglected as in our
example presented in the lecture.
• The influence of modelling error (driving resistances such as aerodynamic drag or tire roll resistance)
and the presence of wind gusts and road inclination makes it necessary to add an integral part to the
control law. A derivative part can be used but is optional as only one integrator is present in the open-
loop system.

Even though the velocity control task is simple from a control engineering point of view, it has two
important consequences for the overall performance of the autonomous driving software:
• It contributes heavily to ride comfort. Jerky behavior or noise corrupted sensors leading to oscillating
force requests have to be prevented.
• The performance of the lateral controller can depend significantly on the longitudinal controller. If the
actual speed deviates a few percent from the target speed this might have significant consequences
on the resulting lateral accelerations. This is due to the quadratic influence of velocity on the lateral
acceleration.

1- 24
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp

Alexander Wischnewski, M. Sc.

Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
 Improved velocity control with model knowledge
Setpoint stabilization with PI-controller:

Disturbance 3000N

8 - 26
 Improved velocity control with model knowledge
Setpoint tracking (e.g. changing speed limit) with PI-controller:

Disturbance 3000N

8 - 27
Improved velocity control with model knowledge

Feedforward
Controller
Control Disturbances
value
Measured
Setpoint
value
Feedback
System
Controller

▪ Two degree of freedom structure allows to design separate

dynamics for setpoint changes and disturbance influences
▪ One of the most common structures in industrial applications

8 - 28
Improved velocity control with model knowledge
Vehicle longitudinal
dynamics model
Force
request Velocity

Vehicle mass Integrator

System inverse for feedforward control

Differentiator Vehicle mass

Velocity Setpoint Feedforward

setpoint acceleration force request

Low-pass
8 - 29
 Improved velocity control with model knowledge
Setpoint tracking (e.g. changing speed limit) with PI-controller and
feedforward control:

Disturbance 3000N

8 - 30
Additional Slides

• The two-degree-of-freedom (2-DOF) control structure allows for seperate design of setpoint and
disturbance dynamics. It is well-known for its high performance for trajectory tracking
applications. The general idea is to do „everything we know about the system“ already before a
control error through a setpoint change occurs.
• This concept allows to track trajectories with fast dynamics why keeping smooth and cautious
feedback compensation via the feedback controller.

• Feedforward designs are oriented around the inverse of a plant. The inverse generates a
suitable input sequence such that the plant output matches the desired setpoint trajectory.
• There are many ways to calculate such a feedforward control law:
• Directly obtain the required input sequences or derivatives from the trajectory planning
algorithm/method, e.g. via splines or other basis functions.
• Approximate it via an inverse transfer function model. The exact inverse is usually not
realizable in this form due to the resulting non-causal transfer functions (numerator
polynomial order is higher than denominator polynomial order). This reflects the fact that
numerical inversion takes some time to work correctly.
• Use a model-predictive-control law (see chapter 5 for more details on this concept).

1- 31
Kinematic single track model

▪ Dynamic equations derived from kinematic

constraints only → no tire slip and forces

position heading

Steering Vehicle speed

angle

8 - 32
Additional Slides

Vehicle dynamics models in motion controller design

• There are three main models used for motion controller design:
• Kinematic vehicle models
• Linear single track models
• Nonlinear single track models
• It is only very rarely that more complex models are used for controller design. The testing of the
designed controller should however still be done with more complex models such as a nonlinear dual
track model, considering effects of suspension as well as combined tire effects.
• The kinematic vehicle model is considered sufficient for slow driving at low lateral accelerations. It
represents the key relation between vehicle movement and position dynamics and is usually a good
choice to get started.
• If the speeds or accelerations are in a medium range (high-way driving or accelerations around 3-
5mps2), changing to a linear single track model should be considered. The key advantage here is that
it models the self-steering behavior and gives a coarse estimate of the vehicles lateral dynamics.
• The nonlinear single track model is often used for high-performance applications such as racing. It
models the tire behavior at the handling limits better than its linear counterpart, however, this also
poses difficulties during the control design process.

1- 33
Important relations in vehicle dynamics

Steady-state acceleration GG-Diagram

Steady-state yaw rate

Steady-state curvature

▪ Max. combined accelerations

for overall vehicle
▪ Different forms are possible
depending on the vehicle

8 - 34
Frenet coordinate frame
Progress
along the path
Lateral
deviation

8 - 35
Trajectory specification in Frenet coordinates

Target

8 - 36
Additional Slides

Coordinate choice for the vehicle control problem

• Many simulations models, also the ones we have presented earlier, are naturally formulated in
cartesian coordinates. This space is the most natural to formulate a mechanical problem and
therefore widely used by practioniers and researchers alike.
• One of the key issues here is the strong nonlinearity in with respect to the trajectory. A stabilizing
mechanism has to be adapted to the heading all the time, this renders the problem nonlinear and
difficult to analyze in the theoretical settings we have learned about earlier.
• This issue might be overcome by the choice of a so-called curvilinear coordinate system. It transforms
the eucledian coordinates into a coordinate system which is located along the path. The first
component is the progress along the path, the second component is the lateral deviation with respect
to that path. The coordinate system itself can be defined via a series of points on the path or via a
curvature profile for each point on the path. The actual path can then be obtained from double
integration.
• This curvilinear system will be the base for many of the control designs we see later on. It is a natural
formulation of the split between lateral and longitudinal tracking and therefore well suited for the
analysis of the respective control systems.
• There is a key difficulty: the mapping between cartesian coordinates and curvilinear coordinates might
not be well-defined in very sharp turns or if the path crosses itself. However, it should be kept in mind
that these cases also pose significant difficulty to deal with in control design of cartesian coordinates
and should therefore be considered challenging in general.

1- 37
 Improved lateral control with model knowledge

▪ Reformulation of the lateral path tracking

problem in Frenet-coordinates
▪ Differentiating twice leads to double-integrator
dynamics in the curvilinear coordinate frame
corrective acceleration
in path coordinates

▪ For small errors this is linear w.r.t. the

acceleration in vehicle coordinates

Lateral acceleration required Corrective lateral

to stay on the path acceleration (feedback)
(feedforward)
8 - 38
Improved lateral control with model knowledge

▪ Control law has the structure of a 2-DOF

gain-scheduling PD controller

▪ Calculate steering angle from kinematic

bicycle model

▪ Calculate lateral error derivative from

Velocity heading error

8 - 39
Advanced motion control – I

Tire side slipe angle Linear tire model Nonlinear tire model

Tire lateral force

Cornering
stiffness

▪ Advanced vehicle models require detailed modelling of the tire

▪ Kinematic rolling condition is replaced with tire slip and tire models

8 - 40
Advanced motion control – II

Yaw rate
Neutral steering /
Over- Kinematic bicycle model
steering
Understeering

Steering angle

▪ Self-steering characteristic is most important factor

▪ Depends heavily on tires, suspension setup and vehicle speed
▪ Different reasons for understeer (e.g. front-rear tire balance,
approaching the limits of the tire, …)
8 - 41
Advanced motion control – III

Steering angle
Nonlinear Curvature
Vehicle velocity vehicle
dynamics

Steering angle
Steady-state Curvature
Vehicle velocity self-steering
characteristic

While operating in the stable driving region it is sufficient to know

about the steady-state self-steering characteristics to obtain a
reasonable good feedforward control law!

8 - 42
Advanced motion control – IV

Inverse self-
steering
characteristic

Target Steering
Lateral path curvature Curvature angle
tracking controller + Vehicle
controller P-controller

▪ Nonlinear feedforward controller based on self-steering

characteristic
▪ Curvature feedback can be done using a simple P-controller

8 - 43
Application to autonomous racing @ Roborace 2019

✓ Successful application at
Roborace competition 2019

✓ Close to human driver

performance on qualifying lap

✓ Moderate implementation
efforts required Source: https://www.youtube.com/watch?v=-vqQBuTQhQw

More details in:

✗
Minimum curvature trajectory planning and control for
Depends on curvature and an autonomous race car
acceleration signal quality (https://doi.org/10.1080/00423114.2019.1631455)

✗ A software architecture for the dynamic path planning of

an autonomous racecar at the limits of handling
Difficulties with dynamic (https://doi.org/10.1109/ICCVE45908.2019.8965238)
trajectory planning 8 - 44
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp

Alexander Wischnewski, M. Sc.

Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Basics of model predictive control – I

Source: https://de.wikipedia.org/wiki/Model_Predictive_Control

8 - 46
Basics of model predictive control – II

Steering angle
Vehicle motion
prediction

Time Vehicle
dynamics
model
Longitudinal force

Time

Target
Optimize based on comparison of trajectory
prediction and target trajectory
8 - 47
Basics of model predictive control – III

▪ Solution to this optimization problem is computational expensive

▪ Typical problem sizes are around 20-50 discretization steps

Quadratic Nonlinear
Programming Programming

▪ Quadratic cost function, ▪ Arbitrary cost functions and

linear constraints constraints possible
▪ Works well for linear ▪ Nonlinear dynamics
dynamics ▪ Solvers are based on
▪ Efficient and reliable solvers linearization schemes
available with <10ms ▪ Solution times range from a
computation time few milliseconds to multiple
seconds 8 - 48
Basics of model predictive control – IV

Considers constraints and is suited for multi-input and multi-

✓ output systems

✓ Nonlinear dynamics and complex cost functions

✗ Implementation and solution of numerical problem is a

complicated challenge

✗ Computation times get large for many active constraints or

nonlinear dynamics

✗ Solver might not converge → backup strategy needed

8 - 49
Additional Slides

• Model predictive control is a promising concept – it allows to handle general nonlinear systems, multi-
input/multi-output systems as well as constrained control problems. These advantages make the
concept attractive to researchers and practitioniers within the community. However, this comes at a
price which is related to the requirement to implement the optimization as a tractable (finite
dimension) and finite-time horizon optimization task.
• One challenge within constrained problems is that a controller might tend to be overly optimistic
towards the end of the optimization horizon. This can render the solution of the next timestep
optimization infeasible. There is a great amount of literature on this problem called „recursive
feasibility“. It aims at deriving certain modifications to the optimization problem which guarantee that a
solution to the next optimization problem will be available under certain assumptions.
• The same issue might arise due to inaccuracies within the model or external disturbances, both
effects which are likely to occur. This might be tackled by „Robust MPC“ or Soft-Constraints.
• Despite these control engineering challenges, it remains to find a computational efficient version of
the desired optimization problem. Especially nonlinear problems are time consuming to solve and
usually need a lot of hyperparameter tuning for the optimization algorithms which makes them difficult
to use for online operation in safety critical systems. Many practitioniers therefore add a backup
controller which can take over in case the MPC fails.
• While this control concept seems to be promising, it is still an active research topic with many
contributions to be made until it can be considered a „standard-tool“ which can be deployed easily. In
the case of autonomous vehicle control there are many easier control concepts which allow to reach
reasonable performance levels of about 50-80% of the vehicle capabilities.

1- 50
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp

Alexander Wischnewski, M. Sc.

Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Summary – What did we learn today

• Responsibilities of the controller within the autonomous driving stack

• Recap of control theory and vehicle dynamics modeling

Model predictive control

Model-based feedforward
and feedback control
Performance

Stanley &
Velocity PI

Complexity
8 - 52
Helpful literature for this lecture

Required control engineering basics:

▪ Lecture „Regelungstechnik“ – Prof. Dr.-Ing. Boris Lohmann – Chapter 6 & 7
(Reglerentwurf & Erweiterte Regelungsstrukturen und Zustandsregelung)

Further control engineering materials:

▪ Otto Föllinger, Regelungstechnik, VDE Verlag GmbH
▪ Heinz Unbehauen, Regelungstechnik I, Springer Vieweg
▪ John Doyle, Bruce Francis, Allen Tannenbaum, Feedback Control Theory,
Macmillan Publishing Co.

Further vehicle dynamics materials:

▪ Lecture „Dynamik der Straßenfahrzeuge“ – Prof. Dr.-Ing. Markus Lienkamp
▪ William F. Milliken and Douglas L. Milliken – Race Car Vehicle Dynamics

8 - 53
Related literature

A list of papers for the controllers presented within the literature and more
interesting concepts:

Brian Paden, Michal Cap, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli, A Survey of Motion
Planning and Control Techniques for Self-Driving Urban Vehicles, 2016
Gabriel M. Hoffmann, Claire J. Tomlin, Michael Montemerlo, and Sebastian Thrun, Autonomous
Automobile Trajectory Tracking for Off-Road Driving: Controller Design, Experimental Validation and
Racing, 2009
Moritz Werling, Ein neues Konzept für die Trajektoriengenerierung und -stabilisierung in zeitkritischen
Verkehrsszenarien, 2011
Nitin R. Kapania and J. Christian Gerdes, Design of a feedback-feedforward steering controller for
accurate path tracking and stability at the limits of handling, 2015
Alexander Heilmeier, Alexander Wischnewski, Leonhard Hermansdorfer, Johannes Betz, Markus
Lienkamp, and Boris Lohmann, Minimum curvature trajectory planning and control for an autonomous
race car, 2019
Alexander Liniger, Alexander Domahidi, and Manfred Morari, Optimization‐based autonomous racing of
1: 43 scale RC cars, 2015

8 - 54