Autonomous Driving in The Curriculum of Computer Architecture
Autonomous Driving in The Curriculum of Computer Architecture
Abstract—The need for educated personnel in automotive com- processing platforms and FPGAs combined. These hetero-
panies rises steadily due to the new trend towards autonomous geneous systems offer an increased performance per watt.
driving. Therefore, it is necessary to integrate the required topics Integrating these into the cars is the task of future engineers
into the curriculum of the university students. As a chair for
computer architecture, we added selected aspects of autonomous for which they have to be prepared.
vehicles to the lectures preparing skills in embedded hardware In addition to teaching certain aspects of the presented
and basic algorithm knowledge. Afterwards, the skills can be put new challenges in the standard curriculum of the chair, a
into practice by working in a newly created laboratory which more practical oriented student laboratory was opened. It is
offers the possibility to contribute to a project about the realiza-
intended to be an interdisciplinary community platform for
tion of an autonomous remote controlled car. Due to the different
requirements of the RC car, the laboratory is interdisciplinary in students around the main topic of software defined inverters
design. Overall, it was a huge success attracting many students (SDI) which are required to drive electrical engines. Hence,
who wanted to work on it. A presentation of the finished vehicle the laboratory was named SDI community lab.
to the public was the culmination of this work so far. Here, students of computer sciences, information and com-
Index Terms—autonomous driving, lectures, simulation, sensor, munication technologies, electrical engineering, computational
computer architecture education engineering and mechanical engineering can meet and work on
a project together. The first successful result of the laboratory
I. I NTRODUCTION
is a radio controlled (RC) car that was extended with different
Autonomous driving is the next big innovation in the field kinds of sensors and computing hardware. The result was an
of the automotive industry. Today’s vehicles already offer the autonomous RC car.
means for assisted driving. Nearly every major car company In this paper, we present how to integrate the technical
at least researches the technologies required or even has its aspects of autonomous driving into the students’ curriculum.
own driver assistance system allowing the vehicle to guide Due to the offer to incorporate these topics into a real RC car, a
through certain situations like traffic jams autonomously [1]. lot of students participated. They liked the concept of creating
A very prominent example is Tesla’s autopilot [2] capable their own parts of the vehicle. Additionally, companies like
of steering the car without driver interference but with driver Siemens and Infineon appreciated the additions to the courses
monitoring. Improving these systems requires new engineers since they can now expect new employees that are better
that are knowledgable in this field. That is why it is time to suited for future products. However, due to time constraints,
bring the concepts of autonomous vehicles into the courses of an overall knowledge of project management could not be
the university to prepare the students for the challenges of this imparted. Though, this can be acquired by lectures of other
new field. chairs. These are not covered in this paper. Showing scientific
The algorithms of autonomous vehicles require a huge work to the public is mostly very theoretical. Thus, an aim
amount of computing power. Data from very different sources was to present a working demonstrator of the autonomous RC
like cameras, radar, lidar, ultrasonic and IMU sensors has to be car to the public which was finally done. A significant topic
processed and fused. Due to the limited energy power budget for the daily life of the future like this is a major attraction to
of a car, the use of traditional processor and graphics cores the people.
is not advisable. With the move to electrical engines, saving The paper is structured as follows. At first, the acquired
energy becomes even more important because of the reduced competencies in regard to the project are presented. The
power density of batteries in comparison to fossil fuels. integration into the curriculum of the students is then further
Semiconductor companies like Nvidia already offer hard- elaborated. Finally, the presentation to the public is described,
ware platforms for developing autonomous vehicles like the a conclusion is drawn and the future plans are shown.
Drive PX series [3]. ARM provides the Mali C-71 [4]
and Imagination promotes their PowerVR architecture [5].
II. ACQUIRED C OMPETENCIES
However, car engineers always have to make a compromise
between the power consumption and the available computing The students working on the projects of the SDI community
power. Another approach could be the usage of specialized lab acquire different kinds of competencies which include
choosing the right hardware architecture for a certain task like
This work is supported by the Bavarian Research Foundation (BFS) as
part of their research project “FORMUS3 IC”. The SDI community lab was processing streamed image data or steering an engine, putting
sponsored by Siemens. different embedded devices (e.g. sensors) into operation, and
Demonstrator
Evaluations
Evaluations
ential GPS which is made available by an u-blox C94-M8P
Simulative
Concrete
Differential
RC Receiver RadarNode GPS
Processing Board
GPIOs
USB-To-UART Bridge
IMU Sensor
ZedBoard Raspberry Pi 3 RPLidar
CSI
Wifi
Fig. 2. A plan showing the different components of the RC car the students had to take care of.
platform is possible and allows the decision which components to desktop computers, additional battery packs were still
have to be deployed. necessary to achieve a longer period of time without having
The students had to configure and use processor simulators to switch or recharge them. They were connected to voltage
like QEMU [8] or gem5 [9] which is also used in the chair’s regulators to provide an energy source for the different kinds
research [10]. They obtained the knowledge on how they work of device which are deployed.
and also had the possibility to test it for the RC car use case. Finally, a synthetic plate was fixed to the RC car which
To run the algorithms on the simulator, they additionally had serves as a platform for the circuit boards. Sensors like radar,
to use cross compilers. Since these are also required for the the camera and the ultrasonic sensors were attached to the
real hardware, it is deemed to be a meaningful introduction edges of the plate while the lidar were centered and elevated
into the world of embedded hardware. a bit. The wires were connected directly without any further
It was also possible to evaluate the embedded hardware cable management.
with regards to the tools provided by the Robot Operating
System [11]. The graphical programs offered the possibility to D. Putting It All Together
simply visualize the data obtained from the different sensors.
An overview of the structure can be found in Fig. 2. The
Besides, the mavros plugin is used to allow communication
whole concept was created by the students working on the RC
via MAVLink which is well suited for micro vehicles [12].
car. They had to keep track of the different tasks required to
Data exchange between the different boards is done completely
finish the project. The main computing power comes from a
using this protocol.
ZedBoard with a Xilinx Zynq SoC that offers an FPGA and
Additionally, to verify the VHDL code written for the FPGA
an ARM Cortex-A9 dual core processor. In the overview, it
which was also used, a gate level simulation was realised.
can be seen as the big board in the center. Slightly right of
Since a hardware description cannot be simply debugged
it, a Raspberry Pi 3 with an ARM Cortex-A53 quad core
during the runtime, this is the only option to ensure the
processor is depicted. Additionally, a so-called TAPAS board
correctness of the design because it allows a direct inspection
is used which is a Software Defined Inverter and originally
of the different signals. Identifying invalid or unwanted states
gave the SDI laboratory its name. Its purpose is to drive
of the hardware design is much faster with this approach.
the engine with the requested amount of torque. Accessing
C. Electrical And Mechanical Components the ultrasonic sensors is done using another separate Arduino
The chassis and basic equipment like steering and electrical Uno. As previously mentioned, the expertise to program these
engine was taken from an off-the-shelf RC car. The first different devices was acquired by the means of simulation.
changes were adjustments regarding the control of the servo This preceding platform evaluation is also the reason, why the
motors and the main propulsion. It was necessary to address Nvidia Drive PX was only used as an development platform
these devices from our embedded platforms. The students had since the power consumption was much too high.
to figure out how to connect other integrated circuits without The ZedBoard is connected to the Arduino Uno via an USB
breaking the motors. Furthermore, the integrated radio receiver to UART bridge and thus has access to the information from
initially meant for driving the RC car with the remote control the ultrasonic sensors. Further inputs come from the remote
was modified. An emergency stop for safety reasons and a control receiver of the RC car, the IMU and the sensors of the
generic power on and power off functionality was implemented engines offering the information required for odometry. The
and could be sent from the remote control. Additionally, the servo motors responsible for steering the RC car are controlled
main speed was reduced tremendously to avoid destruction in from the FPGA.
case of inappropriate steering commands like driving into a Communicating with the Raspberry Pi is possible via an-
wall. other USB to UART bridge. The Raspberry Pi acts as a
Although the embedded hardware which is used to control connector to the outer world which is achieved by its integrated
the RC car does not need that much energy in comparison wifi antenna. This allows giving commands from a laptop and
time which is why it was not completely integrated into the CPU
composition. However, it is planned to complete this work Bachelor's and Master's Thesis SDI Projects and Internships
time-of-flight of the electromagnetic wave between a reflector algorithms on embedded hardware [6]. For this, images with
and the sensor. This time is proportional to the distance a resolution of 1242 x 375 pixels were used as an input
between the object and the sensor. By using more input data, it for different processing pipelines which were executed on an
is also possible to determine the Doppler frequency which can Raspberry Pi 3. Multiple filters like the Canny Edge, the Sobel
be used to obtain the velocity of the target. Finally, when using Magnitude, the Row Filter, and the Color Thresholding can
multiple antennas, the angle of the reflector can be determined, be applied before performing the actual lane detection. It is
too. also possible to apply multiple filters beforehand. Each filter
combination was evaluated in combination with different lane
B. Theses And Projects detection mechanisms like the partitioned Hough, the active
When working on theses and projects, the students can apply line modelling, a sliding window approach using histograms,
their knowledge acquired in our lectures to real world exam- a fixed window approach using histograms, and a random lines
ples. They can deepen their understanding and try out new algorithm. Since the best functional quality with regard to the
approaches. For this, they either select a task or topic given F-measure was reached by the random lines algorithm, it was
by a supervisor or they can think of something themselves. optimized for the embedded platform. Finally, an F-measure
The grades are given based on the final report or the created of 81.41% was reached with a frame rate of 54 fps.
thesis and the accomplished practical work. The projects are In cooperation with Infineon, a combined internship was
mandatory for study programs like Computational Engineer- accomplished. Students of Computational Engineering have to
ing, Computer Science and Information and Communication attend a mandatory internship which allows them to put their
Technique and are intended to let the students work on a larger skills into practice. As there is a direct access to the company,
topic themselves. Due to the different kinds of tasks a uniform they were able to easily fulfill this condition. This program
assessment system could not be realized and each work had to was a huge success for the students who were able to gather
be graded separately. Some of the students’ work is presented first experiences in a company and could earn some money as
in this section. well as Infineon who got already knowledgable employees for
In a Master’s thesis the HOG algorithm was optimized to two months.
achieve a higher speed on embedded hardware [14]. The basic A detailed overview about the tasks and topics which were
idea is to use data from other sensors with less computational successfully finished by the students can be found in Table I.
performance requirements to preselect areas of the picture They were partly conceived by students and partly given
which have to be investigated in detail. Only for these regions by the supervisors. A detailed development flow including
of interest, the actual HOG algorithm including the lookup requirements engineering and creating specification sheets was
using a Support Vector Machine is performed. For a self- not part of the overall project. The necessary work was defined
recorded set of images, it was possible to reach a speedup by the supervisors. This approach also prevents losing required
of 1.76 meaning 22 frames per second (fps) for images with knowledge about the whole project when students are finished.
a resolution of 1280x920 in comparison to the standard HOG Additional help was given by industry partners who also
algorithm performing only 12.5 fps on the Nvidia Drive PX briefly supervised the students in the laboratory.
2. This configuration even increased the recall for certain
scenarios because false positives are already eliminated be-
IV. R ESULTS
forehand by the preceding sensor. However, other and even
faster configurations decrease the recall and accuracy. With only minor supervision by the staff of the chair, the
A different approach of the sensor fusion was also imple- students managed to finish their work at the RC car. The
mented having a greater focus on combining the information final demonstrator was presented to the public at an event for
acquired by the single sensors instead of increasing the speed celebrating the 275th year of the existence of our university. At
of one single sensor. It combines the data from the HOG and a stand on the market place the RC car autonomously moved
the radar. Instead of just having single targets without more from one position to a target position while navigating through
information as they are delivered by the radar, determining a very simple parkour. A picture of the final presentation can
if a target is a person is also possible due to fusing the be seen in Fig. 4.
HOG algorithm. On the other hand, using the HOG algorithm The time without needing any recharged battery packs was
alone does not give an exact position of the found person approximately two hours meaning a power of 11.6W for the
which is improved by the means of the radar. Outside of the hardware without the engines. In comparison to the vehicles
laboratory, this algorithm was evaluated on an Infineon AURIX used by the participants of the Audi Autonomous Driving Cup
2G which is commonly used in cars [15]. It was possible to (AADC) [16] in which a team from the university participates
show that the processor is fast enough for algorithms that do this can be considered a huge success. Cars of the AADC
not require much computational power. Since the CPU also only last roughly half an hour with one battery pack and
offers many advanced safety features like lockstep execution, requires vaguely 57.7W just for computing. Additionally, since
even a sensitive sensor fusion that reduces the redundancy in a group of nearly twenty students were working on the RC car,
the input data can be run on the processor without the risk of they were able to gather experience in project management
volatile errors due to environmental impacts. and teamwork. Students of all study programs visible in
Another project done by a student and resulting in a Fig. 3 were participating (see Table I). Thus, interdisciplinary
paper proposal was an evaluation of different lane detection connections between the involved people could be forged.
TABLE I
T HE FINISHED TOPICS AND TASKS , SPLIT BY THE TYPE OF WORK , A BRIEF SUMMARY OF THE TASK AND THE STUDENT ’ S STUDY PROGRAM .
R EFERENCES
[1] N. A. Greenblatt, “Self-driving cars and the law,” IEEE Spectrum,
vol. 53, no. 2, pp. 46–51, Feb 2016.
[2] M. Dikmen and C. Burns, “Trust in autonomous vehicles: The case of
tesla autopilot and summon,” in 2017 IEEE International Conference on
Systems, Man, and Cybernetics (SMC), Oct 2017, pp. 1093–1098.
[3] A. Skende, “Introducing “parker”: Next-generation tegra system-on-
chip,” in 2016 IEEE Hot Chips 28 Symposium (HCS), Aug 2016.
[4] ARM Ltd. Mali camera. Accessed on June, 15th 2018.
[Online]. Available: https://developer.arm.com/products/graphics-and-
multimedia/mali-camera
[5] Imagination Technologies Ltd. Automotive. Accessed on June, 15th
2018. [Online]. Available: https://www.imgtec.com/markets/automotive/
Fig. 4. A photograph of the RC car when it was presented to the public. The [6] M. Reichenbach, L. Liebischer, S. Vaas, and D. Fey, “Comparison of
blue boxes represent an obstacle that was put in the way of the RC car. lane detection algorithms for adas using embedded hardware architec-
tures,” in Proceedings of 2018 Conference on Design and Architectures
for Signal and Image Processing (DASIP). IEEE, 2018, Porto, Portugal.
V. C ONCLUSION [7] S. Kohlbrecher, J. Meyer, T. Graber, K. Petersen, U. Klingauf, and
O. von Stryk, “Hector open source modules for autonomous mapping
Due to the increasing importance of knowledge in creating and navigation with rescue robots,” in RoboCup 2013: Robot World Cup
autonomous vehicles, an integration of this topic into the cur- XVII, S. Behnke, M. Veloso, A. Visser, and R. Xiong, Eds. Berlin,
Heidelberg: Springer Berlin Heidelberg, 2014, pp. 624–631.
riculum seems necessary. To achieve this, our chair included [8] F. Bellard, “Qemu, a fast and portable dynamic translator,” in Proceed-
small topics of this field into the standard lectures. Addition- ings of the Annual Conference on USENIX Annual Technical Conference,
ally, a laboratory was established that offers projects, theses ser. ATEC ’05. Berkeley, CA, USA: USENIX Association, 2005.
and voluntary tasks to create a fully working autonomous [9] N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu,
J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell,
vehicle based on an off-the-shelf RC car. It was possible M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood, “The gem5 simulator,”
to recruit students of many different study programs offered SIGARCH Comput. Archit. News, vol. 39, no. 2, pp. 1–7, Aug. 2011.
at the technical faculty which allowed a great exchange of [10] S. Rachuj, C. Herglotz, M. Reichenbach, A. Kaup, and D. Fey, “A
hybrid approach for runtime analysis using a cycle and instruction
knowledge which was also required to realize the car. accurate model,” in Architecture of Computing Systems – ARCS 2018.
Braunschweig, Germany: Springer, 2018, pp. 85–96.
[11] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs,
VI. F UTURE W ORK R. Wheeler, and A. Y. Ng, “Ros: an open-source robot operating system,”
Because of the huge success of the RC car project, further in ICRA workshop on open source software, vol. 3. Kobe, Japan, 2009.
[12] L. Meier and MAVLink contributors. Mavlink developer guide.
expansions of the laboratory are planned. For this, a tighter Accessed on June, 26th 2018. [Online]. Available: https://mavlink.io/en/
cooperation with student projects like High Octane aiming [13] B. Pfundt, M. Reichenbach, C. Hartmann, K. Häublein, and D. Fey,
for the Formula Student is intended. Additionally, due to “Teaching heterogeneous computer architectures using smart camera
systems,” in 2016 11th European Workshop on Microelectronics Ed-
student requests, a new project for autonomous quadrocopters ucation (EWME), May 2016, pp. 1–6.
is started having similar aims like the presented RC car project. [14] D. Quick, “Performance optimization of the hog algorithm utilizing a
Furthermore, a new study program focus for information and multisensor system,” 2018, Master’s Thesis.
communication technique with autonomous driving as the [15] S. Rachuj, W. Bauer, and D. Fey, “Evaluation of a sensor fusion
algorithm on a real-time processor,” in AmE 2018 - Automotive meets
main topic is going to be integrated into the curriculum. Electronics, M. Wahl, Ed. VDE, Mar 2018, pp. 44–48.
With these new offers an even better education for the new [16] Audi AG. Audi Autonomous Driving Coup 2018. Fetched on
challenges of the automotive industry is provided. 2018-09-03. [Online]. Available: https://www.audi-autonomous-driving-
cup.com/?blogPage=1