1

EECS 4314: Advanced Software Engineering

Assignment 2: Openpilot’s Concrete Architecture Report

Group: Beyond

Authors

Student No. Name of the Author Author’s email

217537739 Nishaant Dhingra [email protected]

218550111 Abdirazak Yusuf [email protected]

218305847 Hashim Khan [email protected]

218452805 Kawshar Patel [email protected]

217692690 ​Mahdiar Shoraka [email protected]

217847328 Raj Panjabi [email protected]

218908368 ​Ryan Lyn [email protected]

217554668 ​Sanchit Duggal [email protected]

Table of Contents

Abstract.............................................................................................................................2
Introduction....................................................................................................................... 2
System Overview and Top-Level Subsystems..................................................................3
Interactions between subsystems.....................................................................................4
Derivation Process............................................................................................................6
Architectural Styles........................................................................................................... 7
Assigned Subsystem........................................................................................................ 9
Conceptual vs Concrete Architecture and Modifications................................................ 12
Use Case........................................................................................................................ 13
Lessons Learned............................................................................................................ 14
Conclusion...................................................................................................................... 15
References..................................................................................................................... 15
 2

Abstract

This report presents the as-built architecture of the Openpilot, focusing on the concrete
architecture of one of its top-level subsystems. The subsystem in focus is logging and
miscellaneous services of Openpilot. The report outlines the methodology followed to
derive the architecture, including subsystem identification and containment file creation.
It highlights architectural styles, design patterns, and notable interactions within the
subsystem. Furthermore, the report compares the concrete architecture with the
conceptual architecture of Openpilot, providing insights into the evolution and
implementation of the subsystem. Lessons learned from the architecture development
process are discussed as well, providing valuable insights for the future. The utilization
of diagrams is involved within this report, dependency diagrams have helped us derive
the concrete architecture and the architectural styles. Additionally, the report includes a
sequence diagram to provide insight into one of the salient features of the openpilot
driving model. Moreover, the report’s main goal is to provide a thorough understanding
of the internal workings and design components of Openpilot’s logging and
miscellaneous services subsystem.

Introduction

Openpilot is one the top technologies in autonomous driving, rapidly evolving due to the
continuous development of the open source community and Commai. Through an
in-depth analysis, the report will contain various information on the concrete architecture
of Openpilot. The report's main focus is on the logging and miscellaneous services
subsystem of Openpilot. The subsystem consists of numerous components that make
up its core functionality that ensures a smooth operation. For instance, the logger holds
a vital role in the software’s architecture considering it is in charge of recording routes
gained from Openpilot’s usage. By dissecting this subsystem’s architecture, we gain
valuable insight into how Openpilot handles essential functionalities such as data
logging, thermal management, communication using websockets, and user interface
interactions.

Moreover, Openpilot’s concrete architecture comprises a tremendous number of Python

scripts that facilitate sensor data collection, processing, and neural network
decision-making. The architecture is built upon various styles including implicit
invocation, layered, and process control approaches. Overall, the subsystem in focus
and the techniques used to build software systems such as design patterns,
architectural styles, and other aspects have been used to create an effective product. A
product that has been used as an efficient driver assistance system for many users out
there. The conceptual architecture was used as a guide to derive the concrete
architecture; having a conceptual understanding of the subsystems and components
has made it easier to create an accurate and complete report. Additionally, the whole
process of deriving concrete architecture has made us learn several key lessons.
 3

System Overview and Top-Level Subsystems

Openpilot uses over 300 Python files divided into different categories such as
submodules and dependencies for the system to be functional. In simple terms,
openpilot's core system functionality involves reading the data from the sensors
(camera, IMU, steering wheel angle sensor, GNSS receiver, etc.), processing that data,
and relaying relevant inputs to a neural network. The model achieved by the neural
network and its hyper-parameters produce outputs that are translated to actions and
utilized to apply commands and adjustments to the actuators. For openpilot to work
smoothly, the following open-source dependencies are used:

1. opendbc: CAN bus library

2. panda: The main interface between the car and openpilot
3. cereal: Publisher-Subscriber messaging specification for robotic systems

Openpilot has 5 main subsystems in addition to the dependencies mentioned above. In

this section of the report, we will talk about 4 of the subsystems which were not our
subsystems of choice (assigned subsystem), but still are important to talk about, as they
give us a holistic view of the concrete architecture of openpilot and how it works as a
homogeneous system. [1],[2]

Sensors and Actuators: It is the main interface between cereal and I/O data. Based on
our derived concrete architecture, It consists of 3 main subsystems (modules). Boardd
It is the manager of the interface between cereal and panda. It reads the data obtained
by the sensors and the peripherals (which are managed by panda) including the GNSS
module, infrared LEDs (for night-time driver monitoring), and the car. Camerad It is the
module responsible for managing the camera data received from both the road and the
driver’s camera. It writes the data directly to visionipc in cereal. It also has auto-focus
and auto-exposure functionalities built-in. The third module is sensord which reads and
configures the remaining sensors including gyro, accelerometer, and light sensors.

Neural Network Runners: This subsystem is responsible for all of the Machine
Learning processes in the system including reading input data, data preprocessing, and
feeding the appropriate set of data into the Neural Network model. Most importantly, It
contains the core logic of the system. Based on our concrete architecture derivation, it
consists of two main modules. Modeld reads the image stream from cereal into the
main driving neural network. Modeld uses this data and the driver’s input to predict the
appropriate driving actions. Dmonitoringd Contains the logic for the driver’s interaction
with the system and the car (e.g., taking over and engaging the system). It also contains
a service for alerting the driver when necessary. This service works based on the
outputs of the dmonitoring model, and other parameters.

Localization and Calibration: It is responsible for the location processing of the car
and calibrating the parameters regarding the position and movement of the car. Based
on our derivation of the concrete architecture of this subsystem, It includes three main
modules. The first module is locationd which is responsible for localizing the car and
 4

describing the interactions with its environment (e.g., the car's speed, if the car is going
up or down). Locationd receives the pre-processed data from Kalman Filter which filters
the noisy data from the sensors. Furthermore, it includes two important files,
calibrationd (image normalization) and paramsd (contains specific car model
parameters). The second important module is ubloxd which has one important job, and
that is parsing the GNSS data which is later used for localization. The third service is
rednose which includes Kalman Filter libraries.

Controls: It is responsible for tasks related to controlling the actuators of the car. In
simple terms, this module is the module that controls and drives the car. We derived
three main modules for this service from the concrete architecture. Radard converts
raw data received from different car models to a canonical format. Plannerd contains
the planning logic which consists of two parts: Lateral (steering) planning and
longitudinal (gas and brake) planning. They both use an MPC solver which calculates
the appropriate mechanical parameters for the ride to be correct and smooth given its
final position and destination in the upcoming few seconds. The third module is
controlsd has two main jobs: 1- It contains the service that controls the car. It receives
the plans mentioned above from plannerd and converts them into appropriate control
signals. 2- It parses the CAN data from the car to a canonical format which is then
published to cereal. [2]

Interactions among the subsystems

Figure 1 illustrates the communication pathways between various subsystems within the
Openpilot system. These subsystems include Neural Network Runners, Localization
and Calibration, Controls, Sensors and Actuators, and System, Logging, and
Miscellaneous Services. Each arrow in the diagram represents the exchange of specific
data between the corresponding subsystems.

The rationale behind these interactions lies in the distributed nature of the Openpilot
system and the specialized functionalities of each subsystem. For instance, Neural
Network Runners are responsible for tasks such as camera odometry and driving
decision predictions, generating essential data for other subsystems to make informed
decisions. The exchange of data between Neural Network Runners and Localization
and Calibration enables accurate localization of the vehicle and calibration of sensors,
contributing to safer and more precise navigation.

Similarly, the interaction between Neural Network Runners and Controls facilitates the
transmission of driving decision predictions and driver monitoring state to control the
vehicle's behavior effectively. Controls, in turn, provide feedback in the form of car state
and control state to ensure smooth operation.
 5

Sensors and Actuators play a crucial role in gathering real-time data from various
sensors installed in the vehicle, such as road cameras and GPS receivers. This data is
then utilized by different subsystems for decision-making and system monitoring. The
bidirectional communication between Sensors and Actuators and System, Logging, and
Miscellaneous Services ensures efficient data exchange for logging purposes and
system maintenance.

Furthermore, the interaction between Localization and Calibration and Controls involves
the transmission of location and calibration data to inform control decisions. This
integration enhances the overall performance and accuracy of the Openpilot system.

Overall, the rationale for these interactions lies in the need for seamless data exchange
and coordination between different subsystems to achieve the objectives of
autonomous driving, including safety, efficiency, and reliability. By leveraging the
specialized functionalities of each subsystem and facilitating data sharing, the Openpilot
system can effectively navigate diverse driving scenarios and ensure a smooth driving
experience for users.

Figure 1: The communication pathways between various subsystems within the Openpilot
system
 6

Derivation Process

To derive the concrete architecture, we used conceptual architecture as a guide to see

what subsystems exist and what are the components of these subsystems. The
conceptual architecture helped us when examining the system implementation to form
the concrete architecture as it provided an initial system structure.

The first thing we did was to download the source code of the openpilot system and run
it through a software tool called Understand which allows us to visualize dependencies
between different parts of the code. Understand generates a dependency graph
according to the directory structure. Though it was very insightful we still needed to
reorganize it according to our subsystems and view the dependencies between them.
So we manually assign modules to each subsystem they belong to based on the 5 main
subsystems we identified along with the components they are made up of mentioned in
the last section. After iteratively assigning all components to the main subsystems we
get our concrete architecture with subsystems interacting with each other.

Figure 2: Concrete Architecture of openpilot

 7

Figure 2 shows all the subsystems that make up the openpilot broken down into their
constituent subsystems. This also allows us to see the dependencies between the
different subsystems. Even though panda, opendbc, and cereal are not categorized as
subsystems they are added in the architecture independently to make their role more
clear as well as see the architectural structure of the system more evident. One of the
most important things highlighted by the concrete architecture is that all the subsystems
interact with cereal and not with each other directly portraying the implicit invocation
architecture style that is discussed in detail later. Another noteworthy aspect is that the
hardware that is made up of the car body has a bidirectional dependency with the
panda since the car’s sensor data is relayed over to the cereal through the panda, and
control data for actuators is passed onto the car hardware through panda as well from
cereal.

Architectural Styles

A variety of architectural styles are necessary for Openpilot, to function properly and
consistently. Openpilot uses several critical architectural techniques to assist it carry out
its complex operations, which span perception, planning, and control. Let's explore the
architectural styles we concluded for openpilot:

Process Control Architectural Style: At the core of openpilot's functionality lies the
process control architectural style, particularly emphasizing feedback loops (see Figure
3 for visualization). Openpilot constantly monitors sensor data from the car’s hardware,
and that data is sent to the machine learning model (Neural Network Runners) where all
the decision planning takes place. The machine learning model determines the
instructions that have to be given to the vehicle’s actuators to decide the speed and
direction. When the machine learning model finalizes its computation on the sensor
data, it sends the instructions to the actuators of the car to adjust the car’s movement.

Advantages of this style – it helps in the continual improvement of the openpilot

software system and makes driving a safer and better experience.

Figure 3: Working of Process-Control Architectural Style in openpilot

 8

Layered Architectural Style: Openpilot adopts a layered architectural style to organize

its software components into distinct layers. We interpret this architectural style in
openpilot as a demonstration of the seamless transition from hardware to software
within the system. This architectural style is seen with 4 components collectively,
namely – hardware of the car, panda, opendbc, and cereal. The top layer (cereal) is
responsible for communicating with all the software subsystems of openpilot, the middle
layer (panda and opendbc) acts as an interface between the software subsystems and
the hardware of the car, and finally, the lowest layer (car’s hardware sensors) is
responsible for reading visual input data using the sensors. Figure 4 shows the layered
architecture style we interpreted openpilot to have by studying its dependency diagram.

Figure 4: Dependency diagram of openpilot showing the layered architectural style

Implicit Invocation Architectural Style: This is arguably the most important and
prominent architectural style employed in openpilot software. It is specifically a
publish-subscribe (pub-sub) architectural style. It is observed that every software
subsystem in openpilot uses this for their modules/services to communicate with the
modules/services of other software subsystems. Every software subsystem is
subscribed to Cereal, which is a publisher that helps individual software subsystems
communicate with each other efficiently (see Figure 3).

Pub-sub architecture focuses on topic-based communication, where the subscribers (all

software subsystems subscribed to cereal) subscribe to topics aligned with their
functionality, ensuring that they receive relevant data only (i.e. data related to their
tasks). Advantages of this style – scalability, flexibility, and modularity. Scalability – as
new features are added, they can simply subscribe to cereal and have efficient
communication with other software subsystems. Flexibility – gives components the
ability to subscribe or unsubscribe based on their needs. Finally, modularity – allows
processes to communicate asynchronously with other processes that are not
necessarily implemented in the same language. To visualize this better, see Figure 3
 9

which shows how each software subsystem is subscribed to cereal for communicating
with each other instead of communicating with the software subsystem on their own
directly.

Figure 5: Dependency diagram of openpilot showing the pub-sub architecture style

Assigned Subsystem

The software subsystem we chose is System, Logging, and Miscellaneous Services.

It is made up of 5 main internal subsystems namely logging, UI, athena, thermald, and
manager. All these subsystems do not interact with each other and perform their
specific functions independently. However, all of them interact with other outer
subsystems using cereal exhibiting Implicit Invocation architecture discussed earlier.

Figure 6: Interaction of System and Logging with other subsystems

 10

The functionalities and interactions with other subsystems of the 5 internal subsystems
are explained below:

logging: Openpilot utilizes a logging system to continuously log camera and sensor
data that is used for training neural network models for enhancing path-predicting
capabilities. The streams from road cameras, wide road cameras as well as driver
cameras are saved in low-resolution versions so that they are small enough for instant
upload on slow internet connections yet remain valuable for analysis and debugging
purposes. All cereal messages exchanged among all subsystems, and system crash
reports are also logged in a serialized capnproto format so we can understand and fix
failures.

thermald: This subsystem manages thermal conditions and hardware states of the
whole system. It defines optimal thermal bands and hardware states and functions to
handle tasks such as reading thermal data, setting off-road alerts, and monitoring
hardware states. Two main threads are spawned to monitor hardware states, manage
thermal conditions, and send status updates to other components. Overall, it ensures
the system operates within safe temperature ranges and manages hardware states
effectively.

athenad: This component serves as a WebSocket client that manages communication

with an external server of OpenPilot. This subsystem deploys a client-server
architecture style where athenad is the client and openpilot is the server deployed at an
accessible URL. Its core functionalities include establishing a WebSocket connection
with the server and handling sending and receiving messages over this connection. It
implements methods to handle various types of messages received from the server,
such as echoing messages, retrieving system information, setting navigation
destinations, etc. It manages an upload queue for files to be uploaded to the server.
Files are uploaded asynchronously, and the script handles retries in case of upload
failures. Several background threads are spawned to handle different tasks, such as
managing WebSocket communication, uploading files, handling logs, and monitoring
system statistics. It includes error-handling mechanisms for handling exceptions during
WebSocket communication, file uploads, and other operations. It also implements a
retry mechanism for reconnecting to the server and retrying failed upload attempts.
Overall, athenad subsystem facilitates communication between the openpilot system
and an external server, allowing for various interactions such as data exchange, file
uploads, and system management.

UI: This component renders the graphical user interface (GUI) on the comma3X device.
While the car is off, it contains a training guide to onboard new users, shows system
status, and exposes some settings. While the car is on, the road-facing camera stream
 11

is shown with overlaid visualizations of the driving path, lane lines, and lead cars. It
subscribes to messages related to device state and control state using
messaging.SubMaster. It defines an update function, periodically invoked by a Timer, to
dynamically adjust the GUI based on received messages about the car's current
location and navigation updates. This function determines the vehicle's on-road or
off-road status, updates label backgrounds and text content based on the control state,
and regulates screen brightness and backlight power as per the vehicle's condition.
Overall, this component provides a real-time interface to display vehicle control state
and alerts to the user.

managerd: This component is a crucial part of the Openpilot system, responsible for
managing various processes and initializing the environment. The manager_init()
function initializes parameters, registers the device, sets up logging, and prepares
processes for execution. The manager_cleanup() function stops all processes and logs
a message indicating the cleanup process. The manager_thread() function runs
continuously, updating the status of managed processes based on the device state and
ensuring necessary processes are running. Signals such as SIGTERM are handled to
ensure proper cleanup and shutdown procedures. Exceptions are caught and logged,
and appropriate actions are taken. If an uninstall, shutdown, or reboot is requested, it's
logged and executed. The main() function initializes the manager, starts the main loop,
handles exceptions, performs cleanup, and executes requested actions. Overall, this
script serves as the backbone for managing processes, handling exceptions, and
ensuring the smooth operation of the Openpilot system.
 12

Conceptual vs Concrete Architecture and Modifications

The conceptual architecture details high-level subsystems that are listed within
documentation as well as major components for the system to function. These are
highlighted either upon conception of the project or when detailing the main
functionalities of the project. The conceptual architecture highlights a blueprint of what
components must exist for the project to fulfill its goals so often it will not go into detail
on how each component will work or interact.

The extraction of the concrete architecture grants the ability to compare the conceptual
architecture with the concrete. This often is done to highlight dependencies between
components and subsystems or to compare the architectures for deviations or
absences between them. Over the lifespan of a project, especially ones that are
community-developed and open source will have many changes to optimize speed, add
functionality, remove redundancy, and fix bugs. In the case of openpilot various
components and tools were refactored over time or their functionalities were merged
creating some differences.

When comparing the conceptual architecture to the concrete the main difference is the
large dependency on a component called Cereal, a central communicator between
multiple components. Based on the conceptual architecture it is implied that data flows
linearly between subsystems, such as from neural network runners to controls to
actuators. In reality, a majority of communication is dependent on cereal where they
notify cereal to publish a packet of information which components can interpret for their
own actions. This was most likely a design decision made by a senior developer to
implement a strong implicit invocation architectural style as the framework for
communication.

Another component of comparing conceptual to concrete is the differences between

them, the absence of components, tools, or modules, and the addition of new ones.
Certain components such as dmonitoringmodeld were noted in the documentation and
had a specific task, but in the transition to concrete or through various updates over the
year, it was refactored. Now the component no longer exists and its functionality was
absorbed into dmonitoringd signifying that it was likely redundant in the eyes of
management. This refactoring and updating also applies to tools used by the system,
Laika is a tool developed by the openpilot team but over time its functionality was
absorbed into a component called ubloxd, and full references to the tool are gone. The
tool was noted in a blog post in 2021 and a follow-up post in 2022 but since then it has
been refactored due to being redundant [2] [3].
 13

Use Case

Scenario: A user is on a road trip from one city to another. The road trip comprises
multiple stops and long hours of driving. The user is excited to embark on this journey
but also a little worried as it is their first time driving this far and of all the challenges that
the road trip poses. Since it's their first time, the user seeks assistance while driving on
the highway.

Solution: The user can activate the Openpilot system in their car, which includes a
robust feature of lane maneuvering. Openpilot employs a series of intricate processes to
ensure safe and efficient navigation where the system utilizes sensors and cameras to
monitor the external and internal conditions to maneuver along the lanes on a highway.

Figure 7: Sequence diagram of the lane-changing

A detailed explanation of how Openpilot’s Lane Changing feature works at a high level:

● Sensors scan the environment and collect data, which is then interpreted by
Panda.
● Panda plays a crucial role as it articulates the vehicle's CAN bus traffic and
efficiently interfaces with OpenDBC where the data undergoes further processing
before being transmitted as messages to Cereal.
● Cereal serves as the main central hub, which orchestrates the flow of
information throughout the Openpilot system.
 14

● From Cereal, the data is shared with the Localization component, where the
state of the vehicle is accurately described.
● With the localized information in hand, Cereal then sends the calibrated values to
the Neural Network which is tasked with predicting the optimal driving path.
● Once the prediction is made, it is shared back with Cereal.
● After having a refined trajectory, Cereal now collaborates with the Control
module.
● Control takes the responsibility of initializing actuator messages, which are
important in translating the planned trajectory into actionable commands.
● These commands are once again shared back with Cereal.
● Cereal finally sends the information to the hardware, ensuring that the planned
lane-changing maneuver is executed seamlessly by integrating both external and
internal conditions.

Lessons Learned

Throughout the whole process of deriving the concrete architecture based on the
understanding of the Openpilot system, there were several lessons learned. It was
evident that a clear grasp of the conceptual framework was necessary in order to
maintain being on a proper course and avoid misinterpretations and inconsistencies.

One of the key assets that played an important role throughout the derivation of
concrete architecture was Understand. Due to Understand’s amazing analysis
capabilities, it offered a lens into the system's implementation, helping in the extraction
of dependencies and visualization of relationships between entities. However, it was
soon discovered that although Understand offered a solid foundation, it did not eliminate
the need for a manual intervention that is required to check the architecture’s alignment
with conceptual needs.

Manual hierarchical decomposition emerged as a key requirement for the architecture

derivation process. Regardless of the features provided by an automated tool, human
intervention was still required when discerning the appropriate hierarchy and subsystem
assignments. Thus revealing a balance between automated and manual derivation that
is required in the process of concrete architecture derivation. Automated services like
Understand, streamline the extraction process and manual human oversight ensures
the accuracy and completeness of the extraction, thus working in unison.

Deriving the concrete architecture helped us learn and understand the deeper insights
into the Openpilot system's organization and interdependencies. These insights helped
in learning and understanding the architectural styles implemented and potential areas
for optimization and future enhancements.
 15

Conclusion

In conclusion, Openpilot served as a great learning experience when diving into its
architecture and offered several key findings that offered insights into the system's
structure and operation. These findings helped us understand Openpilot's functionality
and implications for its overall performance. Openpilot has been identified to have 5
overarching subsystems, the ones that have been highlighted in this report are the
System, Logging, and Misc Services which serve as the backbone of the system’s
functionality.

Using conceptual architecture as a guiding framework allowed the derivation of concrete

architecture. Using the roadmap of the subsystems and components that conceptual
architecture provided, it was easier to use tools like Understand and ensure the
accuracy and completeness of extraction when understanding system implementation,
dependencies, and relationships between different modules. The most striking deviation
from the conceptual architecture to concrete architecture was the significant
dependency on Cereal which served as the main hub for transferring messages
between different modules. Three main architectural styles were identified throughout
the system - Process-Control, Layered, and Publish-Subscribe. These architecture
styles allow the Openpilot system to make real-time adaptation to changing road
conditions, efficient communication, and interactions between subsystems enabling
modular design and asynchronous interactions. The System, Logging, and Misc
Services subsystem plays a crucial role in managing various aspects of the system,
from logging route data to managing thermal conditions to facilitating communication
between the subsystem and external servers.

Thus, discovering Openpilot’s concrete architecture derivation provided a great set of

insights, from identification of subsystems to interaction between subsystems to
understanding architecture styles and overall allowing for future refinement and
optimization of the system through the conceptual and concrete architecture.

References

[1] From vision to architecture: How to use openpilot and live. From Vision To
Architecture: How to use openpilot and live - DESOSA 2020. (2020, March 11).
https://desosa.nl/projects/openpilot/2020/03/11/from-vision-to-architecture

[2] How openpilot works in 2021. comma.ai blog. (2021, October 12).
https://blog.comma.ai/openpilot-in-2021/

[3] Enhancing live localization using Laika. comma.ai blog (2022, August 3).
https://blog.comma.ai/laikad/