Des Autom Embed Syst

DOI 10.1007/s10617-016-9178-0

Testing real-time embedded systems using high level

architecture

Jose Claudio V. S. Junior1 · Alisson V. Brito1 ·

Luis Feliphe Silva Costa1 · Tiago P. Nascimento1 ·
Elmar Uwe Kurt Melcher2

Received: 19 February 2016 / Accepted: 13 August 2016

© Springer Science+Business Media New York 2016

Abstract This work proposes an environment for testing of heterogeneous embedded sys-
tems by means of distributed co-simulation. The test occurs in real-time, co-simulating the
system software and hardware platform using the high level architecture (HLA) as a mid-
dleware. The novelty of this approach is not only providing support for simulations, but
allowing the synchronous integration of heterogeneous simulators with a physical real-time
environment. In this work we use the Ptolemy framework as a simulation platform. The
approach of co-simulation based on HLA allows different simulators and physical devices
to inter-operate, for example robots can operate and be tested with a Ptolemy simulations.
Case studies are presented to prove the concept, showing the successful integration between
Ptolemy and the HLA and test systems using Robot-in-the-loop. The integration of real-time
devices through robot operating system was also presented. The approach proved to be able
to detect logical and physical design flaws.

Keywords Testing · Embedded systems · Robot-in-the-loop · High-level architecture

B Alisson V. Brito
[email protected]
Jose Claudio V. S. Junior
[email protected]
Tiago P. Nascimento
[email protected]
Elmar Uwe Kurt Melcher
[email protected]
1 Center of Informatics, Federal University of Paraiba, João Pessoa, Brazil
2 Department of Systems and Computers, Federal University of Campina Grande, Campina Grande,
Brazil

123
 J. C. V. S. Junior et al.

1 Introduction

Designing complex and reliable systems requires effective approaches for validation and test.
One such approach is Hardware-in-the-loop (HIL) simulation, which means adding the real
device in the simulation loop. This can enhance the quality of testing, reduce design time and
anticipate the interaction with the environment. In spite of its relevance, combining different
architecture designs and protocols in the same joint simulation is always a hard task, even if
some methods exist to assist designers with hardware and simulator integration [3].
High level architecture (HLA) standardized in IEEE 1516-series [9] was developed to help
reducing this gap, providing a feasible way of supporting interacting simulations through a
common architecture for distributed modeling and simulation. HLA interoperability can
also be advantageous to co-simulate hardware devices [23]. In practice, any simulator or
hardware device using HLA for communication and synchronization can be integrated in a
unique global simulation platform, turning simulation integration easier and more effective.
Systems that combine components from different domains are called heterogeneous. Test-
ing heterogeneous systems requires validating modules that were developed separately as
well as their integration. Because of this heterogeneity in embedded system design, testing
becomes an increasingly difficult task. According to [4,17] testing is one of the major prob-
lems within the development of hardware projects. It is estimated that about 70 % of the
resources in a hardware design are spent in this phase [5].
Real-time embedded systems have a close interaction with their environment and strict
time constraints. Thus, testing these systems requires even more complex techniques such as
formalisms to check the system against mathematical models or with a formal specification
[1].
Due to its close coupling with the environment, an RTES normally includes different com-
ponents with heterogeneous models of computation (MoC). Thus, tools with high abstraction
power are necessary in order to model, simulate and test all such MoCs, e.g., finite state
machines (FSM), synchronous data flow (SDF), discrete events (DE) and continuous time
(CT). The Ptolemy II framework [18] is an example of a simulation and modeling tool
intended to support system designs that involve combining different MoCs.
This work proposes an environment for testing of heterogeneous embedded systems
through distributed co-simulation. The tests occur in real-time co-simulating the system soft-
ware and hardware platform using the HLA as a middleware. The novelty of this approach is
not only providing support for simulations, but also allowing the synchronous integration of
heterogeneous simulators with any physical real-time robots. In this work we use the Ptolemy
framework as a simulation platform. In the experiments, robots are used as case study of a
typical real-time system.
The HLA standard was used to provide inter-operation and synchronization between the
embedded system and the models in Ptolemy. As proof of concept two different experiments
were conducted. First, the testing of a navigation algorithm embedded in a simple robot:
xperiments demonstrated the synchronization and flexibility of the approach when different
MoC are integrated in the same simulation. In a second experiment, the integration of real-
time devices through robot operating system (ROS) was presented. In this experiment, real
robots were evaluated in robot-in-the-loop simulation cooperating with virtual robots. Both
logical and physical failures were detected in the test and were able to be solved. This
work can be easily extended to test multiple interacting embedded systems, even if they are
geographically distributed, or real-time devices, monitors and controllers.

123
Testing real-time embedded systems using high level architecture

The remainder of this paper is organized as follows. Section 2 discusses the related work.
The proposed testing technique is presented in Sect. 3, while the proof-of-concept application
is described in Sect. 4. Section 5 presents the experimental results and finally, Sect. 6 puts
forward the concluding remarks.

2 Related works

The fundamental principle of co-simulation is the support for cooperative execution of dif-
ferent simulators [22]. Such simulators can be characterized by a computer system, hardware
resources or a real part of a system. These simulated environments can be homogeneous or
heterogeneous and can be used to perform testing of embedded systems.
The authors in [7] presented a methodology for functional verification of hardware through
parallel co-simulation. The work uses “thread” and “handshaking” provided by the Verilog
programming language interface (PLI) functions to ensure that the co-simulation will be
correct. Parallel threads are inserted, which are autonomous with respect to the execution flow
of the simulator of hardware description language (HDL) threads, enabling data exchanging
among the simulations simultaneously. The synchronization among hardware and software
is made using handshakes through the application program interface (API) provided by the
PLI of Verilog, combined with specific synchronizations functions.
The authors in [19] presented an environment for verification and testing of hardware
using HIL. The work focuses on the integration of Matlab Simulink with HIL and presents a
synchronization of the simulation models using real-time communication. For this purpose,
the computer running Simulink performs the control logic to synchronize with real-time.
These two approaches differ from ours in terms of generalization. As we use HLA standard
for coupling simulators and devices, our approach can be easily generalized to any other
simulation framework that supports this standard. Extension to frameworks that do not support
HLA can generally be done, too, requiring the implementation of an adequate software
interface.
The work in [22] presents a distributed co-simulation environment for integrating vir-
tual components. The co-simulation uses a modified version of HLA named distributed
co-simulation backbone, which is a proprietary standard for data exchanging, requiring
explicit calls to functions of run-time infrastructure (RTI). In general, the operation is similar
to HLA, i.e., a federated co-simulation comprising autonomous components possibly written
in different languages. Despite the similarity with HLA, it is not IEEE 1516 compliant, which
may turn the integration with other HLA compatible simulations a hard task.
Other works use HIL with the purpose of testing embedded systems, but they are restricted
to a limited set of models and devices¡falta citao¿, with specific architectures and MoC, thus
making integration harder. On the other hand, there are works that perform co-simulations
of homogeneous models, only, which do not lend themselves to an easy extension to hetero-
geneous systems given the significant effort required for the adaptation to different hardware
platforms, communication and synchronization protocols.
An approach to connect distributed robots with simulated entities is presented in [13]. The
authors also integrated ROS with high-level architecture (HLA). The focus of that work is
not testing, but the distributed coordination of real-time robots. Thus, no embedded system
design tool was integrated to test the execution of the robots in real-time.
In this work we aim specifically at supporting heterogeneity and synchronization for
testing systems involving different MoC and for this purpose we integrate Ptolemy with real-

123
 J. C. V. S. Junior et al.

time robots using HLA. This is a step forward with respect to other works in a similar line,
such as the integration of multiple instances of Ptolemy for performance improvement [6],
which did not consider robot-in-the-loop, and the work in [10] addressed the integration of
Ptolemy with hardware devices for testing purposes, but without real-time concerns.

3 High-level architecture

HLA is a general-purpose architecture defined by Defence Modelling and Simulation Office to

support reusability and interoperability using a wide number of different types of simulators,
which are maintained by U.S Department of Defense (DoD) [9]. The standard defines this
architecture having three parts: the first explore the framework in a general way and its
main rules, the second deals with the interface specifications between simulators (HLA), and
the third deals with the specification model of data (OMT) which are transferred between
simulators. The HLA has a time managing service to synchronize data between heterogeneous
models. The mains goals of HLA are to make possible the inter-operation of distinct models
and reuse them when necessary to provide a distributed simulation environment for systems
that need large scale computing.
The principal idea of HLA is to separate the specific functionality of each simulator using
a general purpose infrastructure. Each simulator needs to use the RTI to communicate to HLA
and other simulators. The RTI is responsible for the specific structures of each simulator to
interface with the global structure of HLA. Each simulator that is connected to one RTI is
called a Federate. The set of all federates managed by one RTI is called a Federation. In cases
of geographically distributed simulations, it is also possible to have many Federations in a
same simulation environment.
Data formats are defined by the federate object model (FOM), which is based on Object
Orientation concepts. Classes and objects are defined and federates can publish and subscribe
to any objects of that classes. Any publication at an object of the federation will result in
updating of these objects at the federates that subscribe to this object.
Analogous to how a distributed operating system provides services to applications, the
RTI provides services to federates. These services are used, basically, for synchronizing the
federates and control the data exchanging between them. These interfaces are arranged into
seven basic service groups: Federation management, declaration management (DM), Object
management, Ownership management, Time management, data distribution management
and support services (refer to [9] for details).

4 Embedded system design with Ptolemy

Due to its close coupling with the environment, Embedded Systems normally include different
components with heterogeneous MoC. Thus, tools with high abstraction power are necessary
in order to model, simulate and test all such MoCs, e.g., FSM, SDF, DE and CT. The Ptolemy
II framework [18] is an example of a simulation and modeling tool intended to support system
designs that involve combining different MoCs.
In Fig. 1 the main components of Ptolemy’s GUI, called Vergil, are presented. The Director
manages the simulation according to its MoC. There are two kinds of actors. The Atomic
Actors are the indivisible ones and the Composite Actors can aggregate other actors or
complete designs (also with different MoC) inside them. These hierarchical actors are the

123
Testing real-time embedded systems using high level architecture

Fig. 1 Example of Ptolemy GUI [18]

Fig. 2 General architecture of testing with HIL

strategy of Ptolemy to incorporate heterogeneous MoC in a same simulation model. The

actors exchange data among them (called Tokens) via Ports and Relations.

5 Proposed testing technique

Traditional approaches to testing with HIL basically consist of injecting a set of input stimulus
into the hardware for which the expected outputs are known in advance, thus allowing to
compare the results generated by the embedded system with the results from the reference
model [25]. Figure 2 shows the general architecture for testing Embedded Systems with
HIL. The simulator is responsible for generating input stimulus into the embedded system.
On the next step, it checks whether the output data from the hardware is as expected. It is also
important to observe that external stimuli from the environment can influence the embedded
system. This influence can be captured by the simulator through the system outputs. In the
case of a real-time simulation, as in this case, the physical time must also be taken into
consideration.
Concepts of Universal Verification Methodology (UVM) [2] were used for testing. In
particular, a modified Testbench Model was implemented within Ptolemy and integrated
with HLA. The UVM is a standardized methodology for verifying integrated circuit designs.

123
 J. C. V. S. Junior et al.

Fig. 3 Testbench example developed with Ptolemy

It has become the standard for verification of integrated circuits design with support from
many companies and research centers.
Ptolemy provides the capability of modeling and simulating embedded systems running
different MoC while HLA provides synchronous communication of real-time devices. For our
approach, the capability of multiple MoC is useful for testing different parts of the embedded
systems. For example, a controller based on a FSM could also be tested using Ptolemy and
the proposed approach.
In our solution the design under test (DUT) is an actor communicating with the hardware
device through HLA to ensure consistency of the inputs and outputs. These are then compared
with those of the reference model. The DUT represents, thus, the developed embedded system
device being tested, which is integrated in the simulation framework through HLA.
In order to carry out the integration with the UVM testbench, some of its elements were
implemented in Ptolemy (Fig. 3), which are:

– Sequencer responsible for sending the pre-defined stimulus stored in a file to the inputs
of the system models;
– Reference model a model of the device under test written in software, which conforms
to the specification;
– Checker responsible for collecting and analyzing data sent by the reference model and
the DUT. These data are compared upon reception, taking into account the respective
time delay and value. The combination of checker and reference model is also called
Scoredboard;
– Hardware interface communication mechanism for exchanging data between the simu-
lation computer and the hardware device. The interface is hidden in the picture because
it is within the DUT;
– DUT a description of the device being developed, normally coded in a HDL and simulated
in a design tool. Here the DUT refers to a hardware device;
– Display and plotter responsible for displaying the results textually or graphically.

The Checker takes into account the simulation time to check whether the simulator and
embedded system control loops are advancing synchronously. It checks at each simulation

123
Testing real-time embedded systems using high level architecture

Fig. 4 Simulation architecture

cycle whether the output generated by the DUT is the same as the one generated by the
reference model. In negative case, a error message is presented to the user. For testing the
real-time operation, the wall clock time is logged and printed in displays and plotters.
Figure 4 shows the three main developed components, the Ptolemy Federate, the Hardware
Federate and Robot Federate. The first one runs integrated within Ptolemy and has direct
access to time management mechanisms. The Hardware Federate allows the integration of
the system with any external hardware system for testing, even when there is no possibility
to embed HLA into the device. Communication with it must be done through an instance of
the Driver Tx/Rx class, which is responsible for accessing data from the hardware device.
This communication follows a protocol specific to each application.
The Robot Federate allows that all communication with RTI and the control logic of the
robot stay embedded in the hardware. To enable that, a Raspberry Pi board was used running
Certi HLA [16]. Using the PyHLA library [15] all code responsible for the robot control
was embedded and connected to RTI. As robots are real-time systems with different timing
requirements, a different synchronization approach was implemented for the Robot Federate
(as presented in Sect. 5.2). In this case, two approaches were implemented: a Stepped Time
Simulation and a Best-effort synchronization. Using the first option the robot only advances
to the next step when all federates do the same. When using best-effort synchronization the
robot exchanges data while doing its tasks and never waits for the other federates. This lets
the robot become more agile, but causes delayed messages at the Ptolemy Federate. This
synchronization and time constraints are discussed in next sections.
The communication between the federates is managed by the RTI through a gateway
accessible via TCP/IP, allowing the simulation to be performed over the Internet, either in
Local Area Networks or across geographically remote sites. Naturally, the distribution over
large networks imposes longer delays and thus limits the dynamics of the embedded system
under test that the co-simulation can cope with. The integration between the model in software
(Ptolemy) with HLA is further explained in works such as [6].
In the previous works [10,11] is possible to see the complete use of the approach presented
here, with HIL and robot-in-the-loop. In this paper, a new experiment is presented where more

123
 J. C. V. S. Junior et al.

sophisticated robots are integrated with the proposed platform through the integration with
ROS.
In Fig. 4 it is possible to see the architecture of the simulation environment and all necessary
components. The RTI Ambassador implements the methods of RTI and is the responsible
for receiving all requests from the Federate Ambassadors, and sending back commands from
RTI to them.
The Federate Ambassador regulates timing and data exchange of its Federate only, while
the RTI Ambassador regulates timing and manages data exchange for all Federations. For
example, a Federate can advance its own internal time only if it does not affect the other
Federates. Otherwise, it should send a request to advance its time the the RTI Ambassador,
which checks the timing of all Federates and only then grants the advance of the time.
In terms of implementation, the RTI Ambassador is part of the RTI implementation and
is provided by the RTI (in this project we use Certi), and the Federate Ambassador should be
developed for each different simulator (or device) to be co-simulated. In our case, we have
developed the Ptolemy Ambassador and two other Ambassadors for integration of Embedded
Devices and Robots in real-time. For more details see [6,10,14].
In the Ptolemy environment (left side of Fig. 4) the HLADEDirector manages the simu-
lation respecting policies based on Discrete Events, coordinating and enforcing the order of
actors in the domain and determining when to exchange data between actor and the RTI.
Time managing in Ptolemy works like this: the next actor in the execution queue is executed
and removed from the queue. After all actors have been executed once, the Ptolemy time is
advanced. After that, the Ptolemy time is compared with RTI time. If the Ptolemy time is
greater than RTI time, the simulation is paused and no more actors are executed until the RTI
time advances. In other words, this approach guarantees that the Ptolemy time will never be
greater than RTI time, keeping all simulation synchronized.
All data exchanges among federates must be specified according to a Federation Object
Model (FOM). A federate may represent a stand-alone application, a hardware device or a
specific simulator. A set of cooperative federates under the control of certain rules form a
federation.
Each variable of the data model is described in Code 1.
Code 1 Federation object model used in this project
1 (FED
2 ( Federation TestFed)
3 (FEDversion v1.3)
4 (spaces)
5 ( objects
6 ( class ObjectRoot
7 ( attribute privilegeToDelete reliable timestamp)
8 ( class RTIprivate)
9 ( class robot
10 ( attribute id reliable timestamp)
11 ( attribute battery reliable timestamp)
12 ( attribute temperature reliable timestamp)
13 ( attribute sensor1 reliable timestamp)
14 ( attribute sensor2 reliable timestamp)
15 ( attribute sensor3 reliable timestamp)
16 ( attribute location reliable timestamp)
17 ( attribute compass reliable timestamp)
18 ( attribute goto reliable timestamp)
19 ( attribute rotate reliable timestamp)
20 ( attribute activate reliable timestamp) ) ) )
21 ( interactions ) )

123
Testing real-time embedded systems using high level architecture

For more details about FOM rules and syntax, refer to [9]. The description of these
variables is presented as follows:

– Battery responsible for sharing the robot battery level. It can be used to know when a
robot is running out of battery and to analyze the power consumption;
– Temperature used to monitor robot temperature. This is important to develop and
test applications that are sensitive to temperature, like processing units and electric-
mechanical components;
– Sensor 1, sensor 2 and sensor 3 allow the designer to map which sensors should be
monitored;
– Location the robot position using the X, Y and Z coordinates (named GPS in file);
– Compass the robot compass;
– Goto this item is responsible for sending commands to the robot. These commands
are used to control the robots, and each robot should be programmed to accept such
commands;
– Rotate is a specific command used to control the robot rotation in degrees;
– Activate activate or deactivate sensors or actuators of each robot.

5.1 Data exchanging

In order to allow data exchange and time management, some classes were implemented. We
have implemented two actors in Ptolemy: PtolemyFederateActor and HardwareFederateAc-
tor. The PtolemyFederateActor receives all data from the RTI addressed to Ptolemy Federate,
and sends back all data from Ptolemy to the RTI. The HardwareFederateActor represents the
embedded system device and all data sent to this actor is redirected to the device, and all data
from thr device is sent y this actor to the Ptolemy model.
In addition to the actors, other classes have been developed: Ptolemy Federate, Hardware
Federate and Robot Federate. These classes implement the three types of Federates used
in this project. Each Federate implements the data exchanging rules and calls the methods
from Federate Ambassador when any service from RTI is necessary. Hardware Federate and
Robot Federate are independent from Ptolemy. The Hardware Federate is implemented in
C++ using the Certi API, and the Robot Federate is written in Python using the PyHLA API,
also from Certi community.
Among all implemented classes, the most important one is the HLADEDirector. It is an
extension of DEDirector from Ptolemy. In Ptolemy, the director is responsible for imple-
menting the timing and data exchange policies of the MoC. In our implementation the
HLADEDirector regulates the simulation based on Direct Event principles, but respecting
the synchronization with the Federation (regulated by RTI).
The steps for data exchange are presented in Fig. 5. It represents the steps needed for a
message sent by a Ptolemy Federate until it reaches a Hardware Federate.
First of all, the actor which initiates the transfer of data is called HLA Master Actor.
It calls a method to send data to the RTI. This method is part of PtolemyFederateActor,
which communicates with HLADirector. The HLADirector will pass the message to RTI.
The Director will receive it only when the Ptolemy time is synchronized to RTI time. When
the right time is reached, the Director authorizes PtolemyFederate to send the message. When
the message is sent, the HLA Director is once again notified. On the other side, the Hardware
Federate receives the data from RTI and forwards it directly to the device.

123
 J. C. V. S. Junior et al.

Fig. 5 Flow messages from the hla

5.2 Synchronization

Based on the principle of distributed simulation, two main synchronization approaches are
possible, optimistic and conservative [8]. Although HLA is originally based on the conserva-
tive approach, solutions for optimistic approaches exist [24]. The optimistic synchronization
follows a best-effort paradigm, possibly with roll-back. The RTI always tries to keep the
global simulation time synchronized with the local processors, but does not prevent each
local time of advancing faster. With roll-back, in case a late event arrives at the global queue,
the faster components return to their previous states, and only then the late event is processed.
In some cases though, late events are ignored, such as happens with distributed interactive
simulations (DIS). This makes optimistic synchronization better suited when data consis-
tency is less important than advancing the simulation with respect to the wall-clock time
[8].
Conservative synchronization is appropriate for analytic simulations, when the results
using multiple machines must be the same as those obtained from a single machine [8]. In
this case, the local processors must always require from the RTI permission to advance the
local time. Such permission is granted only when it is safe to advance the local time of all
federates, guaranteeing a consistent global time.
As the main target of this work is the testing of algorithms running in the robots, we use
only the conservative approach. Thus, the time elapsing in simulation may not correspond
to the time of the robot in the real scenario. But, a time analysis in this paper presents how
approximated the simulation and real time may be one with respect to the other. Therefore,
both Ptolemy and the embedded system work in lockstep of their control cycles, with each
cycle being one simulation time unit. Thus, the global time advances only when both federates,
i.e., embedded system and Ptolemy model, finish their current control cycles. In this case,
the simulation time and system performance are dictated by the slower federate, which in
turn determines the real-time capabilities of the co-simulation. For this reason co-simulating
mobile units, such as robots as in our case, may require using a slower robot speed if the
simulation computer including communications is slower than the robot’s controller.
In our approach a synchronization strategy called stepped time simulation is used for time
advancing, which ensure that time does not proceed to the next step until all activities at that
current global time are completed.
Figure 6 shows PtolemyFederate initially sending data to RobotFederate, and keep waiting
for a response. Only after that, it can resume its execution. The RobotFederate initially waits

123
Testing real-time embedded systems using high level architecture

Fig. 6 Synchronization model

for receiving data to process and sending a response to PtolemyFederate, then it keeps waiting
for a new event.
After each event cycle (send, wait and process) the PtolemyFederate execute the advance-
Time( ) operation, which notifies the RTI that the global simulation time must be advanced,
thereby ensuring that all events are ordered according to the timestamp of the transmitted
data in relation to the global time.
This does not represent a problem in our case since it is just a proof-of-concept of the
proposed co-simulation approach for testing a navigation algorithm. However, in general it
is undesired or impossible to have the hardware components slowed down by the simulation
model and thus an optimistic synchronization strategy without roll-back should be consid-
ered or else an adequate simulation platform should be used that allows meeting the time
constraints imposed by the hardware.

5.3 Timing analysis

In order to check whether a certain co-simulation meets the time constraints of a certain
real-time system (RTS), particularly when using a conservative lockstep synchronization
approach, it is important to analyze the delays involved.
Let us consider an event that occurs at a certain instant of the wall-clock taction , which is
captured with a certain delay d RT . This implies that the instant the event is observed is given
by tobser v as follows:
tobs = taction + dRT (1)

In general, an upper bound to d RT can be computed using the following equation:

 
dRT = max tcycle
emb PT
, tcycle + CRTI
emb
+ tcycle
RTI
+ CPT
RTI
(2)

123
 J. C. V. S. Junior et al.

where
emb
tcycle : control cycle in the embedded device
C emb
RT I : communication time from embedded device to RTI
RT I
tcycle : RTI simulation cycle
C PRTT I : communication time from RTI to Ptolemy
PT
tcycle : Ptolemy simulation cycle (3)
The maximum value of d RT determines the applicability of the co-simulation approach
to the testing of real-time systems. Basically, the maximum d RT must be within the time
constraints of the associated physical system. In turn, the communication time between the
federates (C emb RT I
RT I and C P T ) depends on the networking technology used and the simulation
cycles depend mainly on the number of events to be processed at each cycle. These events
RT I is
are related with the number of components in action at each cycle. In particular, tcycle
P T is based on the number
dependent on the number of federates in the simulation, while tcycle
emb
of actors in the model and tcycle depends on the actions performed by the embedded system
at each control cycle, such as data acquisition from sensors, execution of control algorithm
and actuation. The work in [6] presents the performance analysis of a distributed simulation
environment with Ptolemy, varying from 1 to 16 machines. The results demonstrated the
delay dependence on the slowest federation and on the number of actors in each simulation
model.

6 Experiments

To test our approach two experiments were developed. In the first experiment we use a robot
without odometry and in second one we use a robot with odometry.

6.1 Robot without odometry

In order to validate our approach, we apply our testing framework to a mobile robot which
is based on Raspberry Pi and uses the A* algorithm [12] for navigation in order to reach the
target position. The robot has a simple obstacle sensor that allows changing the direction of
its steps whenever an obstacle is detected, but it has not accurate odometry, in other words,
it is not possible to know accurately the actual status of the robot (position, location, speed,
etc.).
Figure 7 shows a created map, where the green square represents the startPosition = (0,
0) and the red square the endPosition = (4, 4). The walls are represented by the gray squares
(walls = ((1, 0), (1, 1), (1, 2), (2, 4), (3, 1), (3, 2), (3, 3))). The operational scenario included
a wall between the initial and the target positions in order to cause a reactive movement of
the robot. The same algorithm was implemented using Ptolemy actors to form the reference
model.
As referred before, one positive aspect of HIL simulation is that disturbances arising
from the operational environment can be captured in the co-simulation. In our case, our
approach is able to capture position estimation errors made by the robot caused by friction,
sensors inaccuracy, motors miscalibration, etc. (see Fig. 8). These disturbances can change
the perception of the robot about its own location and would not be easily detected with a
regular testing approach lacking the comparison against a reference model.

123
Testing real-time embedded systems using high level architecture

Fig. 7 Graphic simulation

scenario

Fig. 8 Real and simulated robots trajectories

The robot reference model is presented in Fig. 3. Both real and simulated robots receive
data from the real robot model, every cycle, namely current position and distance to any
obstacle, if detected. However, the robot and simulation act differently. The former executes
in the embedded system hardware and the desired displacement is approximated through
actual actuation in the real motors, while the simulation considers the correct displacement
as desired. If a motor is not correctly calibrated, or there is unexpected friction, there will be
differences between the real robot and its reference model which are detected and classified
as an error. After the mobile robot decides the new position to move to in the next step,
this new position is sent to the RTI and a routine is called to advance global time. This is a
blocking call that returns only after the RTI has assured the lockstep synchronization with
the reference model executing in Ptolemy. Then the robot advances its local time by one time
unit and repeats the control cycle.

6.1.1 Results

Figure 8 shows the results of a simulation where the real trajectory does not fit with reference
model. In a regular testing, it is not possible to notice wrong physical behaviors like these. In

123
 J. C. V. S. Junior et al.

Fig. 9 Real and simulated robots trajectories

real scenarios, embedded systems suffer interference from physical environment. However,
in the co-simulation approach proposed in this work, it is possible to see that the behavior of
the robot is different from the specified at the reference model, possibly resulting in future
failures.
Figure 9 shows one execution trace in which both real and simulated robots take similar
trajectories. This means that the real robot is meeting the specification and reaching the target.
Note that the simulated robot meets the specification by design. Nevertheless, the checker
detected errors in the messages arrival time, meaning that the results from the physical
and simulated robots were occasionally generated at different instants (beyond a predefined
precision). Nevertheless, the synchronization approach that was followed enforced a lockstep
execution of the navigation algorithm in both simulated and real robots with no errors detected
in the value domain, thus the similar trajectories.
As can be seen in Fig. 9, both robot and reference model have performed similar trajec-
tories, passing by the same points: ((0,0), (0, 1), (0, 2), (0, 3), (1, 3), (2, 3), (2, 2), (2, 1), (2,
0), (3, 0), (4, 0), (4, 1), (4, 2), (4, 3), (4, 4)). In order to achieve this result, the robot had to
be calibrated. Considering each square has 33 cm, the commands goFront() and goBack()
should active the engines for 2 s. To turn the robot right and left, the engines are activated for
1 s.
This approach demonstrated to be very useful for calibrating real and simulated robots
aiming at even more accurate models. In fact, using a real robot as reference, it is possible to
tune the simulation model and then, use the simulation model to tune new real robots.
This tuning process was also tested in a second scenario. Now, the same calibration was
used to control the same robot to achieve a point of a 6 × 6 scenario, as can be seen in Fig. 10.

123
Testing real-time embedded systems using high level architecture

Fig. 10 Simulation 6 × 6 scenario—trajectory 1

In Fig. 11, the same scenario was used, but now with different starting and ending points.
In these pictures, it is possible to see that the robot was successfully calibrated thanks to our
approach that enable the co-simulation of a physical robot with a simulated one.
However, this approach might add a delay in the robot which must now wait for an advance
time grant from RTI to advance its execution step. A time analysis is presented in next section
to quantify this delay.

6.1.2 Timing results

With respect to time, we measured the delay components described in Eqs. 2 and 3. One
thousand cycles were executed and each variable was collected. Table 1 shows the average,
maximum, minimum and the standard deviation of all executions.
As expected, the simulation cycle time in Ptolemy is always greater than the loop cycle
in the embedded system. The average of Ptolemy cycle (tcycle P T ) is 160.18 ms meaning that

on average events happening in the robot are sensed by the simulator after that time. In
comparison, the cycle of embedded system (tcycle
emb ) in average was 107.3 ms. It means that the

complete simulation cycle delay d RT for this scenario will be determinated by Ptolemy cycle
time, with average of 179.38 ms. That means that robots with control loop cycle not greater
than this value could be harmlessly added to this scenario without meaning a decreasing in
simulation time.

123
 J. C. V. S. Junior et al.

Fig. 11 Simulation 6×6 scenario—trajectory 2

Table 1 Timing results (ms)

Average Maximum Minimum SD

emb
tcycle 107.3 137 100 5.47
PT
tcycle 160.18 238 103 50.08
C emb
RT I 6.5 8.6 5.3 0.3
RT I
tcycle 6.4 7.2 5.8 0.2
C PRTT I 6.3 8.3 5.1 0.4
d RT 179.38 262.1 119.2 50.98

It is also important to observe the standard deviation of (tcycle

P T ) and (t emb ). The deviation
cycle
from embedded system is low because it is a simple robot with one task in each cycle. On the
other side, the Ptolemy cycle time varies much more due to the number of tasks the system
carries out, i.e., actors running in the simulation model, system processes and threads.
Nevertheless, the current setup can be used for testing purposes given the lockstep syn-
chronization that enforces consistency between the real and simulated parts of the HIL
co-simulation.

123
Testing real-time embedded systems using high level architecture

Fig. 12 Architecture with ROS Federate

6.2 Robot with odometry

This section describes the experiment using a real robot with odometry system.

6.2.1 General architecture

In other to evaluate the testing approach with more accurate robots, the system was integrated
with a ROS, as detailed in [20]. The environment is divided in two parts, the first one is the
ROS environment with ROS Robot Nodes, the Bridge ROS Interface, and ROS Core, which is
responsible for communication between ROS nodes. The second part is the HLA environment
with the RTI, the Bridge Ambassador and the Ptolemy Ambassador.
The intersection point between these environments is the Bridge. This is possible because
the Bridge is an ROS Node and also implements an Ambassador (see Fig. 12). The ROS
topics are responsible for sharing a specific information from the ROS environment (e.g.,
speed, position, data from sensors, etc). The Ambassador is an interface with methods used
by HLA to share information and manage time simulation. It was developed using PyHLA
library [15]. Thus, it is possible to have multiple robots in ROS sharing their data with any
other simulators that implement an HLA Ambassador, like Ptolemy. Two federates were
used. The Ptolemy Federate provides the interface among RTI and Ptolemy. On the other
side it is the ROS Federate that interfaces with ROS environment.

123
 J. C. V. S. Junior et al.

Fig. 13 Initial configuration of robots in simulation stage

In this experiment we have integrated the Stage Robot Simulator, Ptolemy, and a real robot
with odometry, called Turtlebot. The objective of this experiment is to evaluate the capability
of the proposed solution to assist the improvement of real-time systems. Thus, a real robot
was added to a simulation environment to interact with virtual robots. In this scenario, the
real robot uses a virtual sensor (an ultrasonic sensor for distance calculation), present only
in the the virtual robot. This approach is interesting to test sensors in real robots even before
buying them.
The virtual robot acts as a mirror of the real robot. All actions performed by the real
robot are repeated by the virtual one. Both robots send all output data as if they were a unique
instance in simulation, and all input data sent to these robots will be read by both. At each cycle
the virtual robot sends the distance from obstacles read from ultrasonic sensor, and the real
robot sends its actual odometry. As comparison, another robot was added in the simulation,
but controlled by a ROS Node. This allows the comparison of the trajectory of the real with the
virtual robot. Figure 13 shows the initial configuration of robots in simulation (ROS stage).
The goal of the robot is to go forward and avoid any detected obstacle (black box). In
Fig. 14 it is possible to see the trajectory of robots starting from point (x = 0, y = 0). The
green line represents the position collected by the odometry of the real robot. It is possible
to see that it follows the blue line, which represents the instance running in Ptolemy. This
trajectory is the ideal one, generated by the controller in software. The red line represents the
trajectory of the simulated robot receiving the same stimulus as the real robot. The trajectory
of the simulated robot is completely different from the other ones because the model of
friction used by Stage. It is based on a different kind of thread used in our experiments. This
is most noticeable in the corners when the simulated robot turned faster than necessary. Thus,
this approach proved to be useful to detect inconsistency between a simulated model and its
respective physical device.
In a second experiment, one of the wheels of the robots was modified to simulate a phys-
ical defect and then check if the proposed approach is able to detect this kind of problem.
Figure 15 presents the trajectories using the same color pattern from Fig. 14. Once more, the
trajectory of the mirror robot (red) is not as expected, due to the same reason in the previous

123
Testing real-time embedded systems using high level architecture

Fig. 14 Trajectory of robots: (green) real robot; (blue) reference model in Ptolemy; (red) mirror robot in
Stage. (Color figure online)

Fig. 15 Trajectory of robots with defect in real robot

experiment. But now, the trajectory of the real robot (green) is not as expected any more.
In addition to the error caused by the wheel, instead of continuing the route going straight
forward, it changed its direction. This is because the distance sensor of the virtual robot
detected again the virtual obstacle, as it is positioned in a similar position that detected the
obstacle in the previous action.
In a third experiment, the real robot keeps its modified wheel. However, a modification
was made in the control of the robot. In this new control, instead of always going forward,
the robot checks its position at each cycle and adjust it when necessary, in order to keep a
linear trajectory.
Figure 16 presents the trajectory of the real robot (green) and the mirror one (blue). It is
possible to see the improvement, even with a modified wheel.
These experiments demonstrated that the proposed approach is efficient in assisting the
design of real-time systems using HIL simulation. Also, it proved to be possible to simulate
multi-robot scenarios, integrating real and simulated robots (or any other device).

123
 J. C. V. S. Junior et al.

Fig. 16 Trajectory of robots after improving the control logic: (green) real robot; (blue) mirror robot. (Color
figure online)

7 Final considerations

This paper proposed a testing approach for embedded systems through co-simulation using
HIL that allowed the testing of physical devices under real conditions that directly influence
their real-time behavior. Focusing on real-time systems, the approach checks not only the
generated output data but also the synchronization, i.e., the instants of output generation.
Moreover, the approach also captures the reaction of the system to external environmental
factors, which only the HIL co-simulation allows to do.
Our proposal allows the testing of any hardware device provided with HLA interface,
i.e., an HLA Federation, which provides a synchronous and consistent communication with
any simulator through TCP/IP. In particular, in this work we used the Ptolemy simulation
framework but the integration with other frameworks such as, Matlab, VHDL, SystemC, etc.
is also possible. Nevertheless, Ptolemy allows the integration of different MoC in the same
simulation and using the same graphical language.
The integration of real-time devices through ROS was also presented. In experiments, real
robots were evaluated in HIL simulation cooperating with virtual robots. Both logical and
physical failures were detected in the test and were able to be solved.
Although the focus of this work is not test multiple devices at same time, in a recent work,
we have demonstrated that HLA can do be used to integrate and coordinate multiple real-time
robots [21]. As future work, a deep investigation about synchronization will be conducted, in
order to explore the limitations of HLA when co-simulating real-time devices and simulators.

References
1. AbouTrab MS, Brockway M, Counsell S, Hierons RM (2013) Testing real-time embedded systems using
timed automata based approaches. J Syst Softw 86(5): 1209–1223. doi:10.1016/j.jss.2012.12.030. http://
www.sciencedirect.com/science/article/pii/S0164121212003391
2. Accellera UVM (2011) Universal verification methodology i. 0 user’s guide
3. Bacic M (2005) On hardware-in-the-loop simulation. In: 44th IEEE conference on decision and control
and 2005 European control conference. CDC-ECC ’05. pp 3194–3198. doi:10.1109/CDC.2005.1582653

123
Testing real-time embedded systems using high level architecture

4. Bergeron J (2003) Writing testbenches: functional verification of HDL models, vol 2, 2nd edn. Kluwer
Academic Publishers, Norwell
5. Bergeron J (2006) Writing testbenches using system Verilog. Springer, Heidelberg
6. Brito AV, Negreiros ALV, Roth, C, Sander O (2013) Development and evaluation of distributed simulation
of embedded systems using ptolemy and hla. In: 17th IEEE / ACM International symposium on distributed
simulation and real time applications
7. Depra D, Zatt B, Bampi S (2009) A method for hw functional verification through hw/sw co-simulation
in complex systems: H.264/avc decoder as case study. In: Test workshop, 2009. LATW ’09. 10th Latin
American, pp 1–6. doi:10.1109/LATW.2009.4813807
8. Fujimoto RM (ed) (2000) Parallel and distributed simulation systems. Wiley, New York. http://ptolemy.
org/books/Systems
9. IEEE (2010) IEEE standard for modeling and simulation (m&s) high level architecture (hla)– federate
interface specification. IEEE Std 1516.1-2010 (Revision of IEEE Std 1516.1-2000) pp 1–378. doi:10.
1109/IEEESTD.2010.5557728
10. Junior JCVS, Brito AV, Nascimento, TP (2015) Verification of embedded system designs through
hardware-software co-simulation. Int J Inf Electron Eng 5(1): 68. http://www.ijiee.org/index.php?m=
content&c=index&a=show&catid=49&id=548
11. Junior, JCVS, Brito AV, Nascimento TP (2015) Testing real-time embedded systems with hardware-in-
the-loop simulation using high level architecture. In: 2015 Brazilian symposium on computing systems
engineering (SBESC), pp 142–147. doi:10.1109/SBESC.2015.34
12. Loong WY, Long LZ, Hun LC (2011) A star path following mobile robot. In: 2011 4th International
conference on mechatronics (ICOM), pp 1–7. doi:10.1109/ICOM.2011.5937169
13. Nebot P, Torres-Sospedra J, Martínez RJ (2011) A new hla-based distributed control architecture for
agricultural teams of robots in hybrid applications with real and simulated devices or environments.
Sensors 11(4):4385–4400
14. Negreiros ALVd, Brito AV (2012) The development of a methodology with a tool support to the distrib-
uted simulation of heterogeneous and complexes embedded systems. In: 2012 Brazilian symposium on
computing system engineering (SBESC), IEEE, pp 37–42
15. Nongnu: Pyhla python bindings for m&s hla (2014). http://www.nongnu.org/certi/PyHLA/
16. Noulard E, Rousselot JY, Siron P (2009) Certi, an open source rti, why and how. In: Spring simulation
interoperability workshop, pp 23–27
17. Piziali A (2004) Functional verification coverage measurement and analysis, 1st edn. Kluwer Academic
Publishers, Boston
18. Ptolemaeus C (ed) (2014) System design, modeling, and simulation using ptolemy II. Ptolemy.org. http://
ptolemy.org/books/Systems
19. Shah SM, Irfan M (2005) Embedded hardware/software verification and validation using hardware-in-the-
loop simulation. In: Proceedings of the IEEE symposium on emerging technologies, IEEE, pp 494–498
20. Costa LFS, Nascimento TP, Brito AV (2015) Integration of robot operating system and ptolemy for design
of real-time multi-robots environments. In: International embedded systems symposium (IESS), 2015,
IFIP, pp 37–42
21. Simo R, Santos, LHS, Brito AV (2015) An adaptive approach for real-time communication of multi-
robots based on hla. In: Latin American network operations and management symposium (LANOMS),
pp 92–98. doi:10.1109/LANOMS.2015.7332676
22. Souza URF, Sperb JK, de Mello BA, Wagner FR (2003) Tangram-virtual integration of heterogeneous ip
components in a distributed co-simulation environment. In: Proceedings 16th symposium on, integrated
circuits and systems design, 2003. SBCCI 2003, IEEE, pp 125–130
23. Su K, Fu W (2011) Designing hardware-in-the-loop simulation system for missile with high level archi-
tecture. In: 2011 International conference on mechatronic science, Electric Engineering and Computer
(MEC), pp 1525–1527. doi:10.1109/MEC.2011.6025763
24. Wang X, Turner S, Low M, Gan BP (2004) Optimistic synchronization in hla based distributed simulation.
In: 18th Workshop on parallel and distributed simulation, 2004. PADS 2004. pp 123–130. doi:10.1109/
PADS.2004.1301293
25. Zatt B, Azevedo A, Agostini L, Bampi S (2006) Validação de uma arquitetura para compensação de
movimento segundo o padrão h. 264/avc. In: XII IBERCHIP WORKSHOP, Costa Rica

123