SICE-ICASE International Joint Conference 2006

Oct. 18-2 1, 2006 in Bexco, Busan, Korea

Automated Testing for Automotive Embedded Systems
Dae-Hyun Kum', Joonwoo Son2, Seon-bong Lee3 and Ivan Wilson4
Department of Mechatronics, Daegu Gyeongbuk Institute of Science & Technology, Daegu, Korea
(Tel: +82-53-430-8455; E-mail: kumdhgdgist.ac.kr)
2Department of Mechatronics, Daegu Gyeongbuk Institute of Science & Technology, Daegu, Korea
(Tel : +82-53-430-8453; E-mail: jsongdgist.ac.kr)
3Department of Mechatronics, Daegu Gyeongbuk Institute of Science & Technology, Daegu, Korea
(Tel: +82-53-430-8450; E-mail: mechagdgist.ac.kr)
4DsysD Ltd, Nottingham, England
(Tel: +44-115-988121 1; E-mail: ivangDsysD.com)
Abstract: The automotive embedded system becomes even more complex and occupies more portion of vehicle price
lately. Therefore, automotive industries apply a model-based approach for the embedded system development. The
model-based approach improves quality and reduces cycle time by using modeling and its ability to simulate to
introduce early validation of requirements. This paper describes automatic test generation and execution techniques for
automotive embedded systems to maximize the early validation capabilities of model-based development process. The
proposed techniques can cover from the functional high level model validation to the auto-generated software
validation. For the functional model validation, a virtual prototype is used on the PC environment. A rapid prototype
and target execution and monitoring system, Target Suite, is developed for the generated software validation. We also
develop an automatic testing and analysis environment to manage the test executions and the corresponding results. The
proposed model-based automated validation techniques will go far forward to improving the reliability and
cost-effectiveness of the products
Keywords: Automated testing, Model-based development process, Automotive Embedded system

1. INTRODUCTION embedded systems. The proposed approach can cover

from high level model validation to the auto-generated
The Automotive embedded system becomes even code validation in a real ECU(Electronic Control Unit)
more complex and occupies more portion of vehicle Functional model is developed by using the I-Logix
price lately. In order to design products which improve Statemate. Test cases are automatically generated by
reliability and quality as well as reduce development ATG(Automatic Test Generation) which allows
time and cost, automotive industries have been focusing automatic test generation out of Statemate design. A
on well structured development process such as
rapid prototype and target execution system,
model-based approach [1-2]. But to meet demand for Target-Suite is developed for validation of
high quality and lower cost, it is becoming increasingly auto-generated code from the behavior model. In
necessary to rely on a new test systems and
addition, Test-Generator and Test-Analyzer is
methodologies in all phases of the product life cycle. developed to automate and improve test process.
Erroneous automotive embedded software may cause
serious car accident related to human safety. Therefore,
it is essential to detect any errors in the embedded 2. TEST PROCESS
software through various test processes. The embedded 2.1 Model-based development process
software is often subject to change to meet customer's
needs and its functionalities, so that it is desirable to Model-based development process(MBDP) is
automate testing to improve efficiency of process. introduced to improve quality and reduce cycle time.
Test automation consists of the following three steps: Fig. 1 describes the Model-based development process
test case generation, test execution and result analysis. that follows typical V-process.
Many research efforts have been made at these Development is started with requirement capturing
automation techniques. During the requirement and analysis. The success of any product development is
capturing phase, automated testing can be realized by depending on the creation of clear, complete and
introducing formal language [3-5]. Automatic unambiguous requirements. Functional model should be
generation of test cases from statecharts, control flow, created according to the requirement capturing
information flow and MSCs(Message Sequence Charts) document. The vehicle electronic system is too large to
are widely used for specifying the dynamic behavior of
be addressed at a time. So it is often broken down
the model [6-8], and automation concerning test system according to the functional groups. Once the main
and process has been presented [9]. However, research sub-system has been identified, it is ensured that signals
on seamless test automation based on model-based
for input and output of the system are clearly defined.
approach has not been applied. Virtual prototype of functional model enables us to test
In this paper, several techniques are presented that and validate the software functionality without hardware
involves seamless test automation for automotive in the early stage. When behavior model is released,

89-950038-5-5 98560/06/$10 C 2006 ICASE

4414
ECU software development and network system After test execution, the test results should be
development are started at the same time. analyzed to unveil the error that caused a test to fail and
Once the system design and ECU design have been discovered errors are needed to be fixed. Test results
validated on virtual environment, the designs should be and related information should be documented
moved into the implementation phase. To generate code throughout all test processes.
for embedded applications, all the software designs,
implementation details and constraints have to be 3. TEST AUTOMATION TECHNIQUES
imposed into the model such that the generated code can
satisfy all the requirements. Finally all source codes 3.1 Overview of test automation
including generated application code, I/0 drivers, Since there are huge numbers of test cases to execute
communication kernel and operating system code are in automotive embedded software, it is obvious that
integrated. Automation is required to maximize coverage and
reliability.
In this paper, several techniques are presented
regarding automated testing that can cover requirements
testing, functional model testing and auto-generated
code testing. Fig. 2 describes the test phases and
corresponding tool chains for the proposed techniques.
Usually automated testing is consists of test case
generation, test execution, and result analysis shown as
following figure. It is important that these steps are
connected seamlessly at each test phase

TVst Case I i xctidur Roert

sItdo te
I7st-Gen4 SiIuflaton Tos4Anedyze

Fig. 1 Model-based test and development process Sidemle

Ai lsimuldbri
AT' To eAhWyzw
2.2. Model-based test process
ecorded III
Test process also follows model-based development lin ;frnom
ITard-btJk
process, but testing should be performed separately to
the design. Fig. 1 describes model-based test process
linked to development process. Fig. 2 Tool chain for the test automation
Clear understanding of the system requirements
should be preceded before getting down to testing. Test-Generator, Test-Analyzer and Target-Suite are
Exact I/0 signals to be tested and test specification are developed for seamless test automation in this paper.
defined by analysis of overall system structure, Test-Generator allows test engineer to generate test
functional behavior so that test result is reliable and cases that can be executed automatically in the
automated testing is performed efficiently. Statemate simulation. Fig. 3 shows Test-Generator
Test cases which are simple, atomic and formal interface that is developed using Excel and Visual Basic
should be prepared based on the test requirements. The language. If time, steps, input signals and values are
reason is that, complex test cases can make errors when written in the Excel cells and push GENERATION
executing the test case. button, a file containing test cases is generated
A test system capable of automatic testing is needed. automatically.
Depending on the development process and the system
under test, different test systems are chosen. The
following table 1 shows a selection of test techniques.
Table 1 Test phases and techniques
Test phase Test technique
Requirement testing Model in the loop
Functional model testing Model in the loop
Implementation testing Software in the loop
Integration testing Hardware in the loop
Fig. 3 Test-Generator

4415
 Since usually test cases are from hundreds to millions, Static error checking should be preceded prior to test
it is time consuming and not easy to verify numerous execution. If there is any static error, test execution may
test cases one by one. In this research, Test-Analyzer is not be operated and exact test result can not be expected.
developed in order to analyze test result automatically. Check model from the Statemate menu helps to unveil
It compares test results from the simulation of the static error in the model.
behavior model with expected results. Fig. 4 shows
analysis result by Test-Analyzer. If an error is detected, 3.3 Functional model testing
the cell is filled with orange color. Target-Suite will be
introduced for the auto-generated code testing in chapter In order to generate test cases automatically uses
3.4 in detail. ATG which is Statemate plug-in program. Test stimuli
and the expected responses are computed by based on
selected activity charts and test goal is selected to cover
OUTPUT-VARIABLES ([l PASS L* FAIL)
DR&VWIN BTNPRRLT
outputs, states or transition. Two kinds of files are
e PpDOORKEY CMD_WIN_ DRVLWINt BTNLRRRT
DRVLWIN DOOR_OPEI
Time
Tmne St',
CYL-DRV MTR'DRV BTl_PASS DRV obtained after ATG execution. One is test cases which
0 4 0 0 0 0 0 0 can be executed in the Statemate simulation. The other
1 8 2 0 2 2 2 1
2 11 2 0 4 4 4 1 is report file including test coverage information.
3 15 1 0 4 1
4 20 1 Detected Error 4 1
5 23 1 1 4 1
6 27 1 1 4 4 4 1
7 30 1 1 4 4 4 1
8 33 1 4 4 4 1
9 37 1 1 4 4 4 1
10 40 1 1 4 4 4 1
11 43 1 1 4 4 4 1

Fig. 4 Test-Analyzer

3.2 Requirement testing

Requirement testing is performed to ensure that
created behavior model is consistent with the
requirements. Requirement based test cases might be
automatically generated by translating requirements into
formal language, however it is known that the most
reasonable method is to express requirements in
narrative language. In this study, test cases are
generated manually by test engineer with Fig. 6 Model in the loop simulation in Statemate
Test-Generator, generated test cases are executed in the
Statemate simulation and the results are analyzed with Statemate simulation executes generated test cases
Test-Analyzer. automatically. Stimuli and simulated output values are
Fig. 5 shows test cases generated by Test-Generator recorded into files during the simulation. The simulated
based on the functional requirements. Generated test
responses are compared with the expected result with
cases are executed automatically in the Satemate
simulation. And then test results are easily evaluated Test-Analyzer. If an error is detected, testing should be
with Test-Analyzer. performed step by step to find cause of an error. Fig. 6
illustrates error fixing process using virtual panel and
Project Haie: WIN TEST 01 state charts.
Work Area Directory: d:/10 Working/MODERATO I{UIMDH/wa win test 01
Profile Hane: test ddm 102
Since early functional model may contain errors, in
Date/Tine Produced: Wed Apr 19 09 :19-:10 2006 order to correct these errors, generating test cases, test
Recorded tiRe Rode: Relative
Recording starts at simlulatian time: 0.000000 execution and analysis result are repeated several times.
#Data Section:
This process is inevitable but labor-intensive and
0.000000 0 DDM TARGET SUITE:DOOR KEY CYL DRU IO N 0 time-consuming. Automating this process leads to
1.000000 4 DDMITARGET SUITE:DRU WIH BTHIRRLTIDO N 2
1.000000 4 DDMI TARGET SUITE:DRU WIH BTH PASS IO N 2 increasing correctness and reliability and reducing time
1.000000 4 DDMI TARGET SUITE:DOOR OPEN DRU IO C 1
2.000000 8 DDM TARGET SUITE: DRU WIH BTH RRLT IDO N 4
and efforts on testing.
2.000000 8 DDMI TARGET SUITE:DRU WIMHBTHIPASS IDO N 4
2.000000 8 DDMn TARGET SUITE:PWR WIH ENABLED C 0 3.4 Auto-generated code testing
3.000000 11 DDM TARGET SUITE:DOOR KEY CYLDRU IO N 2
3.000000 11 DDM TARGET SUITE: DRU WIN BTH RRRTIOIN 0
4 .000000 15 DDMHTARGET SUITE:DOOR KEY CYL DRUI O N 2 After completing validation of behavior model, C
4 .000000 15 DD_TARGET_SUITE:WI N_BTN_DRU N 3
4.00 000 15 DDM TARGETSUVITE: DRU WIN_BTN_RRRT_IO N 0 code is generated automatically by Statemate. There
14.000000 15 DDM TARGET SUITE:PWR WIN ENABLED C 1| might be differences in task execution time and values
between simulation of behavior model running in PC
Fig. 5 Test cases generated by Test-Generator and auto-generated code running in a real ECU
environment. Target-Suite provides automated testing
environment for the auto-generated code.

4416
 Target-Suite consists of hardware equipment and Target-Monitor analyzes and compares execution
control software. Hardware comprises of two main time and values between different environments and
boards: one is target board and the other is data shows the test results on PC. Fig. 9 shows validating
acquisition board. And there are two control software results by Target-Monitor.
programs. One is Target-Practice which can map I/0
signals of the functional model with a real ECU model. 4. CASE STUDY
The other is Target-Monitor which analyzes and shows
the execution result on PC monitor. Fig. 7 depicts the Proposed test automation techniques for automotive
picture of Target-Suite. embedded systems are applied to window lift system.
4.1 window lift system
The functional model of window lift system is
designed by Statemate. This system consists of 4 door
modules and BCM(Body Control Module) and each
ECU is connected by CAN/LIN network. Fig. 8 shows
physical architecture of window lift system and activity
charts of DDM.

CIMt>RLDM
iLLSC
Fig. 7 Target-Suite UN ...D ,RM2U

To test generated application code running in ECU

environment, Statemate I/0 signals should be mapped to
ECU I/0 ports. When generating code, these mapped
I/0 signals are considered automatically. Fig. 8 shows
signal mapping from functional model to target ECU by 9 I" O "III
WOR_IKEY-CYL-DIRV_IO 0 , :

drag and drop. I

*:ItDR_WEIN_tiRV
D
,CH1 LD_ H-NL.OCK SM

~~~~~~~~. DR_ _SIO

1S4_¢Tli
D'V

Hl,fT_UEP_L"T__DRV L T_OPERATlfIi_DRV
1 W1H-lD"-PiT_DRSt, -11: CHaLMIHL"TRLDRV
j IIHl, TAhLLL 15RV:A i I

----------_
___ ___ __--_---- ---

Fig. 10 Functional model of window lift system

4.2 Requirements testing

Fig. 8 Target-Practice According to the functional requirements, about 50
test cases are generated for each ECU model with
Test-Generator. Statemate simulation executes
generated test cases automatically and Test-Analyzer
analyzes the simulated result in order to verify the
model.
4.3 Functional model testing
ATG was used to generate test cases automatically in
order to validate each ECU model. Test goal is set to
cover all states and transitions and about 300 test cases
are generated for each ECU model.
Generated test cases are automatically executed in
the Statemate simulation and the simulated responses
are analyzed by Test-Analyzer which compares with the
Fig. 9 Target-Monitor expected result form ATG.

4417
4.4 Auto-generated code testing Generation of Functional Tests using MSCs,
After the model validation is complete, software of SDL and TTCN", Computer Communications,
each ECU is auto-generated and downloaded into Volume 24, Issues 3-4, Page 374-393, 2001
Target-Suite to test performance and functionality of the [9] D. L. kaleita and N. Hartmann, "Test
software before hardware is built. Recorded file during Development Challenges for Evolving
Statemate simulation is used for the stimuli to the Automotive Electronic Technologies", SAE,
Target-Suite. Execution time and response value is 2004-21-0015, 2004.
validated using Target-Monitor.

5. CONCLUSION
In this research test automation techniques are
proposed for functional model and auto-generated code
running in target ECU. Case study of window lift
system is introduced to apply this automation process.
Erroneous automotive embedded system may result
in serious accident or problem related to human life.
Error detection process is simple and repeated process
that is time consuming. Adaptation of proposed
techniques can reduce time and efforts on testing
dramatically. Furthermore test reliability and
consistency could be increased.
In the future study, test automation process is
expended to network and integration test to realize test
automation through the entire process.

REFERENCES
[1] J. Son, I. Wilson, W. Lee and S. Lee, "Model Based
Embedded System Development for In-Vehicle
Network Systems", SAE, 2006-01-0862, 2006.
[2] Y. Dong, M. Li and R. Josey, "Model Based
Software Development for Automotive
Electronic Control Units", SAE, 2004-21-0038,
2004
[3] R. Weber, K. Thelen, A. Srivastava and J.
Krueger, "Automated Validation Test
Generation", Digital Avionics Systems
Conference ofIEEE, Page 99-104,1994.
[4] R. W. Butler, "An Introduction to Requirements
Capture Using PVS: Specification of a Simple
Autopilot", NASA Technical Memorandum
110255, 1996
[5] C. L. Heitmeyer, R. D. Jeffords and B.G. Labaw,
"Automated Consistency Checking of
Requirements Specifications", ACM
Transactions on Software Engineering and
Methodology, vol. 5, No. 3 , Page 231-261,
1996.
[6] Y. G. Kim, H. S. Hong, D. H. Bae and S.D.Cha,
"Test Cases Generation from UML State
Diagrams", IEE proceedings, online no.
199990602, 1999
[7] N. H. Lee and S. D. Cha, "Generation Test
Sequences from a Set of MSCs", The
International Journal of Computer and
Telecommunications Networking, Volume 42,
Issue 3, Page 405 - 417, 2003
[8] R. L. Probert, H. Ural and A.W.Williams, "Rapid

4418