Model Based Testing

Model Based
Testing
Presenters
Ashish Mote
Amit Chigteri
© 2005 KPIT Cummins Infosystems Limited
We value our relationship

Prerequisites & Class Purpose
• For this session basic knowledge of Matlab-Simulink, Code

generation & S-function is required.
•To know about different types of testing used in software

industry, their objectives & when shall it performed??
• To learn different stages of testing & their use in model

based development(Simulink based development)
7-Mar-12 2 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Agenda
• Testing, SDLC’s, Objectives of testing and testing artifacts
• Classification of Testing Strategies based on different factors
• Model Based Testing – MIL, Plant Modeling, SIL, PIL & HIL
• Testing Automation, automation for MBT

Software Testing

What is testing?
• It is an activity aimed at evaluating an attribute or capability of a
program or system and determining that it meets its required results
• Software testing is an investigation conducted to provide

stakeholders with information about the quality of the product or
service under test.
• It is to be noted that testing can never completely identify all the

defects within software.
• When to perform testing?

Software Development Life Cycles
Waterfall model Modified Waterfall model
V model Agile Model

Testing Objective & Phases
• The objectives with which mostly testing is done software industries
are
1. Whether there are any bugs in implemented software.
2. Whether software/build is meeting requirement.
3. What is performance of system with this software.
• Traditionally testing was limited to debugging, but later when

separation was introduced, phases & goals in software testing
industry were classified as
• Until 1956 - Debugging oriented
• 1957–1978 - Demonstration oriented
• 1979–1982 - Destruction oriented
• 1983–1987 - Evaluation oriented
• 1988–2000 - Prevention oriented

Testing Artifacts
1. Test Plan :- A test specification is called a test plan.
2. Traceability Matrix :- A traceability matrix is a table that

correlates requirements or design documents to test documents.
3. Test Vector :- It is a systematic collection in form of array/matrix

of values for a particular input or output for model under test.
4. Test Case & Test Suite :- Test plan is overall brief plan of
complete testing. A test case more detailed version for a small
part of test plan for a specific requirement. Collection of test
cases is known as Test suite.
5. Test Data :- All the values of test inputs, expected outputs, actual
outputs along with test environment related data are collected
together in file. This data is called Test data.

Classification of testing

• Based on Scope
• Functional Testing
• Non-functional Testing(Performance, Scalability, Usability, Security
etc)
• Based on Method
• Static
• Dynamic
• Based On Knowledge tester has

• White Box Testing
• Black Box Testing
• Grey Box Testing

• Based on Level of testing

• Unit Testing/ Component Testing
• Integration Testing
• System Testing
• System Integration Testing
• Based on way test is executed

• Manual Testing
• Automated Testing
•Based on who is tester

• Alpha Testing
• Beta Testing

Model Based Testing

Model Based Testing
•The term ‘model-based testing’ is widely used today with

slightly different meanings. In the automotive industry it is
normally used to describe all testing activities in the context of
model-based development projects.
•One major benefit of model-based development is the

possibility to “do things as early as possible”. In terms of
testing this means to test the functionality in the model before
the software is implemented and integrated into the final ECU.
• Other advantages also include possibility of automation by use

of scripts, automatic test case generation, Specific
coding/modelling guideline checks using Model Advisor
Levels of Model based Testing
• Between the initial modelling and the integration into the

ECU there are intermediate integration levels of testing.
• These integration levels are

• Model-In-Loop i.e. MIL Testing
• Software-In-Loop i.e. SIL Testing
• Processer-In-Loop i.e. PIL Testing
• Hardware-In-Loop i.e. HIL Testing

Model In Loop(MIL)

MIL
• The first integration level is based on the model of the
system itself.
• Testing an embedded system on MIL level means that the
model and its environment are simulated (interpreted) in the
modelling framework without any physical hardware
components.
• Generally Objective of MIL level testing are
• Functionality/requirements testing
• Overflows/ Underflows detection
• Division by 0 and few other arithmetic errors

MIL Continued
• Model in Loop testing helps in finding bugs in implementation
in very early stage of software life cycle.
• For MIL/SIL in Matlab/Simulink Environment there are 2 ways
of testing
1. By using test vectors
2. By using Plant model(s)
MIL Using Test Vectors MIL Using Plant Modeling

MIL Continued
• For MIL/SIL testing wrapper model shall be created first from
design model.
• A design model is model which has all data defined & from
which code generated.
• Wrapper model is also knows as test model, Simulation model
or Feature/Component Simulation Model
• Design model is also known Code-gen Model, Controller Model
or Actual Model in very crude manner

Testing using test vectors
• Test vector is systematic collection in form of array/matrix of
values for a particular input or output for model under test.
• Collection of such vectors together which can test 1 or more
requirement or even a complete single feature is known as Test case.
• Collection of all test cases together that can completely test model
under test in a data file(.mat file in Matlab/Simulink) is knows as Test
Data for model based testing.
• These vectors can be handwritten by tester or can be automatically

generated using Matlab based testing tools
• Sometimes these test vectors can also come from actual vehicle test
data.
• These test vectors shall be created in such way that they are re-
usable for SIL as well PIL & HIL.

Testing Using Plant Model
• Plant Model is model made for simulating physical systems or
physical components
• Plant modelling shall be able to simulate physical

components that have sufficient fidelity to test the
various execution paths within the software.
• Testing becomes more effective when plant models with

right fidelity are used instead of simple test vectors.
• Plant models can be made to replicate behavior any

physical components such as sensors, actuators using
Simulink & Simscape which is also a tool by Mathworks

Plant Model Example
• Suppose a Proportional controller for “controlling heater for
maintaining room temperature within given range” is designed
using Simulink.
• This controller is suppose provide Duty cycle input to heater
relay or percentage duty cycle to device driver of the heater
• To test this controller a plant model of room can be made as

in snapshots on next slides which can test how efficient is it’s
proportional control.

Snapshot of top level of test model

Inside Plant Model

Software In Loop(SIL)

SIL
•Testing an embedded system on SIL level means that the
embedded software i.e. generated code is tested within
a simulated environment model but without any
hardware.
• Since the environments virtual, a real-time environment
is not necessary.
SIL Using Plant Modeling SIL Using Test Vectors

SIL Continued
• For SIL testing few objectives can be same as MIL, but
majorly following are objectives of SIL testing
• MIL v/s SIL behavior comparison to find differences if any
• Dead (or unreachable)Code finding
• Code Coverage testing to see how much part of code is really being
tested in test case you have designed.
• Out-of-bound array access
•If some legacy code has been mixed with Simulink-RTW
generated or Simulink-Targetlink code generation then Other
code errors that can show up, especially in a mix with hand-
generated code, include:
• Illegally de-referenced pointers
• Read-only access to non-initialized data
• Dangerous type conversions
• Usually SIL are performed on Windows/Linux based desktop.

SIL Continued
• For SIL testing in Matlab/Simulink Environment following

Steps shall be performed
• Define all data’s(Variables) correctly if not done at MIL stage
• From code generation model generate the code
• From this code make a separate S-function or for subsystem you
want to test directly make a S-function.
• In model used for MIL testing replace model with S-Function.
• Input/Import the test data to workspace & Run the model.
• Plot and analyze outputs. Compare with previously saved MIL
testing results if in scope of testing.

Processor In Loop(PIL)

PIL
7 March 2012 29 29 Version 1.0

PIL Conti
•Embedded controllers are integrated in embedded devices with proprietary hardware (ECU).
•Testing on PiL level is similar to SiL tests, but the embedded software runs on a target board
with the target processor or on a target processor emulator.
•Tests on PiL level are important because they can reveal faults that are caused by the target
compiler or by the processor architecture.
•It is the last integration level which allows debugging during tests in a cheap and
manageable way.
•Therefore the effort spent by PiL testing is worthwhile in almost all cases.
7 March 2012 30 30 Version 1.0

Hardware In Loop(HIL)

HIL
• When testing the embedded system on HIL level the
software runs on the final ECU.
• However the environment around the ECU is still a
simulated one. ECU and environment interact via the
digital and analog electrical connectors of the ECU.
•The objective of testing on HIL level is to reveal faults in
the low-level services of the ECU and in the I/O services.

Kind of bugs that can be found at each level
7 March 2012 33 Version 1.0

Correct time in V model for each type of these testings
7 March 2012 34 Version 1.0

Testing Automation

Test Automation
• Test automation is the use of software to automate testing procedures.
•Test Automation can be used

• To setup test environment
• To control the execution of tests
• To compare actual outcomes to predicted outcomes
• To set up of test preconditions
• To setup other test control and test reporting functions
• Test Automation objectives are

• To reduce efforts and save time in repetitive type manual testing
• To reduce human errors that can get introduced in manual testing
• To find certain classes of defects which cannot be effectively found out or can
be missed by manual testing

Test Automation Approaches
• There are 2 general approaches to test automation are
1. Code driven testing

2. GUI driven testing
• Code driven testing(Scripting Based) :- The interfaces(ways of

inputting) to components, modules or libraries are tested with a
variety of input arguments to validate that the results that are
returned are correct.
• GUI driven testing :- A testing framework generates user interface
events such as keystrokes and mouse clicks, and observes the
changes that result in the user interface, to validate that the
observable behavior of the program is correct

Test Automation in Model Based Development/Testing
• What Can be automated?

1. Wrapper Model Creation
2. Test vector creation
3. Test result display, comparison and recording
4. Test result report creation
5. Run time bug finding and reporting
6. Coverage tracking
• Reactis is one stand alone application that reads .mdl files

created in Matlab-Simulink/Stateflow environment

Overview of Reactis
• A model-based design environment involving Reactis, Simulink and

Stateflow is depicted in Figure below
• Reactis contains three core components:

1. Tester:- Provides automated test generation from models.
2. Simulator:- Enables users to visualize model execution to debug
models and track coverage.
3. Validator:- Offers automated checks of Simulink models for
violations of user-specified requirements.
Overview of Reactis Conti…
• The generated test suites provide comprehensive coverage of

different test coverage metrics - including the Modified
Condition/Decision Coverage (MC/DC) test coverage measure
• The automated test data later can be used for different purposes
like
1. Implementation conformance
2. Model testing and debugging
3. Reverse engineering of models from source

Implementation conformance

Model testing and debugging

Reverse engineering of models from source


Model Based Testing

Uploaded by

Copyright:

Available Formats

Model Based Testing

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Model Based Testing

Uploaded by

Copyright:

Available Formats

Model Based

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

• For this session basic knowledge of Matlab-Simulink, Code

•To know about different types of testing used in software

• To learn different stages of testing & their use in model

7-Mar-12 2 Version 1.0

• Testing, SDLC’s, Objectives of testing and testing artifacts

• Classification of Testing Strategies based on different factors

• Testing Automation, automation for MBT

7-Mar-12 3 Version 1.0

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

• Software testing is an investigation conducted to provide

• It is to be noted that testing can never completely identify all the

• When to perform testing?

7-Mar-12 5 Version 1.0

V model Agile Model

7-Mar-12 6 Version 1.0

• Traditionally testing was limited to debugging, but later when

7-Mar-12 7 Version 1.0

2. Traceability Matrix :- A traceability matrix is a table that

3. Test Vector :- It is a systematic collection in form of array/matrix

7-Mar-12 8 Version 1.0

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

• Based On Knowledge tester has

7-Mar-12 10 Version 1.0

• Based on Level of testing

• Based on way test is executed

•Based on who is tester

7-Mar-12 11 Version 1.0

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

•The term ‘model-based testing’ is widely used today with

•One major benefit of model-based development is the

• Other advantages also include possibility of automation by use

• Between the initial modelling and the integration into the

• These integration levels are

7-Mar-12 14 Version 1.0

7-Mar-12 15 Version 1.0

7-Mar-12 16 Version 1.0

MIL Using Test Vectors MIL Using Plant Modeling

7-Mar-12 17 Version 1.0

7-Mar-12 18 Version 1.0

• These vectors can be handwritten by tester or can be automatically

7-Mar-12 19 Version 1.0

• Plant modelling shall be able to simulate physical

• Testing becomes more effective when plant models with

• Plant models can be made to replicate behavior any

7-Mar-12 20 Version 1.0

• To test this controller a plant model of room can be made as

7-Mar-12 21 Version 1.0

7-Mar-12 22 Version 1.0

7-Mar-12 23 Version 1.0

7-Mar-12 24 Version 1.0

SIL Using Plant Modeling SIL Using Test Vectors

• Usually SIL are performed on Windows/Linux based desktop.

• For SIL testing in Matlab/Simulink Environment following

7-Mar-12 27 Version 1.0