DO Qualification Kit

Simulink® Test™
Tool Qualification Plan

R2020b

June 16, 2020 qualkitdo_sltest_tqp

How to Contact MathWorks
Latest news: www.mathworks.com
Sales and services: www.mathworks.com/sales_and_services
User community: www.mathworks.com/matlabcentral
Technical support: www.mathworks.com/support/contact_us
Phone: 508-647-7000

The MathWorks, Inc.

1 Apple Hill Drive
Natick, MA 01760-2098
DO Qualification Kit Simulink® Test™ Tool Qualification Plan
© COPYRIGHT 2015-2020 by The MathWorks, Inc.
The software described in this document is furnished under a license agreement. The software may be used or
copied only under the terms of the license agreement. No part of this manual may be photocopied or reproduced
in any form without prior written consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or
through the federal government of the United States. By accepting delivery of the Program or Documentation, the
government hereby agrees that this software or documentation qualifies as commercial computer software or
commercial computer software documentation as such terms are used or defined in FAR 12.212, DFARS Part
227.72, and DFARS 252.227-7014. Accordingly, the terms and conditions of this Agreement and only those rights
specified in this Agreement, shall pertain to and govern the use, modification, reproduction, release, performance,
display, and disclosure of the Program and Documentation by the federal government (or other entity acquiring for
or through the federal government) and shall supersede any conflicting contractual terms or conditions. If this
License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See
www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be
trademarks or registered trademarks of their respective holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents
for more information.

June 16, 2020 qualkitdo_sltest_tqp

Revision History
September 2015 New for DO Qualification Kit Version 3.0 (Applies to Release 2015b)
March 2016 Revised for DO Qualification Kit Version 3.1 (Applies to Release 2016a)
September 2016 Revised for DO Qualification Kit Version 3.2 (Applies to Release 2016b)
March 2017 Revised for DO Qualification Kit Version 3.3 (Applies to Release 2017a)
September 2017 Revised for DO Qualification Kit Version 3.4 (Applies to Release 2017b)
March 2018 Revised for DO Qualification Kit Version 3.5 (Applies to Release 2018a)
September 2018 Revised for DO Qualification Kit Version 3.6 (Applies to Release 2018b)
March 2019 Revised for DO Qualification Kit Version 3.7 (Applies to Release 2019a)
September 2019 Revised for DO Qualification Kit Version 3.8 (Applies to Release 2019b)
March 2020 Revised for DO Qualification Kit Version 3.9 (Applies to Release 2020a)
September 2020 Revised for DO Qualification Kit Version 3.10 (Applies to Release 2020b)

June 16, 2020 qualkitdo_sltest_tqp

Contents
1 Introduction ................................................................................................................................................ 1-1
2 Tool Overview and Identification ............................................................................................................... 2-1
2.1 Simulink Test Product Description ...................................................................................................... 2-1
2.2 Simulink Test Product Identification................................................................................................... 2-1
3 Certification Considerations ....................................................................................................................... 3-1
3.1 Requirement for Qualification ............................................................................................................ 3-1
3.2 Certification Credit ............................................................................................................................. 3-2
4 Tool Development Life Cycle – Tool User ................................................................................................... 4-1
4.1 Tool Artifacts ...................................................................................................................................... 4-1
4.2 Planning .............................................................................................................................................. 4-1
4.3 Tool Operational Requirements ......................................................................................................... 4-1
4.4 Integration .......................................................................................................................................... 4-2
4.5 Verification ......................................................................................................................................... 4-2
4.6 Configuration Management ............................................................................................................... 4-2
4.7 Quality Assurance ............................................................................................................................... 4-2
4.8 Qualification Liaisons .......................................................................................................................... 4-2
5 Additional Considerations .......................................................................................................................... 5-1
5.1 Customer Bug Reporting Considerations ........................................................................................... 5-1
5.2 Tool Validation After Installing Updates ............................................................................................. 5-1
5.3 Automation of Tool Qualification ....................................................................................................... 5-1
6 Tool Life Cycle Data .................................................................................................................................... 6-1
7 Schedule ..................................................................................................................................................... 7-1

June 16, 2020 qualkitdo_sltest_tqp i

1 Introduction

This document comprises the Tool Qualification Plan (Reference DO-330 Section 10.1.2) for the following
capabilities of Simulink® Test™. It is intended for use in the DO-178C and DO-330 tool qualification process
for Criteria 3 TQL-5 tools.

• Comparing simulation results or executable object code test results against expected results to
determine pass/fail simulation or test status.
• Generating reports that contain information on pass/fail simulation or test status, and requirements
tracing

June 16, 2020 qualkitdo_sltest_tqp 1-1

2 Tool Overview and Identification

2.1 Simulink Test Product Description

Develop, manage, and execute simulation-based tests
Simulink® Test™ provides tools for authoring, managing, and executing systematic, simulation-based tests of
models, generated code, and simulated or physical hardware. It includes simulation, baseline, and
equivalence test templates that let you perform functional, unit, regression, and back-to-back testing using
software-in-the-loop (SIL), processor-in-the-loop (PIL), and real-time hardware-in-the-loop (HIL) modes.

With Simulink Test you can create nonintrusive test harnesses to isolate the component under test. You can
define requirements-based assessments using a text-based language, and specify test input, expected
outputs, and tolerances in a variety of formats, including Microsoft® Excel. Simulink Test includes a Test
Sequence block that lets you construct complex test sequences and assessments, and a test manager for
managing and executing tests. Observer blocks let you access any signal or state in the design without
changing the model or the model interface. Large sets of tests can be organized and executed in parallel or
on continuous integration systems.

You can trace tests to requirements (with Simulink Requirements™) and generate reports that include test
coverage information from Simulink Coverage™.

2.2 Simulink Test Product Identification

Software Tool Version (Release) Tool Vendor
Simulink Test Version 3.3 (R2020b) The MathWorks, Inc.
1 Apple Hill Drive
DO Qualification Kit Version 3.10 (R2020b) Natick, MA, 01760-2098 USA

June 16, 2020 qualkitdo_sltest_tqp 2-1

3 Certification Considerations

This section provides the certification considerations for the following capabilities of the Simulink Test
verification tool:

• Compare simulation results or executable object code test results against expected results to determine
pass/fail simulation or test status.
• Generate reports that contain information on pass/fail simulation status, and requirements tracing.

3.1 Requirement for Qualification

To determine whether a tool must be qualified, the following questions based on guidance of paragraph
12.2.1 of DO-178C are addressed:

Question Answer
Can the tool insert an error into the airborne software or fail to detect an existing error in the Yes1
software within the scope of its intended usage?

Will the output of the tool not be verified as specified in Section 6 of DO-178C, DO-278A, DO- Yes
331, DO-332 or DO-333?

Are processes of DO-178C, DO-278A, DO-331, DO-332 or DO-333 eliminated, reduced, or Yes
automated by the use of the tool? Will you use output from the tool to meet an objective or
replace an objective of DO-178C, DO-278A, DO-331, DO-332 or DO-333, Annex A or Annex C?

Because the answers to the preceding questions are yes, Simulink Test shall be qualified.

To determine the type of qualification (Criteria 1, Criteria 2, or Criteria 3), the following questions based on
guidance of paragraph 12.2.2 of DO-178C about the tool are addressed:

Question Answer
Is the tool output part of the airborne software, such that the output can insert an error into No
the software?

Could the tool fail to detect an error in the airborne software and is the tool also used to No
justify the elimination or reduction of either of the following:
• Verification processes other than that automated by the tool.
• Development processes that could have an impact on the airborne software.
Could the tool fail to detect an error in the airborne software? Yes

Because the answer to the preceding first and second questions are no, Simulink Test can be qualified as a
Criteria 3 tool.

1 Simulink Test might fail to detect an error.

June 16, 2020 qualkitdo_sltest_tqp 3-1

3.2 Certification Credit
The following table shows the certification credit being sought for Simulink Test.

Table 1 Simulink Test with Respect to DO-331, DO-178, and DO-278 Objectives

DO-331 DO-178/278
Objective Table Reference Table Reference Credit Taken

(MB10) Simulation MB.A-3 MB.6.8.3.2.c N/A N/A Partial2 – Simulink Test is used
results are correct MB.C-3 to compare actual simulation
and discrepancies results to expected results;
explained discrepancies have to be
analyzed and explained
manually.

(MB16) Simulation MB.A-4 MB.6.8.3.2.c N/A N/A Partial2 – Simulink Test is used
results are correct MB.C-4 to compare actual simulation
and discrepancies results to expected results;
explained discrepancies have to be
analyzed and explained
manually.

(1) Executable Object N/A N/A A-6 6.4.a Partial3 – Simulink Test is used
Code complies with to compare actual executable
high-level object code test results to
requirements expected results. The expected
results must be provided as
input to this tool.

(2) Executable Object N/A N/A A-6 6.4.b Partial3 – Simulink Test is used
Code is robust with to compare actual executable
high-level object code test results to
requirements expected results. The expected
results must be provided as
input to this tool.

(3) Executable Object N/A N/A A-6 6.4.c Partial3 – Simulink Test is used
Code complies with to compare actual executable
low-level object code test results to
requirements expected results. The expected
results must be provided as
input to this tool.

(4) Executable Object N/A N/A A-6 6.4.d Partial3 – Simulink Test is used
Code is robust with to compare actual executable
low-level object code test results to
requirements expected results. The expected
results must be provided as
input to this tool.

2 This credit is taken only if the Simulink® and Stateflow® models are considered as a Design or Specification Models for the project and
simulation is used as a means of compliance for objectives 1, 2, 4, or 7 of Tables MB.A-3, MB.A-4, MB.C-3 or MB.C-4.
3 This credit is taken only if Simulink Test is used to automatically verify output from execution of executable object code.

June 16, 2020 qualkitdo_sltest_tqp 3-2

 DO-331 DO-178/278
Objective Table Reference Table Reference Credit Taken

(3) Test coverage of N/A N/A A-7 6.4.4.a Partial3 – Simulink Test is used
high-level to generate a report for
requirements is traceability between high-level
achieved requirements and test. The
links between high-level
requirements and test must be
set manually.

(4) Test coverage of N/A N/A A-7 6.4.4.b Partial3 – Simulink Test is used
low-level to generate a report for
requirements is traceability between high-level
achieved requirements and test. The
links between high-level
requirements and test must be
set manually.

June 16, 2020 qualkitdo_sltest_tqp 3-3

4 Tool Development Life Cycle – Tool
User

4.1 Tool Artifacts

The following table identifies the Simulink Test artifacts that are available in the Artifacts Explorer, the
Simulink Test folder. The User’s Guide and Reference artifact are in the \doc subfolder.

Artifact File
Simulink Test Tool Qualification Plan, R2020b (this qualkitdo_sltest_tqp.docx/pdf
document)

Simulink Test Tool Operational Requirements, R2020b qualkitdo_sltest_tor.docx/pdf

Simulink Test Test Cases and Procedures, R2020b qualkitdo_sltest_tcp.docx/pdf

Execute Qualification Tests and Review Test Results for qualkitdo_sltest_run.mlx

Simulink Test, R2020b

Simulink Test User’s Guide, R2020b sltest_ug.pdf

Simulink Test Reference, R2020b sltest_ref.pdf

4.2 Planning
The Plan for Software Aspects of Certification (PSAC) designates that the Simulink Test will be qualified as a
Criteria 3, TQL-5 tool.

This document provides the Tool Qualification Plan for Simulink Test.

4.3 Tool Operational Requirements

Tool operational requirements for Simulink Test are in the Simulink Test Tool Operational Requirements
artifact.

User information for Simulink Test is in:

• Simulink Test User’s Guide

• Simulink Test Reference

The applicant will:

• Review the tool operational requirements for applicability to the project under consideration.
• Configure the tool operational requirements in a configuration management system.

June 16, 2020 qualkitdo_sltest_tqp 4-1

4.4 Integration
Instructions for installing Simulink Test are available at the MathWorks Documentation Center, R2020b >
Installation.

The applicant will:

• Install Simulink Test using the installation instructions

• Create a Tool Installation Report (DO-330, Table T-0 objective 3)

4.5 Verification
Requirements-based test procedures are developed from the tool operational requirements by MathWorks
and provided to the tool user to execute in the installed environment. The test cases and procedures will be
developed in the form of the Simulink Test test file, which exercises the capabilities being qualified in
Simulink Test.

For general information about the testing procedure, see Simulink Test Test Cases and Procedures.
Executing the test procedure generates results that are published in tool qualification reports; see artifact
Execute Qualification Tests and Review Test Results Simulink Test for more information.

The applicant will:

• Review the test cases and procedures for applicability to the project under consideration.
• Configure the test cases and procedures in a configuration management system.
• Execute the test cases and procedures in the installed environment.
• Review the test results and confirm they are correct and discrepancies are explained.
• Configure the test results in a configuration management system

4.6 Configuration Management

The applicant will implement the project-specific configuration management activities to handle the
shipped and user-generated tool qualification data (DO-330, Table T-8).

4.7 Quality Assurance

The applicant will implement the project-specific quality assurance for the tool qualification activities
performed by the tool users (DO-330, Table T-9).

4.8 Qualification Liaisons

The applicant will implement the project-specific qualification liaisons activities (DO-330, Table T-10).

June 16, 2020 qualkitdo_sltest_tqp 4-2

5 Additional Considerations

5.1 Customer Bug Reporting Considerations

MathWorks reports known critical bugs brought to its attention on its bug report system at
www.mathworks.com/support/bugreports. The bug reports are an integral part of the documentation for
each release.

The bug report system provides an interface for customers to view and submit bug reports. Users can track
the status of open bugs. Users can choose to receive notifications for new or updated bug reports. The bug
reports on this web site include internally and externally nominated bugs. If applicable, bug reports include
provisions for known workarounds or file replacements. Customers can use the bug report mechanism to
nominate bugs. These nominations are processed and evaluated by The MathWorks, Inc. development
organization.

To open the DO Qualification Kit bug report checks, from the Simulink toolstrip:

• Open the DO Qualification Kit app and select Bug reports.

• Open the Modeling tab and select Model Advisor.

The Model Advisor opens. Select and run the check for the bug report you would like to review.

5.2 Tool Validation After Installing Updates

Changing the tool installation can affect tool behavior, requiring the user to revalidate the tool. When the
user installs a release update or a patch to address issues in a bug report, the user must repeat the tool
qualification steps (rerun tests and review results) to verify that the tool satisfies the tool operational
requirements.

5.3 Automation of Tool Qualification

The DO Qualification Kit includes tests for automated comparison, which compare generated results (actual
reports) against the expected results (baseline reports). The results of the comparison provide assurance
that the tool operates correctly in the installed environment. The test cases and procedures for the
automation capability are documented in the Simulink Test Test Cases and Procedures document.

June 16, 2020 qualkitdo_sltest_tqp 5-1

6 Tool Life Cycle Data

The following table shows the life cycle data for the Simulink Test capability. The table maps the documents
and artifacts to DO-330 life cycle data items. For additional information about the documents and artifacts,
contact MathWorks.

In the following table, the Tool User Action column defines the responsibility of the tool user with regard to
making the data available to certification authorities. For data marked “Submit”, the tool-user must deliver
the data to the certification authorities. When marked “Available”, the data must be available at the tool-
user’s or tool vendor’s site for inspection by the certification authorities.

Table 2 Life Cycle Data

Reference and Data Support Documents or Artifacts Tool User Note

Action
DO-330, section 10.1.1 <Insert reference to the PSAC or Submit MathWorks provides a template for
Tool-Specific Information in PSAA artifact> the PSAC.
Plan for Software Aspects of The tool user can customize the
Certification (PSAC) template as appropriate for their
project.
The information in these sections
can be used to create the artifact:
• Tool Overview and Identification
on page 2-1
• Certification Credit on page 3-2
DO-330, section 10.1.2 <Insert reference to the Simulink Submit MathWorks provides the tool
Tool Qualification Plan Test Tool Qualification Plan (this specific TQP artifact.
document) artifact> The tool user can customize the
artifact as appropriate for their
project.

DO-330, section 10.3.1 <Insert reference to the Simulink Available Authored by MathWorks.
Tool Operational Test Tool Operational Requirement
Requirements artifact and tool user
documentation>

DO-330, section 10.3.3 and Execute Qualification Tests and Available MathWorks provides test cases.
10.2.5 Review Test Results for Simulink Test The tool user can add or remove
Test Cases and Procedures Simulink Test Test Cases and test cases as appropriate for the
Procedures tests used in their project.
Test Cases
qualkitdo_sltest_rs.rpt

June 16, 2020 qualkitdo_sltest_tqp 6-1

Reference and Data Support Documents or Artifacts Tool User Note
Action
DO-330 section 10.3.4 and Qualification reports. Available MathWorks provides the tool
10.2.6 qualification reports.
Test Results The tool user provides test results
as appropriate for the tests
executed in their project.

DO-330 section 10.1.16 <Insert reference to Software Submit Provided by tool user.
Tool-specific information in Accomplishment Summary artifact>
Software Accomplishment
Summary (SAS)

DO-330 section 10.1.15 <Insert reference to the Tool Submit Provided by tool user.
Tool Accomplishment Qualification Accomplishment
Summary Summary artifact>

June 16, 2020 qualkitdo_sltest_tqp 6-2

7 Schedule

<Insert tool schedule>

June 16, 2020 qualkitdo_sltest_tqp 7-1